Title: BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: small language models

reasoning

policy optimization

chain of thought

reinforcement Learning

GRPO
Abstract: Small language models (SLMs) struggle to learn complex reasoning behaviors, especially when high-quality traces are scarce or difficult to learn from. A typical approach for training such models combines a supervised fine-tuning (SFT) stage, often to distill reasoning capabilities from a larger model, followed by a reinforcement learning (RL) stage such as Group Relative Policy Optimization (GRPO). In this paper, we investigate the fundamental limitations of this SFT + RL paradigm and propose methods to overcome them. Using a toy student-expert model over Markov chains, we demonstrate that the SFT + RL strategy can fail completely when (1) the expert's traces are too difficult for the small model to express, or (2) the small model's initialization achieves exponentially sparse rewards as task complexity grows. To address these, we introduce BREAD, a GRPO variant that bridges SFT and RL via partial expert guidance and branch rollouts. When self-generated traces fail, BREAD adaptively inserts short expert prefixes/hints, allowing the small model to complete the rest of the reasoning path, and ensuring that each update includes at least one successful trace. This mechanism both densifies the reward signal and induces a natural learning curriculum. BREAD requires fewer than 40\% of ground-truth traces, consistently outperforming standard GRPO while speeding up the training by about 3$\times$. Importantly, we find that BREAD helps the model solve problems that are otherwise unsolvable by the SFT + RL strategy, highlighting how branch rollouts and expert guidance can aid SLM reasoning.

BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning

Abstract

Small language models (SLMs) struggle to learn complex reasoning behaviors, especially when high-quality traces are scarce or difficult to learn from. The standard training approach combines a supervised fine-tuning (SFT) stage, often to distill capabilities of a larger model, followed by a reinforcement learning (RL) stage such as Group Relative Policy Optimization (GRPO). In this paper, we investigate the fundamental limitations of this SFT + RL paradigm and propose methods to overcome them. Under a suitable theoretical model, we demonstrate that the SFT + RL strategy can fail completely when (1) the expert's traces are too difficult for the small model to express, or (2) the small model's initialization has exponentially small likelihood of success. To address these, we introduce BREAD: a GRPO variant that unifies the SFT and RL stages via partial expert guidance and branched rollouts. When self-generated traces fail, BREAD adaptively inserts short expert prefixes/hints, allowing the small model to complete the rest of the reasoning path, and ensuring that each update includes at least one successful trace. This mechanism both densifies the reward signal and induces a natural learning curriculum. BREAD requires fewer than 40% of ground-truth traces, consistently outperforming standard GRPO while speeding up the training by about 3 ˆ . Importantly, we demonstrate that BREAD helps the model solve problems that are otherwise unsolvable by the SFT + RL strategy, highlighting how branched rollouts and expert guidance can substantially boost SLM reasoning.

1 Introduction

Over the past few years, we have witnessed a significant push toward enhancing language model reasoning, which has led to highly capable frontier models such as OpenAI o1 [17], Gemini 2.5 [19], and DeepSeek R1 [13]. These models can generate longer chain-of-thought (CoT) traces and utilize more test-time compute to tackle challenging tasks [24]. Despite these innovations, reasoning with small language models (SLM) remains a challenge. For instance, the large DeepSeek-R1 [13] has 671B parameters in total whereas the SFT-distilled model sizes range from 1.5B to 70B, and their performance substantially degrades at the 1.5B model (see Table 5 in [13]).

This work studies optimization strategies to enhance SLMs, with emphasis on reasoning tasks. Two popular optimization strategies for training LLMs are supervised fine-tuning (SFT) and reinforcement learning (such as GRPO or proximal policy optimization). Often, an SFT phase is employed, followed by an RL phase. While this two-stage procedure has found success, for SLMs, long-context reasoning

Yingcong Li

Samet Oymak

Supervised

Fine-tuning

Too difficult to learn

Reinforcement

Learning

Weak base model,

Sparse reward

BREAD

+Expert trace,

Dense reward

Fine-tuned

problems can pose unique challenges due to the misalignment between the expert and student models and potentially sparse rewards.

For example, consider a scenario in which each token generated by an expert/teacher model requires the smaller student model to produce K intermediate tokens to express it. In other words, the expert thinks and outputs K steps ahead, from the student's point of view. In practice, this situation can arise when the expert model is a ˆ K deeper/looped version of the student. Naturally, such expert traces might be too challenging for the small model to learn from 2 . On the other hand, success of the RL phase often relies on good supervised initialization during the SFT phase. In Section 2.1, we provide a mathematical setting capturing this intuition and demonstrate that SFT + RL can fail for small models (see Figure 3), especially on difficult problems.

To address the difficulties of small models in learning from complex traces during fine-tuning, we propose our algorithm, Branch Rollouts and Expert Anchors for Densified rewards, depicted in Figures 1 and 2. BREAD gracefully integrates the SFT and RL phases by anchoring the optimization process with the expert traces, while allowing the small base model to acquire progressively more flexibility as it becomes a stronger problem solver. Assuming that an expert trace is available (e.g. by querying a large expert model), BREAD updates the model with a correct trace; however, its traces are progressively more self-generated.

Contributions: Our specific contributions are as follows:

Methodology: We introduce BREAD: a GRPO variant that generalizes conventional SFT and RL phases through the use of branched rollouts to automatically adapt to the problem difficulty. BREAD induces a learning curriculum along the reasoning trace, thereby densifying the reward signal and tackling compositional tasks step-by-step. We investigate the theoretical properties of BREAD in a suitable student-expert setting. In this setting, we establish how the expert model's trace can be uninformative to the student and how the subsequent RL phase can fail due to sparse rewards. In contrast, we show that BREAD can solve the target problem efficiently (low training cost) and concisely (short trace length).

Empirical impact: Our experimental results show that BREAD matches or surpasses SFT + vanilla GRPO while using « 20% of the correct trace tokens. By reducing both the number of rollouts and the total optimization steps, BREAD reduces the overall training compute by « 75% relative to vanilla GRPO. Additionally, we construct a slice of difficult problems and demonstrate that BREAD, when trained on this set, achieves substantially higher accuracy compared to SFT+GRPO, corroborating its fundamental benefits in solving difficult problems that are otherwise unsolvable by SFT+GRPO. Finally, we provide an empirical study of how branched rollouts from expert hints (i.e., using only parts of the expert trace) densifies the reward signal.

The rest of the paper is organized as follows: Section 5 dicusses the related work on language model reasoning and traditional RL methods. Section 1 explains our algorithm BREAD, which also contains the observations inspiring our algorithm design. Section 4 presents and discusses our main experiment results to demonstrate the effectiveness of BREAD. Section 6 concludes the paper and discusses the limitation and future directions to improve in this domain.

Question

❌

❌

❌

Question

✅

❌

Hint[0:cur]

Question

Step 3

Policy update

If all

✅

, shorten hint

If all

❌

If all

❌

, lengthen hint

Else, terminate the search

Hint[0:cur-]

Question

Hint[0:cur+]

Question

Step 2

Episode anchor search

(EAS)

Else

❌

Step 1

Regular rollout

2 Proposed Method: BREAD

In this work, we propose the B ranched R ollouts and E xpert A nchors for D ensified RL (BREAD) algorithm. We will first describe the algorithm at a high level, followed by a mathematical toy model example (Section 2.1), and the key observations (Section 3.1, Section 3.2) that support its design.

In BREAD, for each question q paired with the answer a , the workflow of BREAD is shown in Figure 2 and proceeds as follows:

Regular rollout: Sample a group of G rollouts t o i u G i ' 1 .

Episode anchor search: If the success rate of the initial group is low (e.g. lower than a threshold), do a binary search to find a short hint ρ that contains the 'Expert Anchor' in the expert solution. 'Expert Anchor' means the success rate of a new sampled output group t o 1 i u G i ' 1 , resulting from the question appended with the hint from the expert trace p q, ρ q , is within a pre-defined range.

Policy updates: Optimize the policy via the following objective:

where

The details of the algorithm can be found in Algorithm 1. Next, we will discuss the key observations that motivate the design of BREAD.

2.1 Theoretical Insights into SFT, Reinforcement Learning, and BREAD

Challenges of SFT: It is known that a L times deeper LLM can internally simulate L chain-of-thought steps of a smaller LLM [31]. This implies that the expert model can generate a dense reasoning trace which necessitates a L ˆ longer simplified trace for the student model to digest via SFT. Otherwise, the student model may fail to learn from SFT as we will elaborate further below. Recent work [22] makes related empirical observation that SLMs can struggle to learn from strong reasoners.

Insights from compositionality: Reasoning problems are inherently compositional hence we need a chain-of-thought [39] to solve such problems. Suppose the problem contains T steps/subtasks to solve, each with successful completion probability of ϵ for the base student model. Then, we receive a final trajectory-level correctness reward with probability of ϵ T assuming steps are independently completed. In contrast, by branching out from the expert trace at step T ´ τ , BREAD only needs to complete the τ downstream steps, increasing the success to ϵ τ . Notably, by increasing τ from 1 to T , BREAD facilitates curriculum learning by tackling individual problem steps one at a time requiring only Ω p 1 { ϵ q samples to attain success per step. Below we further formalize this discussion under a Markov chain model.

Markov chain model: Let us model the expert and student models through Markov chains as follows: Imagine a Markov chain over the States t 0 , 1 , 2 , . . . , K u . We envision that the student model is

0

2000

4000

6000

8000

10000

Training step

0.0

0.2

0.4

0.6

0.8

1.0

Test accuracy

BREAD,

=0.05

BREAD,

=0.025

BREAD,

=0.01

SFT,

=0.05

SFT,

=0.025

SFT,

=0.01

GRPO,

=0.05

GRPO,

=0.025

GRPO,

=0.01

(a) Varying

ϵ

- Initialization difficulty

0

2000

4000

6000

8000

10000

Training step

0.0

0.2

0.4

0.6

0.8

1.0

Test accuracy

BREAD,

K

=30

BREAD,

K

=50

BREAD,

K

=100

SFT,

K

=30

SFT,

K

=50

SFT,

K

=100

GRPO,

K

=30

GRPO,

K

=50

GRPO,

K

=100

(b) Varying

K

- Number of states

forbidden from implementing certain state transitions. Specifically, its connections are local and consecutive state transitions obey | x t ` 1 ´ x t | ď d where the agent reaches state x t at time t , and d denotes its maximum allowable jump distance of the small model. We consider the learnability of the following navigation task by the student.

Navigation Task: Start a trace from State 0. Obtain a reward of 1 upon reaching State K

.

Suppose the expert model solves the Navigation Task by generating the trace T e ' r x 0 ' 0 , x 1 , x 2 , . . . , x T ´ 1 , x T ' K s where x t P t 1 , . . . , K ´ 1 u for 1 ď t ď T ´ 1 . During SFT, the student Markov chain can only learn from T e when the expert transitions p x t , x t ` 1 q are learnable, i.e., | x t ´ x t ` 1 | ď d . If no transition satisfies this, the small model cannot learn from SFT.

Without a good SFT-induced initialization, pure RL is known to suffer from sparse rewards [37]. In the context of the Navigation Task, once SFT fails, the student model can only improve through vanilla RL if it can generate a trace that successfully runs from 0 to K to earn a nonzero reward. In contrast, if the student employs BREAD, rather than starting from 0 , it can create a rollout starting from x T ´ 1 . Thus, it only needs to complete a much easier task starting from x T ´ 1 , earning a reward by reaching the final state x T ' K . In the context of compositionality, generation of traces from x t ´ 1 to x t corresponds to solving t 'th subtask.

To proceed, we have experimented with the Navigation Task controlled by the parameters number of states K ` 1 , the student model's jump capacity d , and initialization quality ϵ . Specifically, for each state i , the student model can jump to a neighbor j ‰ i , | i ´ j | ď d with probability at most ϵ . Hence, larger the ϵ , the more model struggles under trace length constraint. Figure 3 demonstrates how optimization of SFT and GRPO can suffer as a function of K and ϵ respectively. Importantly, we also display the performance of BREAD which exhibits a much more favorable performance. The reader is referred to the supplementary material for full experimental details. To proceed, we provide a concrete theoretical analysis of this discussion in the next section.

2.2 Formal Guarantees under a Random Walk Model

Setting and assumptions: The expert trace starts at State 0 and ends at State K by following the trajectory x 0 ' 0 , x 1 , . . . , x T ' K . We assume that c ¨ K { T ě x t ` 1 ´ x t ě K { cT for some c ě 1 and all t , and that the interval r K { cT, cK { T s contains at least one positive integer. In words, the expert model has a jump size of Ω p K { T q .

We assume that the student model is initialized as a symmetric random walk . That means, it can only move to left or right neighbor (jump size d ' 1 and ϵ ' 0 . 5 ) and it is allowed to visit negative states. Thus, optimizing the student model is same as choosing optimal neighbor transition probabilities for a random walk. In our context, the ideal transition rule is always moving to the right (from i to i ` 1 ).

Finally, we enforce a maximum trace length of L and, during RL, the model only receives reward if the trace successfully reaches state K within L steps. With these, we have the following lemma that establishes that SFT on the expert trace followed by reinforcement learning cannot improve the student model with high probability unless L ě Ω p K 2 q .

Lemma 1 (Failure of SFT+RL) Suppose the student model is a Markov chain with d ' 1 initialized as a symmetric random walk. Suppose the expert trace p x t q T t ' 1 always jumps two or more (i.e. K ą

c ¨ T ). Given the expert trace and at most N ď e K 2 { 4 L traces generated during the RL phase, SFT and GRPO training have no impact on the student model with probability at least 2 e ´ K 2 { 4 L .

In words, this theorem states that the student is not able to learn from expert due to limited expressivity and it is not benefiting from the RL phase due to never obtaining nonzero reward. This failure happens unless the trace is long in the sense that L ě Ω p K 2 q . Indeed, this is the time it takes for a symmetric random walk (initial student model) to hit State K .

Our next result proves how BREAD can overcome this bottleneck and address this bottleneck while conforming to a shorter trace length. To this aim, we examine a variation of BREAD that relies on a simple memorization of successful traces.

Algorithm: BREAD variant. The algorithm starts from the right most end of the expert trajectory and incrementally learns the correct completion. Concretely,

, During the first round, create IID rollouts of length at most L starting from the prefix p x t q T ´ 1 t ' 0 . Once a reward of 1 is achieved, record the generated suffix S 1 ' p x 1 t q t ě T

!

!

!

"

!

#

!

$

.

, At round τ ď T , create IID rollouts of length at most L starting from the prefix p x t q T ´ τ t ' 0 . Terminate a rollout once it hits a state s hit in S τ ´ 1 and complete the trace by replaying S τ ´ 1 from s hit. Declare success if the total length is at most L and set S τ to be the union of generated rollout and S τ ´ 1 continued from s hit .

This algorithm incrementally learns sub-trajectories and stitch them to eventually obtain a full trajectory S T that solves the entire problem. This is depicted in Figure 4. During inference, the model simply replays the final memorized trace S T (yellow dashed curve in Figure 4).

We have the following theorem that establish the sample complexity of this BREAD variant.

Theorem 1 (Success of BREAD) Suppose L ě 5 c 2 K 2 { T and assume BREAD generates t rollouts at each round 1 ď τ ď T . Then, with probability at least 1 ´ Te ´ t , BREAD succeeds in all T rounds in creating S τ while never violating the maximum trace length L . Thus, BREAD memorizes a successful student trace S T with length at most L with the same probability.

Afew remarks are in place: First, when L 9 K 2 { T , BREAD succeeds by using a total of Ω p T log p T qq traces over T rounds whereas SFT+RL from Lemma 1 fails with probability at least 1 ´ 2 e ´ Ω p T q confirming our intuition on reward densification benefits. Secondly, suppose T 9 K i.e. the expert model has constant jump size. In this case, BREAD solves the problem in linear time as it requires L Á O p K q . In contrast, SFT+RL requires L Á O p K 2 q to solve the problem. In the next section, we discuss how these theory and insights are in line with the SFT and RL performance on real reasoning tasks with state-of-the-art models.

3 Empirical Insights into BREAD

3.1 Limitations of SFT and Reinforcement Learning for SLMs

RL-only evaluations: We begin by assessing how well RL works with SLMs in isolation. Experiments with vanilla GRPO [33], displayed in Figure 5, show that it merely sharpens the capabilities the model already exhibits. GRPO relies on sampling a mix of good and bad traces. But when a small base model fails to produce any high-quality tracenearly half of the queries in our evaluations in Figure 5a-learning stalls due to lack of reward signals (Figure 5b). Related limitations are noted for RL with Verifiable Rewards of [42], which struggles to elicit fundamentally new reasoning patterns.

SFT-only evaluations: SFT can introduce new knowledge into the model. However, as discussed earlier, traces generated by much stronger models can be too complex for SLMs to learn resulting in poor initialization for RL. To demonstrate this, we start from Qwen2.5-1.5B-Instruct and Qwen2.53B-Instruct base models and fine-tune them with 1000 difficult questions, paired with reasoning traces generated by Gemini Thinking Experimental [11] and Deepseek-R1 [13], following [35]. These reasoning traces were previously shown to improve the reasoning capability of large models such as Qwen2.5-32B-Instruct and Qwen2.5-14B-Instruct by [24]. However, in our experiment results shown in Table 1, when SFT was performed on small models, accuracy actually decreases . For example,

batch/solve_none

Qwen2.5-3B-Instruct

125

120

115

110

105

100

Step

95

0

20

40

60

80

(a) Number of samples where all 8 rollouts fail

val/test_score/Al-MO/NuminaMath-CoT

Qwen2.5-3B-Instruct

0.51

0.5

0.49

0.48

0.47

Step

10

20

30

40

50

60

70

80

(b) Validation accuracy over training steps

Increasing Curriculum Difficulty

GSM8k

→

MATH

→

AIME

→

IMO

Along

data

Along

reasoning

trace

expert

trace

hard

medium

easy

(a) Illustration of hint ratio (how much of the expert trace is provided) and curriculum learning.

0.0

0.2

0.4

0.6

0.8

Hint Ratio

0.2

0.3

0.4

0.5

0.6

0.7

Correctness Ratio

Base

accuracy dropped from 0.257 to 0.177 on the GPQA dataset when the 1.5B parameter model was fine-tuned on S1K traces, and dropped even further to 0.121 when fine-tuned on the more verbose S1K-1.1 data. This corroborates our central intuition: when traces used for SFT exceed an SLM's learning capacity, they can hurt the model performance rather than helping.

These findings-that for small models, RL alone, or SFT + RL, are insufficient-motivate BREAD which uses expert traces directly in the RL phase instead of relying on SFT to learn them first.

3.2 Why BREAD: Expert Traces Provide Critical Guidance and Curriculum for RL

While Section 3.1 shows that an SLM can struggle to imitate traces generated by powerful LLMs, we posit that the SLM can still understand them well enough to extract useful information , namely by using part of the complex traces as hints to lower the problem difficulty. To illustrate this, we define the 'hint ratio' as the fraction of the expert trace used (in terms of number of episodes), as illustrated in Figure 6a. A higher hint ratio means more guidance for the model. In Figure 6b, we plot the model accuracy for different hint ratios, where the hint is appended to the inference query. We can see that our intuition is correct: even providing part of the hint in a very simple way (by appending to the inference query) helps improve accuracy. In contrast, Figure 7 shows that SFT only on the same traces gets very limited improvement and can even hurt. BREAD is motivated by this observation and uses expert hints in a more sophisticated way, by integrating them directly into the RL create denser rewards.

We can also view BREAD through the lens of curriculum learning. As shown in Figure 6a, as we branch out from the expert trace earlier, solving the problem becomes more challenging. The Episode Anchor Search (EAS) step in BREAD automatically adjusts the branching point, producing a self-paced curriculum. In this sense, BREAD serves as a generalized curriculum-learning framework

which (1) allows us to utilize SFT data within RL, (2) adapts the optimization to the problem difficulty, and (3) generates dense rewards by creating a curriculum along the reasoning trace. By branching along the expert trace at adaptive cut-points, BREAD automatically surfaces the most informative training samples and tunes their difficulty on the fly. This relieves us from hand-crafting a data-level schedule and lets the model discover its own curriculum.

A final surprising observation from Figure 6b is that conditioning RL training on questions with partial hints not only improves performance on hinted queries (e.g., orange dots with hint ratio 0.3 and above), but also improves accuracy on questions without any hints (i.e., orange dot with hint ratio of 0). This highlights the model's ability to generalize from partial short traces to full long traces.

Algorithm 1 BREAD : GRPO with Branched Rollouts and Expert Anchors (see Algorithm 2 in Appendix for full version.)

4 Experiments

Baseline algorithms. We evaluate the effectiveness of our method BREAD (as described in Algorithm 1) on mathematical tasks, which can be easily adapted to other reasoning tasks such as coding, commonsense reasoning, etc. We compare with the following baselines:

GRPO : Standard GRPO.

SFT : Standard supervised fine-tuning on the full training set. We denote SFT on various data splits as SFT(X) , e.g. SFT(full) for the whole dataset, SFT(difficult) for the hardest subset, SFT(random) for a sizematched random subset, and SFT(selected) for the EAS-chosen subset. The precise discussion is detailed with the experiments results.

SFT + GRPO : This means SFT is run followed by GRPO . In other words, GRPO continues from the final checkpoint of the same SFT run. We denote GRPO fine-tuning initialized from an SFT model trained on data split X as SFT(X) + GRPO .

GRPO w/ Expert Trace : During GRPO training, the entire expert trace is injected as an additional rollout for all queries. This is a strong baseline as it contains the expert. Details in Appendix C.

Training settings. We adopt the verl framework [34] for training. We utilize the Adam optimizer [20] with a constant learning rate of 1 ˆ 10 ´ 6 . For rollout, the prompt batch size is 256 and we sample 8 responses for each prompt. For training, the mini-batch size is 64. To find the appropriate branching point during the Episode Anchor Search (EAS) step of BREAD (Section 1), we first split the expert

trace into sentences (on easier datasets like MATH [15], where the expert traces are generally short) or into paragraphs (on harder datasets like NuminaMth-CoT [21] and OpenR1-Math-220K [9]). Then, we aggregate the split partitions into K ' 10 episodes evenly, and proceed with binary search. See details in Appendix B.3.

4.1 Main Results

BREAD outperforms all baselines in terms of test accuracy. Figure 7 plots the training curves of all methods on the NuminaMath-CoT benchmark, starting from the Qwen-2.5-3B-Instruct base model. In Figure 7a, we visualize the test accuracy against training steps for BREAD and the baseline methods. BREAD outperforms all the baselines. Let us discuss each baseline in turn. BREAD improves final accuracy by more than 15% over vanilla GRPO . SFT can offer a stronger starting point for RL, which we can see by comparing the SFT and SFT+GRPO curves, but BREAD is still better. (Note that training SFT for more iterations does not further improve performance, see Appendix B.1.) SFT(Difficult) represents SFT trained on the hardest questions from NuminaMathCoT, obtained by sorting samples by solution length. Intuitively, problems that require longer solutions are typically more complex and involve more reasoning steps. Comparing the SFT(Difficult) and SFT(Difficult)+GRPO curves, we can see that SFT with too complex expert traces can even hurt the performance of SLM, which further hurts the later GRPO stage. Finally, from the GRPO w/ Expert trace curve, we see that adding the expert trace as an extra rollout during GRPO mitigates sparse rewards and shows noticeable performance improvement, but it is still worse than BREAD. We suspect this is because there is a distribution gap between small and large model's reasoning traces, which makes SLM imitation of large models hard, while BREAD can reduce the gap during learning by letting the SLM figure out the reasoning steps by itself more. See further discussion below on 'hard questions' for elaboration. We also provide more results among different base models and datasets in Table 2 and Appendix B.

BREAD reduces training time. BREAD is also markedly more training-efficient. It can reach the accuracy of the best baseline, SFT+GRPO , in just 25% of the training steps (Figure 7a), translating to roughly a 75% reduction in total compute FLOPs (Figure 7b). We estimated the FLOPs based on the common cost evaluation method used in recent scaling-law studies [35, 16, 30], by counting a forward pass as 2 ND and a backward pass as 4 ND . Here N is the number of model parameters and D is the total token count processed in that pass. For calculation details, see Appendix A.1. Note that we estimate D as the average length of all expert traces in the training set, but according our measurements, BREAD's actual generation length is always shorter. Therefore for BREAD, its actual token count, and thus its FLOP cost, is even lower than the estimated values reported Figure 7b. We also discuss the potential of reducing the number of rollout needed to reduce training cost in Appendix B.2.

0

50

100

150

200

250

300

Training step

0.45

0.50

0.55

0.60

0.65

Test accuracy

(a) Test accuracy over training steps

0.0

0.5

1.0

1.5

2.0

2.5

Estimate FLOPs (B)

1e9

0.45

0.50

0.55

0.60

0.65

Test accuracy

BREAD

GRPO w/ Expert trace

SFT

SFT(Difficult)

SFT+GRPO

SFT(Difficult)+GRPO

GRPO

(b) Test accuracy vs estimated FLOPs

BREAD succeeds on the hard questions, where other baselines fail. To understand the gains of BREAD, we train and evaluate each method exclusively on very difficult problems. The goal is to investigate whether expert hints in BREAD can help SLMs learn new information particularly from these hard questions. We conduct the experiment on the NuminaMath-CoT dataset and the Qwen2.5-3B-Instruct as the base model. To build the hard dataset, we first run ordinary SFT and from the training dataset, we select the 500 questions for which three independent generations produce no correct trace (pass@ 3 ' 0 ). These tasks are unsolved by the small model, and their expert traces

5

10

15

20

25

30

35

40

Training step

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

Test accuracy

BREAD

SFT

GRPO

GRPO w/ Expert trace

SFT + GRPO

(a) Test accuracy over training steps

5

10

15

20

25

30

35

40

Training step

0.2

0.4

0.6

0.8

1.0

Solve none ratio

GRPO

BREAD (before EAS)

BREAD (after EAS)

proved too complex for SFT to learn. The 500 samples were split into 80/20 train/test subsets. We then again used Qwen2.5-3B-Instruct as the base model and trained each method on this hard subset. The results are shown in Figure 8a. All baselines show little or no improvement on the test set after training on the hard questions. In contrast, BREAD achieves a clear upward performance with continued training, demonstrating that its partial expert guidance and branched rollout strategy can provide learnable information, even when standard SFT and vanilla GRPO fail. We argue that BREAD succeeds because it adaptively reduce problem difficulty and densifies the rewards. As shown in Figure 8b, BREAD sharply lowers the solve-none ratio, getting more informative samples and richer feedback. This enables BREAD to learn effectively even from very hard questions and complex reasoning traces.

BREAD improves sample efficiency during training. SLMs distilled via large-scale SFT can achieve strong reasoning capability, such as DeepSeek-R1-Distill [13]. However, the distillation pipeline is prohibitively expensive. For example, this model is trained with 800k samples from both reasoning and non-reasoning domains curated with DeepSeek-R1, containing 671 billion parameters. A single forward pass through such a huge model already costs more FLOPs than 25 RL training steps with 8 rollouts. The pipeline also needs expensive sample filtering and trace post-processing, like the heavyweight data-collection procedure in [24, 22]. What makes things worse is that, as illustrated in Table 1, each target model requires training with different samples, multiplying the cost.

In contrast, BREAD is far more sample-efficient, achieving comparable gains with a small fraction of the expert trace and without any heavyweight sample selection stage. To show this, we created two trace-budget-matched baselines, SFT(selected) and SFT(random) . For fair comparison, we first record the expert traces actually requested by BREAD via Episode Anchor Search ( EAS ). On NuminaMath-CoT, this corresponded to 36.7% of samples, and on the easier Math [15] dataset the fraction falls to 19.1% (during 300 training steps of Qwen-2.5-3B-Instruct). We then supervised fine-tune base models with the same number of traces To create SFT(selected) , we select the exact subset chosen by BREAD. To create SFT(random) , we use an equally sized randomly picked subset. Finally, we created SFT(full) , which uses all the expert traces, take a high cost. Each of the SFT(x) phases was followed by a GRPO phase. As the results in Table 2 show, BREAD outperforms nearly all the SFT(X)+GRPO baselines in terms of accuracy, and can even approach the performance of the expensive DeepSeek-R1-Distill model. Notably, while DeepSeek-R1-Distill gains from vast and diverse training data, BREAD achieves nearly comparable accuracy without requiring this data. We also observe that small models do not necessarily benefit from SFT with expert traces, as evidenced by the Qwen-2.5-1.5B-Instruct run on NuminaMath-CoT, where SFT(full)+GRPO has relatively low accuracy. Also, which samples are most useful for SFT is uncertain: on the MATH dataset, SFT(selected)+GRPO has higher accuracy than SFT(random)+GRPO with Qwen-3B-Instruct, but it is the opposite for Qwen-1.5B-Instruct.

5 Related Work

Supervised fine-tuning (SFT) and Reinforcement Learning (RL). Recently, there is a debate about whether SFT or RL can truly improve the reasoning ability of language models (LMs). With the emergence of [33, 13, 36], more and more reinforcement finetuned reasoning models demonstrate the importance of RL. in reasoning tasks. [6] prove that RL can enhance the reasoning ability of LMs

while SFT can only force LMs to memorize knowledge. However, [42] argues that the base model already has the reasoning ability while Reinforcement Learning with Verifiable Rewards (RLVR) barely increases the probability of correct reasoning trace. Furthermore, distillation works such as [13, 24] prove that SFT can help SLMs acquire reasoning capability comparable to the expert/teacher models. While the current popular pipeline to train a reasoning LM is SFT followed by RL, we argue the need for stronger integration of SFT and RL because, for hard questions, the expert solution might not be suitable for base models to learn while RL can struggle to discover even a single correct trace.

Efficient RL and Reasoning. While reasoning LMs become more powerful, computation is more demanding during the training and deployment procedure. Therefore, researchers recently paid more attention to efficient reasoning in both directions. Pivotal Token Search (PTS) [1] accelerates the Direct Preference Optimization (DPO) [27] training by identifying tokens in a language model generation that significantly impact the probability of success for the reasoning task. Following the memory efficiency but training time inefficiency introduced by GRPO [33, 13], many follow-up works improve the training convergence speed, including DAPO [41], CPPO [23], PODS [40] and DUMP [38]. [36, 43, 2] saves the token usage during inference by training with one or multiple levels of length penalty. Meanwhile, meta reinforcement fine-tuning (MRT) [26] makes inference token wasted less in meaningless reasoning steps by making the success rate steadily increase with the number of reasoning episodes. There are also orthogonal approaches to efficiency e.g. by forming cascades of small and large language models [5, 44, 14]. Compared with these previous works, we not only decrease the supervised signals during training by only introducing an expert solution when the current model cannot solve the current task with a probability relatively high enough, but guide the model with the expert hint to increase the RL training efficiency. which can provide a denser reward for speeding up the RL training procedure.

Related classical RL methods. DAgger [29] intermittently injects expert actions to correct agent behavior. Instead, BREAD adaptively inserts expert hints only when the agent fails and allows the model to complete the remaining of the reasoning trace, allowing for a natural curriculum. GoExplore [8] trains an agent that can solve all Atari games by branching from intermediate states, while BREAD branches out from expert traces. Methods like Hindsight Experience Replay (HER) [3] and potential-based reward shaping (PBRS) [25] densifies learning signals under sparse rewards through goal relabeling or external shaping functions, but BREAD can achieve a similar effect through hint-based rollout initialization. Finally, while prior works like Kickstarting [32] and Expert Iteration [4] explore teacher-student transfer via full trajectories or alternating control, BREAD explicitly focuses on sparse expert intervention with minimal demonstrations. Some earlier work, such as Model-Based Policy Optimization (MBPO) [18], propose to automatically update the predictive model during policy training and only learn through short rollouts, but it does not include the hint from expert traces, which will lead to the improvement limitation in LLM post-training domain.

6 Discussion and Limitations

Our work introduces BREAD as a principled algorithm to densify the reward signal using branched rollouts from the expert trace. BREAD significantly enhances sample complexity, training time, and eventual accuracy over employing GRPO. It is also theoretically well motivated and allows the model to overcome fundamental bottlenecks of SFT+GRPO. BREAD has also a few limitations: Firstly, we focus on SLMs and assume that there is a strong expert/teacher model to provide high-quality traces. This assumption can be weakened as we can use BREAD whenever successful traces are obtained and use standard RL otherwise. Secondly, it is possible that SLM may fail to obtain a reward signal even from partial traces of the teacher. In the extreme scenario, the student model can't even conclude even if the full solution by the expert is presented as it can't follow the argument at all. This shortcoming could potentially be addressed by training or prompting the expert model to generate simpler more digestible traces. We leave these as future directions.

References

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905 , 2024.

Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697 , 2025.

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems , 30, 2017.

Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems , 30, 2017.

Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176 , 2023.

Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161 , 2025.

Rick Durrett. Probability: theory and examples , volume 49. Cambridge university press, 2019.

Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature , 590(7847):580-586, 2021.

Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025.

William Feller. An introduction to probability theory and its applications, Volume 2 , volume 2. John Wiley & Sons, 1991.

Google. Gemini 2.0 flash thinking mode (gemini-2.0f lash-thinking-exp-1219), 2024. https: //cloud.google.com/vertex-ai/generative-ai/docs/thinking .

Google. Gemini 2.0 flash thinking mode ( gemini-2.0-flash-thinking-exp-1219 ), December 2024. Accessed 15 May 2025. 'Last updated 14 May 2025' on the page.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025.

Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. In The Twelfth International Conference on Learning Representations , 2024.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 , 2021.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720 , 2024.

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems , 32, 2019.

Koray Kavukcuoglu. Gemini 2.5: Our most intelligent ai model. https://blog.google/ technology/google-deepmind/gemini-model-thinking-updates-march-2025/ , March 2025. Accessed 15 May 2025.

Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https://huggingface.co/AI-MO/NuminaMath-CoT](https://github.com/ project-numina/aimo-progress-prize/blob/main/report/numina_dataset.pdf) , 2024.

Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. arXiv preprint arXiv:2502.12143 , 2025.

Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-based reasoning models. arXiv preprint arXiv:2503.22342 , 2025.

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393 , 2025.

Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml , volume 99, pages 278-287. Citeseer, 1999.

Yuxiao Qu, Matthew YR Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine-tuning. arXiv preprint arXiv:2503.07572 , 2025.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36:53728-53741, 2023.

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling , 2024.

Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011.

Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.00448 , 2023.

Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint arXiv:2502.17416 , 2025.

Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojciech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstarting deep reinforcement learning. arXiv preprint arXiv:1803.03835 , 2018.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024.

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256 , 2024.

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 , 2024.

Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 , 2025.

Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 , 2017.

Zhenting Wang, Guofeng Cui, Kun Wan, and Wentian Zhao. Dump: Automated distributionlevel curriculum learning for rl-based llm post-training. arXiv preprint arXiv:2504.09710 , 2025.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837, 2022.

Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter. Not all rollouts are useful: Downsampling rollouts in llm reinforcement learning. arXiv preprint arXiv:2504.13818 , 2025.

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 , 2025.

Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837 , 2025.

Xuechen Zhang, Zijian Huang, Chenchun Ni, Ziyang Xiong, Jiasi Chen, and Samet Oymak. Making small language models efficient reasoners: Intervention, supervision, reinforcement. arXiv preprint arXiv:2505.07961 , 2025.

Xuechen Zhang, Zijian Huang, Ege Onur Taga, Carlee Joe-Wong, Samet Oymak, and Jiasi Chen. Efficient contextual llm cascades through budget-constrained policy learning. Conference on Neural Information Processing Systems , 2024.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While '[Yes] " is generally preferable to '[No] ", it is perfectly acceptable to answer '[No] " provided a proper justification is given (e.g., 'error bars are not reported because it would be too computationally expensive" or 'we were unable to find the license for the dataset we used"). In general, answering '[No] " or '[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS Paper Checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We list our contribution in the end of the introduction section Section 1, which are supported by later methodology Section 1 and experiment sections Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation in the last section (Section 6).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In the Section 2.1, we define our theretical model and proves our claims in details.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe the main steps to reproduce our experiments including algorithm, dataset and base models in Section 1 and Section 4. We also provided details about algorithms and hyperparameters in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include our code in the spplementary materials and will also provide checkpoints to make sure our experiments are reproducible.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experiment details are stated in Section 4 and the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We use several large datasets with several base models to demonstrate the generalization ability of our methods.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the computation resources needed in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper does not contain anything that is harmful potentially.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification:We discuss the broader impact which is also potential positive ssocietal impacts in Section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not have negative impacts potentially.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We correctly cite all codebase and datasets that is used during our experiments in Section 4 and Appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We include our code for data generation, model training and inference and result aggregation in the supplementary zip file.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not relate to human subject study.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not relate to human subject study.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The development and design of our method does not involve LLMs.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

The Supplementary Material is organized as follows:

In Appendix A, we explain the calculation of estimated FLOPs (Appendix A.1) and additional details about the training and evaluation of BREAD (Appendix A.2).

In Appendix B, we show that SFT further cannot improve the performance of models (Appendix B.1), BREAD can reduce the number of rollouts required during the sampling stage in training (Appendix B.2), and the distribution of the step number of expert solutions (Appendix B.3).

In Appendix C, we explain the baseline GRPO w/ Expert Trace in detail and compare it to RL with an SFT loss.

In Appendix D, we present a detailed formulation of the Markov model introduced in Sections 2.1 and 2.2, and provide concise proof of related theorems.

Algorithm 2 BREAD : GRPO with Branch Rollouts and Expert Anchors (full version)

A Experiment details

A.1 FLOPs

We compute the estimate FLOPS following [35, 16, 30]. The supervised finetuning, which include one forward and one backward phase, people use a common approximation 6 ND [16], and for inference, which include only one forward phase, people always use 2 ND [30]. Here N represents model parameters, D is the total token count processed in that pass. So a forward phase takes 2 ND while a backward phase take 4 ND . For estimation, we define the average length of a single question in one inference time as D sample, so D ' D sample ˆ n rollout for RL and D ' D sample for SFT. For estimation, we define the average length of a single question in one inference time as D sample , so D ' D sample ˆ n rollout for RL and D ' D sample for SFT. GRPO with eight rollouts, n rollout ' 8 including eight forward and eight backward phase needs 6 ˆ 8 ˆ ND rollout. BREAD will take more forward pass to do the binary search, so we use 6 ˚ 8 ˚ ND rollout ` 4 ND additional . GRPO w/ Expert trace includes including eight forward and nine backward phase, so we use 6 ˆ 8 ˆ ND rollout ` 4 ˆ ND rollout . We estimate D with the average length of all expert trace in the training dataset. We truly record the additional number of generation D additional ' D rollout ˆ n additional_rollout. Notably, the generation length of BREAD is always shorter than expert trace and vanilla GRPO. So our BREAD can reduce even more computation resource compare to 75% shown in Figure 7b.

A.2 Experiment Setting Details for BREAD

In addition to Section 4, we list additional details of our experiments here. During training, we set max_prompt_length ' 2048 for MATH experiments and max_prompt_length ' 4096 for NuminaMath-CoT experiments. For both training and evaluation, we set max_response_length ' 4096 for MATH experiments and max_response_length ' 8192 for NuminaMath-CoT experiments. During training, we set the number of rollouts as 8. We used a low-variance KL divergence and set the coefficient for KL divergence as 0.001. We set the temperature at 0.6 for both training and evaluation.

For the implementation details of the episode splitting for the expert trace during training, we split the expert trace with sentences separated by '. ' or '\n' for MATH and paragraphs separated by '\n\n' for NuminaMath-CoT. Note that the expert traces can be either human provided solutions or correct solutions provided by larger expert models, and the splitting method here can also be changed to split according to a specific keyword list. After splitting, we aggregate the traces into 10 episodes evenly for all expert traces. The reason why we implement this way is that we want to make sure that the number of episodes of the expert traces is not too large, which can guarantee that the EAS step does not take too much time.

During training, assume we have a question Q and an expert hint ρ ( or without ρ during inference), the template of the prompt is as follows:

{'content': '
system\nYou are a helpful assistant. You first thinks about the reasoning process in the mind and then provides the user with the answer.
\n
user\n{ Q,ρ } Show your work in <think> <\think> tags. And return the final answer within \\boxed{}.
\n
assistant\nLet me solve this step by step.\n<think>', 'role': 'user'}

For the hardware requirements, all of experiments are done with 8 L40S 40GB GPUs except the training starting from Qwen2.5-3B-Instruct as the base model, which requires 8 80G H100 GPUs.

B Additional Experiments

B.1 SFT training saturates and does not help after a while

Supplement to Figure 7a. As shown in Figure 9, training SFT for more iterations does not further improve test accuracy. In other words, we already use the model that SFT can achieve. For a fair comparison, we therefore use the 300-step checkpoint as the starting point for SFT+GRPO .

0

50

100

150

200

250

300

350

400

Training step

0.49

0.50

0.51

0.52

0.53

0.54

Test accuracy

SFT

B.2 BREAD can reduce the number of rollouts needed

Another straightforward way to reduce training cost is to reduce the number of rollouts, since FLOPs scale linearly with that count. However, for vanilla GRPO, fewer rollouts lead to lower accuracy, whereas BREAD maintains its performance even as the number of rollouts decreases. The result is shown in Figure 10. As we have shown in the main body, when the task is so difficult that the base policy rarely produces correct traces, vanilla GRPO fails. To study rollout efficiency under conditions where GRPO can learn, we moved to the MATH dataset with a Qwen-2.5-3B-Instruct base model. With eight rollouts, GRPO does increase accuracy. However, performance will decrease if the rollout budget is reduced from 8 to 5. In contrast, BREAD reaches the same accuracy with just five rollouts, cutting training FLOPs while preserving performance.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Estimate FLOPs (B)

1e9

0.62

0.63

0.64

0.65

0.66

0.67

0.68

0.69

Test accuracy

BREAD(5 rollouts)

GRPO(5 rollouts)

GRPO(8 rollouts)

B.3 Episode details

As described in Appendix A.2, we plot the distribution of the number of steps for our two datasets before episode aggregation. As shown in Figure 11, we can see that that most expert traces contain less than 20 steps, while few of them contain much more steps, which may slow down the training sif there ised if there is no episode aggregation.

B.4 BREAD can reduce the number of rollouts needed

We also conduct experiments with LLaMA3.2-3b-Instruct model. The results are shown in Table 3. These experiments support that the method generalizes beyond the Qwen family and is not modelspecific, and the effectiveness is more significant when the task is harder.

B.5 BREAD compare to other methods.

Regarding recent works, we would like to highlight that our focus is on challenging tasks where existing methods like DAPO [41] are not specifically designed to perform well. So, these tasks are

0

50

100

150

200

Number of Solution Steps

0

1000

2000

3000

4000

Frequency

Distribution of Solution Steps

(a) Distribution plot of MATH solution step numbers.

0

20

40

60

80

100

Number of Solution Steps

0

25000

50000

75000

100000

125000

150000

175000

200000

Frequency

Distribution of Solution Steps

(b) Distribution plot of NuminaMath-CoT solution step numbers.

typically not included in the evaluation benchmarks of DAPO and similar methods. For example we train and evaluate DAPO on the same difficult dataset used in Figure 8, under the same settings. We observed that DAPO shows little improvement. This aligns with its design: DAPO's dynamic sampling strategy tends to filter out hard questions rather than try to solve them, making it less useful for the types of tasks we target with BREAD. The results are shown in Table 4.

C GRPO w/ Expert Trace Details and its Connection with RL with SFT Loss

The baseline GRPO w/ Expert Trace (GRPO-ET) generally follows the same procedure as the standard GRPO. Instead, it enforces one of the G rollouts as the expert trace. Therefore, the objective function is

where

All notations in Equations (3) and (4) are the same as [13], while S represents the expert trace, and all variables whose subscript contains S represent the corresponding variables.

Here, we can see a clear connection between GRPO w/ Expert Trace and GRPO (rollout number is G ´ 1 ) with an SFT loss. Suppose that we want to deploy GRPO while adding the SFT entropy loss to the standard GRPO loss, the objective function is

where

There are 3 main difference between Equation (3) and Equation (5):

A coefficient for the expert trace loss 1 G | o S |

A KL divergence term ´ βD KL p π θ || π ref q

An Advantage term min ´ r i,t p θ q ˆ A S,t , clip p r i,t p θ q , 1 ´ ε, 1 ` ε q ˆ A S,t ¯

r i,t p θ q ˆ A S,t , which is exactly the entropy loss with a coefficient ˆ A S,t if we replace π old p o i,t | q, o i ă t q in r i,t p θ q with the one hot embedding of the expert trace tokens. Therefore, GRPO w/ Expert Trace can not only have better training consistency, but can also assign different credits according to the token probability ratio between the old and the current policy.

Algorithm 3 GRPO w/ Expert Trace (GRPO-ET)

D Supporting Material for Section 2.1: Proofs and Experimental Details

D.1 Experimental Details

Recall that we consider a Markov chain with K ` 1 states (indexed by 0 , 1 , 2 , ¨ ¨ ¨ , K ) and a p K ` 1 q ˆ p K ` 1 q transition matrix. We assume that, for the expert/large model, all p i Ñ j q transitions are learnable in the transition matrix. However, for a small/student model, not all p i Ñ j q transitions

are learnable and we denote the learnable state transitions by P Ă r K ` 1 s ˆ r K ` 1 s where r K s ' t 0 , 1 , 2 , . . . , K u .

We first introduce the following definition of pretrained small model utilized in our experiments.

Definition 1 (Pretrained Small Model) A pretrained small model is defined as a Markov model M ' p S , P q , where S ' t 0 , 1 , 2 , . . . , K u is the state space and P P R p K ` 1 qˆp K ` 1 q is the transition matrix. The model satisfies the following properties:

Let d ě 1 denote the maximum allowed jump between states. The set of learnable state transitions P satisfies:

For some ϵ ! 1 { d , the transition probabilities satisfy:

In the following, we describe the BREAD algorithm for the Markov model. Our algorithm is in line with the theory presented in Section 2.2 and incrementally learns a solution by optimizing the local transition dynamics starting from the right end of the expert trace.

Algorithm 4 BREAD_Markov

Experimental settings for Figure 3: We conduct experiments by fine-tuning a Markov chain model (cf. Definition 1) using three different methods: SFT, GRPO, and BREAD. To introduce additional randomness in our experiments, for each state i P r K s , we randomly select one of its connected states (e.g., among i ´ d, . . . , i, i ` d states) and assign it the highest transition probability of 1 ´ Θ p ϵ q , instead of always assigning the highest probability to P p i Ñ i q . Due to symmetricity, this is not expected to change the fundamental behavior of the algorithms but provides more variability. In both Figs. 3a and 3b, d ' 2 , T max ' 2 ¨ K and SFT is infeasible with expert jump size of 3 as we set T ' K { 3 . In Fig. 3a, we fix K ' 30 and vary ϵ P t 0 . 01 , 0 . 025 , 0 . 05 u . In contrast, in Fig. 3b, the ϵ ' 0 . 05 is remained unchanged and the number of states varies in K P t 30 , 50 , 100 u . Each experiments is trained for 10000 iterations with 1000 trajectories sampled with each iteration.

D.2 Proofs in Section 2.2

D.2.1 Proof of Lemma 1

Since the expert model always jumps two or more whereas the student is restricted to a jump size of at most one, the SFT phase (specifically maximizing log-likehood using expert trace) will have no impact on the student model. This is because the supervised loss, induced by the negative loglikelihood, is not optimizable beyond infinity due to 0 transition probabilities assigned between x t Ñ x t ` 1 . To conclude the proof, we show that, no trajectory sampled during the RL phase will receive a reward with high probability. Since the SFT phase has no impact on the base student model, the student still follows a symmetric random walk. Let p x s i q L i ' 1 denote this random walk which is a sum of IID Rademacher variables. The classical concentration bounds on the hitting-time of random walks state [7, 10]

Thus, with access to N ď e K 2 { 4 L samples, applying union bound, we guarantee that no trace generated by the student model receives a reward.

D.2.2 Proof of Theorem 1

We first recall the following lemma on random walk hitting times.

Lemma 2 (Lower bound on random walk hitting time, [10]) If X i is a symmetric random walk initialized at 0 , we have that P p max 1 ď i ď L X i ě K q ě 1 ´ 0 . 8 K { ? L .

To proceed, the proof will be done inductively. Suppose for all rounds i ă τ , the followings hold:

Each round requires at most t traces to achieve success with probability at least 1 ´ e ´ t .

All sub-traces generated by the small model at round i (starting from the expert's x T ´ i ) that are stitched to S i ´ 1 (trace constructed so far) is at most length L { T .

To complete the induction, we consider the new round τ and prove the two statements above. At round τ , we start from State x T ´ τ and aim to reach a State within S τ ´ 1 . This process will follow a symmetric random walk initialized at x T ´ τ . By design, x T ´ τ ` 1 is within S τ ´ 1 . As a result, the trajectory length of the subtask is upper bounded by the time it takes for the random walk to reach from x T ´ τ to x T ´ τ ` 1 . Consequently, the total trajectory length is upper bounded by the sum over all subtasks. Since each subtask is independent and identically distributed, we focus on analyzing a single subtask with maximal trace length L { T . Recall from the assumptions in Section 2.2 that | x T ´ τ ´ x T ´ τ ` 1 | ď c ¨ K { T . To this aim, we apply Lemma 2 which ensures that, probability of hitting within L { T trace length is at least 1 ´ 0 . 8 c ¨ K { T ? L { T ' 1 ´ 0 . 8 c ¨ K ? LT . Since L ě 5 c 2 K 2 { T , probability of failure for each trace is ď 1 { e . Over t IID traces, probability of success at any round i is at least 1 ´ e ´ t . Aggregating these events, we find that the probability of success over T rounds is at least 1 ´ Te ´ t given t BREAD rollouts at each round.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores reasoning with (relatively) small language models. The authors propose BREAD, a method that combines RL and SFT via a simple strategy: if the policy model cannot solve the given question, it is given a hint via a partial expert trajectory on this prompt. More precisely, BREAD does a binary search to figure out the length of the partial trajectory that leads to the student model to achieve pass rates within a pre-defined range. The authors show that this simple procedure outperforms combinations of supervised finetuning (SFT) and reinforcement learning (GRPO).\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths \n\n**S1.** The paper is clear and easy to follow.\n\n**S2.** The proposed method is very simple and makes intuitive sense. It is easy to implement, and it has a very clear intuition. It can probably also be generalized to other kinds of \"hints\" beyond partial trajectories.\n\n**S3.** Section 3.1 presents a toy model where the improvement from BREAD can be proved theoretically.\n\n**S4.** The empirical evaluation shows consistent improvements relative to other combinations of SFT and GRPO.\n\n# Weaknesses\n\n**W1.** If I understand correctly, the policy is updated by training on all the tokens, including the expert tokens which were not generated by the policy. If this is correct, I believe that the policy update is no longer a policy gradient update, since the trajectories can be significantly off-policy. \n\n**W2 (minor).** Please say explicitly in the Policy Update section of Section 3 that the update (1) is the standard GRPO update. Otherwise, it is not clear where this objective comes from.\n\n**W3.** The presentation often does not provide all the details of the results. For example, in Section 3.2 RL-only paragraph, it is unclear what model is used, what dataset, etc. Without these details, the information on how many problems are unsolved in each batch seems pointless. In the SFT-only section, it is stated that the SFT is done on \"1000 difficult questions\". What are those questions? Also, it is unclear what are S1K and S1K-1.1. \n\n**W4.** Similarly, it is unclear what exactly is shown in Figure 5b. It is said that RL is using 40% hints, but then there is a full range of values for the \"\"Hint ratio\" in the plot, and there is an improvement for 0% hint ratio. Please clarify what exactly is shown.\n\n**W5.** In an ideal world, it would be great to see a comparison with GRPO and possibly newer methods like DAPO [1] in a standardized setting such that would be possible to directly compare to existing results in the literature (e.g. Qwen2.5-32B on AIME). I realize that this may be computationally challenging, and possibly 32B is already not a \"small\" LM. I note that the authors provide a comparison to DeepSeek-Distill in Table 2.\n\n[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale\n\nQUESTIONS:\n**Q1.** When you do policy updates, do you train on all tokens, including the expert tokens? Or only the completion tokens generated by the student? Is it true that the method is no longer a policy gradient method, as the trajectories are off-policy?\n\n**Q2.** Do I understand correctly that all plots in Fig 6 and Fig 7 are test pass rates, where the test set is separate from the train set, and you are not training on the test set?\n\n**Q3.** Same question for Table 2: for these results, do you use any transcripts on MATH and Numina? What are the questions that you distill on?\n\n**Q4.** Do I understand correctly that in your SFT experiments you use a much lower amount of SFT compared to the industry methods such as DeepSeek-Distill? What is the size of the \"full\" dataset in your experiments?\n\nLIMITATIONS:\nLimitations adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Branched Rollouts from Expert Anchors, a novel approach that combines supervised fine-tuning and reinforcement learning to enhance reasoning in small language models. The authors identify the limitations of traditional SFT + RL training, particularly for SLMs struggling to learn from complex expert traces and sparse rewards. BREAD addresses these issues by incorporating partial expert guidance and branched rollouts, allowing for more efficient learning. The method reduces the need for extensive expert traces, accelerates training, and improves model performance, especially on challenging tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The BREAD framework effectively bridges the gap between supervised fine-tuning and reinforcement learning , addressing the issues of sparse rewards and weak initialization in small language models , ensuring more efficient learning.\n2. BREAD demonstrates substantial improvements in training efficiency, reducing the number of required rollouts and thus lowering computational costs while achieving higher accuracy than traditional GRPO and SFT methods.\n3. BREAD outperforms existing methods, including GRPO and SFT+RL, on challenging tasks, especially those that are unsolvable using standard SFT+RL, by providing expert guidance and adaptive curriculum learning.\n4. The paper offers a thorough theoretical analysis, formalizing the limitations of SFT+RL and justifying the design of BREAD, supported by mathematical insights into how expert traces can enhance model performance.\n5. Experimental results consistently show that BREAD not only achieves superior accuracy but also accelerates training time.\n\nWeaknesses:\n\n1. Section 3.2 on the Deficiencies of SFT-only: a) The authors have not provided sufficient evidence to demonstrate that the performance degradation observed with SFT-only is due to the model's insufficient learning capability rather than biases present in the SFT data. b) The authors have concluded that SFT-only leads to performance degradation based on a limited set of SFT datasets (only two) and a narrow range of SFT methods (two methods). I believe these experiments are insufficient to robustly prove the claim that SFT-only causes performance degradation.\n\n2. Bias in the Datasets Used in Table 2: The datasets used in Table 2 are biased, as they are all mathematics-related, which may not adequately demonstrate the generalizability of the proposed method. Validating the method on other reasoning datasets would provide a better assessment of its broader applicability.\n\nQUESTIONS:\n1. Have the authors tested whether models with 7B parameters or larger, when using the same SFT and RL methods as described in this paper to improve model COT capabilities, experience the issues of insufficient RL rewards or performance degradation caused by the SFT-only method mentioned in the paper?\n\n2. Does the length of the COT reasoning path affect the training algorithm proposed in this paper? Typically, how many reasoning steps are involved in the expert traces constructed in this paper?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a reinforcement learning (RL)-based approach to address the challenges of insufficient exploration by small models on difficult tasks and distributional shift caused by instruction tuning. The authors leverage partial expert demonstrations as starting points to guide model continuation and optimize exploration. Experimental results indicate that the proposed method improves the performance of reasoning tasks on the Qwen series models. However, the paper exhibits significant shortcomings in terms of methodological novelty, experimental design, and generalizability.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The problem addressed in this paper is important. On challenging tasks, models often fail to explore correct solution paths, making training difficult.\n2. The paper is clearly written and easy to understand.\n\n### Weaknesses\n\n1. Lack of originality in the core idea: The novelty of the proposed method is questionable. The two challenges identified by the authors—difficulty in exploring correct paths on hard problems and distributional shift caused by SFT—are not unique to small models; these issues also exist for larger models.\nMoreover, the idea of using partial demonstrations as starting points to facilitate exploration has already been proposed in recent works, such as R^3 [1], which use expert demonstrations to construct curricula for better exploration. \nEarlier work can be traced back to OpenAI’s RL research [3]. R^3 also utilizes demonstration-based continuation and validates its effectiveness on mathematical tasks. The main differences between this paper and existing work appear to be the choice of RL algorithm (GRPO vs. PPO) and the focus on small models in this paper. Additionally, relevant prior work is not adequately cited. The authors should clarify the novelty of their proposed approach.\n\n2. Experiments are conducted only on the Qwen series models, which have been shown to exhibit the “performance improvement under random rewards” phenomenon. For example, Shao et al. [4] demonstrated that Qwen models show significantly greater gains under spurious rewards compared to other models, raising concerns about the generalizability of the results. The authors should include evaluations on additional small models to validate the universality of their method.\n\n3. The two core challenges identified by the authors do not appear to be specific to small models; larger models are also affected. I am curious about the effectiveness of the proposed approach on larger models.\n\n4. Lack of comparison with other methods: The experimental section does not include comparisons with other approaches aimed at improving SLM reasoning capabilities, such as R^3. The authors should compare their method against these baselines to better highlight its advantages.\n\nReferences:\n\n[1] Xi, Z., Chen, W., Hong, B., Jin, S., Zheng, R., He, W., Ding, Y., Liu, S., Guo, X., Wang, J., & Guo, H. (2024, July). Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning. In International Conference on Machine Learning (pp. 54030-54048). PMLR.\n\n[3] Florensa, Carlos, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. \"Reverse curriculum generation for reinforcement learning.\" In Conference on robot learning, pp. 482-495. PMLR, 2017.\n\n[4] Shao, Rulin, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du et al. \"Spurious Rewards: Rethinking Training Signals in RLVR.\" arXiv preprint arXiv:2506.10947 (2025).\n\nQUESTIONS:\nPlease address all the issues mentioned in the Weaknesses section as thoroughly as possible.\n\nLIMITATIONS:\nSome of the limitations faced by this paper have been discussed in the Weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes BREAD: a method for improving GRPO training of small language models on hard reasoning problems using backplay.

STRENGTHS AND WEAKNESSES:
**Strengths:**

- The paper is clear and easy to follow
- The method demonstrates strong results for SLMs on hard problem sets

**Weaknesses**:

- BREAD is essentially an application of backplay [1,2] to reasoning problems. Despite this, no references on backplay are cited or discussed.
- Evaluation is done only on the Qwen2.5 model series. A more thorough evaluation investigating what characteristics of a model necessitate backplay supervision would be interesting.

Missing references:
- [1] Salimans, Tim and Richard J. Chen. “Learning Montezuma's Revenge from a Single Demonstration.” ArXiv abs/1812.03381 (2018): n. pag.
- [2] Resnick, Cinjon et al. “Backplay: "Man muss immer umkehren".” ArXiv abs/1807.06919 (2018): n. pag.

QUESTIONS:
- Is there a correlation between a model's averaged pass@1 accuracy on a question and the amount of hint that needs to be shown before reaching the correct answer? Could this be leveraged to warm-start the binary search?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces new GRPO-like algorithms for training Small Reasoning models. The authors leverage the observation that small models required longer traces to match larger models, hence, SFT might hurt performance. Moreover, because GRPO relies on having a good initialization from SFT, subsequent RL training can more likely fail.  Hence, the authors designed BREAD, an algorithm that leverages the expert demonstrations (reasoning traces) to adaptively choose how much of the expert trace to prepend to achieve a threshold success rate of the sampled group. This allows the model to obtain a kind of curriculum that improves RL training and bypasses the SFT training phase. The empirical evaluation successfully shows that their algorithm improves over all baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is very well written and provides a clear exposition of the motivation and design of BREAD;\n- The authors include a reasonable set of baselines, including naive variations of GRPO, and show that their algorithm improves over all of them.\n- The authors provide experiments that allows to further understand the impact of their algorithm on compute, sample efficiency and effect of the hardness of the tasks/questions\n\nWeaknesses:\n- The toy model builds a nice intuition for the problem motivation but it may be too limited. Though I understand this may not be the main contribution of this work.\n- In terms of the BREAD objective, I believe there might be a problem with off-policy. See questions.\n\nQUESTIONS:\n- I appreciate the toy model provided that serves to motivate the algorithm design. I’m mostly interested in understanding how much we can trust it. Where do you get the motivation of modelling the limited capacity of small models by limiting the learnable graphs in the Markov chain? \n- Can we apply BREAD for general LLM reasoning or does it only apply to SLMs? Do you expect that without the bottleneck of the model capacity, BREAD could still provide an advantage over GRPO?\n- As I understand it, the expert guidance is incorporated by modifying the search process. Effectively changing the policy \\pi that is being executed, doesn’t this introduce off-policy issues?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}