Reinforcement Learning Teachers of Test Time Scaling
Authors: Edoardo Cetin
Tianyu Zhao
Yujin Tang
Venue: NeurIPS 2025 poster ()
Keywords: RL
reasoning
LLMs
language
math
reinforcement learning
language models
Primary area: deep_learning
TL;DR: We introduce a new class of Reinforcement Learned Teachers trained to provide effective reasoning traces for downstream distillation, yielding more effective data for distillation and cold-starting than orders of magnitude larger reasoning LMs.

Abstract: Training reasoning language models (LMs) with reinforcement learning (RL) for one-hot correctness inherently relies on the LM being able to explore and solve its task with some chance at initialization. Furthermore, a key use case of reasoning LMs is to act as teachers for distilling new students and cold-starting future RL iterations rather than being deployed themselves. From these considerations, we introduce a new framework that avoids RL's exploration challenge by training a new class of Reinforcement-Learned Teachers (RLTs) focused on yielding the most effective downstream distillation. RLTs are prompted with both the question and solution to each problem, and tasked to simply "connect-the-dots" with detailed explanations tailored for their students. We train RLTs with dense rewards obtained by feeding each explanation to the student and testing its understanding of the problem's solution. In practice, the raw outputs of a 7B RLT provide higher final performance on competition and graduate-level tasks than existing distillation and cold-starting pipelines that collect and postprocess the reasoning traces of orders of magnitude larger LMs. Furthermore, RLTs maintain their effectiveness when training larger students and when applied zero-shot to out-of-distribution tasks, unlocking new levels of efficiency and re-usability for the RL reasoning framework.

Reinforcement Learning Teachers of Test Time Scaling
Edoardo Cetin, Tianyu Zhao, Yujin Tang
Sakana AI, Japan
{edo, wwwp:/.gyyaiinatnaaaaaaaaa
Abstract
Training reasoning language models (LMs) with reinforcement learning (RL) for one-hot correctness inherently relies on the LM being able to explore and solve its task with some chance initialization. Furthermore, a key use case of reasoning LMs is to act as teachers for distilling new students and cold-starting future RL iterations rather than being deployed themselves. From these considerations, we introduce a new framework that avoids RL's exploration challenge by training a new class of sawaforremeettLearnee Teachers (RLTs) focused on yielding the most effective downstream distillation. RLTs are prompted with both the question and solution each problem, and tasked simply cconnect-the-dott with detailed explanations tailored for their students. We train RLTs with dense rewards obtained by feeding each explanation to the student and testing its understanding of the problem's 's solution. In practice, the raw outputs of 7B RLT provide higher final performance on competition and graduate-level tasks than existing distillation and cold-starting pipelines that collect and postprocess the reasoning traces of orders of magnitude larger LMs. Furthermore, RLTS maintain their effectiveness when training larger students and when applied zero-shot to -ittr--distriittionn tasks, unlocking new levels of efficiency and re-usability for the RL reasoning framework.
gidub..ommSaknnAARRIT
Introduction
50.5
75
Exploration is of the critical chal- lenges in reinforcement learning (RL) and has been a core focus of its lit- erature ([+3]. Sparse rewards cannot yield any learning signal unless the agent is already capable of solving the given task at initialization. With the rise of RL for open-ended reason- ing (RL reasoning) inducing a new form of language model (LM) scal- ing (46 beyond woomp-enginneriig and search 77 [) exploration has re- emerged as a key challenge. A canon- ical motivation for RL is the poten- tial to bootstrap from partial solutions, guided by informative reward func- tion, and learn entirely new tasks from scratch. However, the nature of one- hot correctness rewards used in the RL
69.5
Ovarall performance
48
70
Raw 78 RUT traces
66.4
46
65
44
60
42
60
53.2
38
50
Bespoke-7B R8I1
RUF-78
RET AL
Qwen-332 328
Soy 328 (Ow())
$2.326 (Gemin))
Bespola 328 1821
RLF 320
Owenn-
Bespake
Figure RLTs provide better student distillation and RL cold-starts than orders of magnitude larger LMs across com- petition and graduate-level tasks (AIME, MATH, GPQA). This holds when distilling students of the same size (Left) and also 32B students, much larger than the RLT itself (Right).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

reasoning framework fails to provide a dense form of guidance to assess relative progress, focusing instead on reinforeing correct responses in the initial model's pool of pass-at-k attempts without true extrapolation beyond the LM's initial latent abilities [9] As a result, mostly large, already- capable models have been shown to improve consistently beyond cheaper and simpler supervised optimization 4
Due to this fundamental limitation, coupled with RL's training instability, distillation has emerged as another ubiquitous component of current reasoning paradigms. In this case, the test-time role of LMs trained with RL is act as a teacher providing instructive reasoning traces for a student to solve new problems. This teacher-student paradigm is widely adopted both to train smaller, less-capable models [6] [0] and even to cold-start future RL iterations for better final convergence with the teacher's s own initial checkpoint acting as the student [ However, the problem-soly soolemm--ovvigggg skills reinforced by correctness- based rewards have been shown not be entirely aligned with the goal of downstream distillation [12]. To account for this mismatch, current pipelines significantly rely on heuristic-driven postprocessing of the teacher's outputs for effective student transfer
12 [3]
Based on these considerations, we propose a framework that avoids RL's exploration challenge with new class of specialized Reinforeement-Learnee Teachers (RLTs) trained specifically to yield effective downstream distillation. Our main intuition is simple: the ability of real-world teachers is not measured by whether they can come up on their own with complex theorems, proofs, or answers from scratch. Instead, what matters is their ability to make use of readily available solutions and devise instructive explanations for their students. Thus, we depart from the traditional RL reasoning framework, tasking model to first think and then come up with a new solution for the first time. Instead, RLTS are tasked with the easier problem of providing an effective explanation with the problem's s solution already given within their prompt. We train RLTs with dense and informative rewards obtained from the student's log probabilities. These rewards provide an intuitive measure of the student's is understanding of each problem's 's ground-truth solution following the teacher's s explanations, and the interpretability of the logical leaps in the explanations themselves.
By distilling students from the raw outputs of a lightweight RLT with 7B parameters, we demonstrate significantly higher performance than using existing pipelines relying on reasoning LMs with orders of magnitude more parameters (Figure D. We show our framework provides superior benefits even when distilling the RLTT's explanation to train larger 32B students and to cold-start traditional RI. optimization. Furthermore, we showcase how RLTs can be transferred zero-shot to new domains and still produce effective distillation datasets that yield yet better final students than direct RL with access to the task's reward. Overall, these results highlight the potential of our new method for overcoming the large costs of RL by focusing on stronger, smaller, and highly reusable specialized teachers, while removing the current reliance on expensive and heuristic-driven distillation pipelines.
We share our code and pretrained checkpoints]] to facilitate future research in RL reasoning and distillation. In summary, our main contributions are threefold:
We introduce the RLT framework, tackling exploration with a simpler dense-reward that aligns the objective of RL training to providing effective downstream student distillation.
We show how distilling the raw outputs of 7B RLT directly outperforms training students with carefully postprocessed reasoning traces from orders magnitude larger LMs.
We demonstrate that RLTs also allow for better cold-starts for traditional RL, effective distillation to larger students, and even zero-shot transfer to new reasoning domains.
2 Inducing reasoning in language models
2.1 Reinforcement learning
The RL post-training recipe for inducing reasoning behavior was recently popularized by the DeepSeek R1 line of work [14] 4]. By fine-tuning on a dataset of questions D ...,....9. gN3 with verifiable solutions {s1...., 8N , Guo et al. [4] show effective "reasoning" behavior emerges out of a 671 B--parmmeter LM [66, significantly pushing its performance on challenging math and coding tasks. Their training is conducted with GRPO (15), an online RI. algorithm that foregoes

---

RL reasoning format
RL teaching format
System
System
Your role as an assistant Involves thoroughly exploring questions through systematic long thinking process..
Your role teacher Involves thoroughly exploring questions and solutions systematic long explanations..
User
User
Retur your final response within lboxed0. The domain of the function Salx) xx4 4x*2 4$ $Ea linftyJS. What is the range?
Return your final response within lbaxed(). The domain of the function Sqfx) xx44 + 4x*2 + 4$ is $j0, UnftyS. What is the range?
<| +bbggnn of solution] >>
Expected completion
the range of V. q(0) V la V[boxeed[[4, Vnfty))).
Assistant
<| [ndd_ of soluuiion
< beggtn of_ thought >>
Expected completion
Okey, so nned to find the of the function q01) x" 4x" 4,
lend_ of_ =fthhought]
Assistant
lbegin_of of f_solution]
< beginn of [explanattion]
the range of V[ q(1) is V(bbxeeff44, linfty)HU.
Okay, need find the range the function q(x) =x* 4xt </end_ of eexplanation]
<|end_ Sill_oo__eee of_solution]
Figure 2: Left: RL format asking an LM think and solve hard problems from scratch. Right: RLT format asking an LM to produce instructive step-by-ster explanations given access to the solutions.
the use of a critic model with simple Monte-Carlo value estimate. GRPO prompts the LM T to produce set of G >>> 1 "grouped" outputs 01, ... for each sampled question q E D, optimizing:
1
((0) = Eg (<v(1a)()
E G 1=1
(A, - 9 ( Terr)
(1)
Here, the "advantages" A, are obtained by normalizing each output's reward r, within each group:
A Т: mean(fr)......
std((rr....,,at)
key component of their design is a system prompt that asks the LM to decompose each generated output o; into two separate formatted sections separated by <think> and <solution> tags, denoted to. and Sol- This structure is forced by assigning rewards r; = -1 to unformatted completions, тi -05 to wrong but formatted completions, and r; = 1 only to correct and formatted completions. Training with this strategy, Guo et al. [4] show the LM's completion length gradually grows with reflection, verification, and self-correction steps emerging, mirroring human chain-of-thoughle
2.2 Supervised distillation
Supervised distillation is another critical step used to train recent reasoning models complement RL's shortcomings. For any online RL objective like Equation | to avoid collapse, the model must already possess a non-trivial chance of producing correct responses with non-zero gradients at initialization. This defining property makes the RL objective much less applicable than cross- entropy objectives that always include the correct response's 's information in the model's gradients. As a consequence of this dichotomy, distilling the reasoning traces of large RL-trained LMs with supervised learning is not only cheaper but has also been shown be notably more effective than performing RL itself for inducing reasoning in smaller, less-capable models 44
6 17 12 10]. Furthermore, RL appears prone to instabilities and output degradation, especially during extended training sessions. Due to this second limitation, DeepSeek R1 and several other models [4] []] perform RL training over multiple iterations. This is done by using the RL-trained models at the end of each intermediate iteration only to, once again, collect distillation datasets used for "cold-starting" their original initial checkpoint and obtain a stronger initialization point for the next RL itcration.
Constructing a dataset of distillation prompts DSD = {d1..... dN} } involves using the RL-trained LM Ty with reasoning system prompt to answer a corpus of verifiable questions, which can be chosen with several heuristics 6 17 12 [0]. The LM's output reasoning traces for each question (l()) are then filtered by comparing them with the ground-truth solutions ensure their correctness. Commonly, these reasoning traces are also post post-processed via additional "manual" steps of refinements, such as asking other closed-sourced LMs to remove grammatical issues and refactor the reasoning steps into nicer, consistent format. In fact, Li et al. [12 even argues that the structure and format of the thinking data a critical component to make weaker models actually understand and learn how to reason from distillation, potentially even more important than correctness itself.
3

---

3 Reinforcement learning teachers
3.1 The implications of training teacher models as students
We distinguish two separate training and inference "roles" that can be performed by LMs in the modern reasoning framework. As detailed in Section 2 after RL training, LMs To are often not deployed themselves but rather used to obtain reasoning distillation datasets for fine-tuning weaker models and cold-starting future RL iterations. Thus, these models can be effectively seen as teachers, providing explanations for future student models n to learn from.
This teacher-studee paradigm highlights a potential mismatch between the objective used for RL training and the teacher's test-time role. In traditional settings, teachers are trained with sparse correctness rewards to improve their ability to solve hard problems from scratch. This objective not only precludes the applicability of RL. training for tasks beyond the base model's original capabilities, due to its inherent exploration challenge, but is also not aligned with the teacher's s actual end goal: producing reasoning traces from which students T, can learn the necessary skills to derive correct solutions themselves. Based on these considerations, we propose a different training framework for RL reasoning models to be deployed as teachers that avoids RL's exploration challenge and breaks this objective mismatch. Our framework comprises much easier task formulation, a dense reward objective, and a carefully designed training recipe, allowing us to learn a new class of specialized Reinforcement Learned Teachers (RLTs).
RLT format
Distillation format
3.2 Aligning the task of teacher models
Quastion tokens (t))
Question takona (4))
Masked
In the traditional RL paradigm, the solution 8i to each problem is never explicitly provided to the model and is only employed for checking the correctness of the corresponding solutions within the LM's completions sp,, Precluding direct access or information about the solutions aligns training with the test-time objective of solving entirely new test problems from scratch, but is precisely what makes exploration chal- lenging, as the model receives no gradients until its first successful attempt. Our key observation, however, is that the test-time "teaching" objec- tive of producing effective distillation datasets DSD for questions with known solutions, can be greatly facilitated by explicitly providing access to such solutions as is the case for real-world teachers, who can rely on access to readily avail- able solutions and, thus, focus entirely on how instructive their explanations are for students.
Begin salation tag
Begin of think tag
Teacher
Unmagleed
Solution tokena (s)
Distilled think tokens (t.,)
End of solction tag
End of think tag
Ragli of axplanation tag
Begin of solution tag Solution tokens
Completion (n)
Explanation tokena (f,,)
End explanation tag
End of sollution tag
SS "Explanation usefulness understand question solution"
log likellhood
Solution tokans (r)
TT
Student solution log probs (Agl,,( q.)
kL
"Explanation clarity the student's mind"
Explanation tokens (t..)
Distille think tokens (t,))
To this end, illustrated in Figure 2 RLTs are prompted with a new formatting style, providing both the question and solution each problem as inputs, and are tasked to produce instructive step-by-step explanations, connecting the dots between the two. We design our prompts to allow direct reuse of the teacher's outputs for student distillation while keeping the task nat- ural, appending the solution tokens s; and tags to the RLT's 's system prompt and input question before generating each completion. At test-time, constructing the corresponding question completions for the student distillation datasets d: E DAD then as simple as extracting the think tokens from the teacher's 's outputs by replacing surrounding explanation with think tags and appending back the solutions s;.
Teachar think log proba logTuec,,155.n)
Student think log probs (o,
KL Divergence
Figure 3: The tokens from the RLT's explanations are copied into the student format to measure its understanding with our reward terms.

---

3.3 Evaluating the quality of explanations
The reward function to train RLTs is made of two terms to incentivize explanations o, that lead the student R to recover correct solutions s; and are also logical continuations from questions alone under the student's perspective. In particular, following the procedure from the previous subsection, for each completion o; from the teacher To. extract the think tokens to, and format the corresponding student distillation prompt de by prepending the question qi and appending the ground-truth solution St. As illustrated in Figure B] each distillation prompt is then fed as input to the student model to obtain a set of per-token log probabilities, which are processed into our two reward terms as follows:
ris: quantifying the student Ts understanding of the solutions s; given the question qi and think tokens lo in context. This first reward term is computed with the student's log probabilities over the solution tokens. We reduce this vector to a scalar by applying both average and minimum operations over the different per-token values:
r (0;, 81, 9i) = avg (log " - a min "") where T = (8 (o.-))) (3)
ii p quantifying whether the think tokens for themselves are interpretable logical continu- ations from the student's perspective as compared with the teacher's. This second reward term is computed with the KL divergence over the same think tokens between the teacher's distribution (under the RLT's format with both qi and in context) and the student's 's (with only the question in context). Similarly to ri we reduce this vector to a scalar by applying both average and maximum operations over the different per-token values:
KL Si, 9:) = avg DKL. (To
for || lo: }} + a max { {DKL. (TØ for 1|T )
(4)
where
to: T., (to: I 14),
To
mo(to; si, gi). for
Finally, the RLT rewards are obtained by combining these two terms with weighting coefficient A: RLT y (oi, Si, qi) - A -r (o;, 85, q:)
KL
(5)
Each term our reward function serves a precise purpose. First, optimizing SS will produce explanations containing think tokens to, that maximize the student's likelihood of reaching the correct solution 8;- However, this term alone does not differentiate between explanations that guide the student step-by-step and those that increase the solution's 's likelihood without a logical path that can be learned from. An extreme instance of the latter would be an explanation simply repeating the solution tokens increase likelihood, failing to provide the student with general examples of reasoning methods that can be applied when approaching new problems. Thus, introducing KL fills precisely this gap, aligning the teacher's distribution toward the student's such that each think token in the output explanations cannot have too low probability when formatted in the distillation prompt d, with only the question qi and the previous think tokens in context.
Intuitively, introducing this term regularizes for each step in the logical path traced by the teacher's s explanation to still make sense in the "student's mind" given only its prior understanding and the question itself. We note that if we instead compared two distributions conditioned on the question alone, the KI. would vanish, and KL would fail penalize the RLT steps that the teacher could not have generated without the solution in context. Additionally, combining the average with min/max reduction terms ensures the rewards do not forego any individual token, regardless of the solution length or the number of think tokens in the teacher's 's explanations. For instance, their omission could bias SS based on the length of the solutions or lead the teacher to prefer long explanations only to reduce the influence on ,KL of hard but necessary individual logical steps. For further discussion, we refer to Appendix [ where we empirically analyze and validate each design choice in our reward function in terms of final performance (Table 14) and concrete qualitative differences of the resulting through explanations (Figures (through 12).
3.4 Pulling everything together: the RLT training paradigm
The RLT framework can be used with any RI. algorithm (e.g., 19] with minimal modifications to the LM's conditioning and reward, as described in the above subsections. For all our main
5

---

Table 1: RLTs and prior distillation pipelines across model (7B and 32B) and data size (IK and 17K).
Model
Data size
AIME 2024
MATH 500
GPQA Diamond
Overall
QwQ-32B
N.A.
50.00
90.60
54.50
65.03
DeepSeek-RI
800K+
79.80
97.30
71.50
82.87
Swen2.5--B-Insstne
N.A.
10.00
74.20
33.30
39.17
Bespoke-7B-Ik
IK
13.30
80.00
33.80
42.37
RLT-7B-1K (Ours)
IK
20.00
80.40
41.90
47.43
Bespoke-7B
17K
20.00
82.00
37.80
46.60
RLT-7B (Ours)
17K
23.30
82.80
42.40
49.50
Qwen2.5-32B-Instee
N.A.
26.70
84.00
49.00
53,23
s1-32B
IK
50.00
92.60
56.60
66.40
s1-32B budget forcing
IK
56.70
93.00
59.60
69.77
Bespoke-32B-IK
IK
46.70
92.60
57.50
65.60
RLT-32B-18 1K (Ours)
IK
60.00
94.00
60.10
71.37
Sky-TI-32B
17K
43.30
82.40
56.80
60.83
Bespoke-32B
17K
63.30
93.00
58.10
71.47
RLT-32B (Ours)
17K
66.70
93.40
59.60
73.23
experiments, we employ the simple GRPO recipe detailed in Section training objective:
resulting in the following
JRLT (0) E B,,eDD fo)s~=o((leo)))
RLT
(RLL(G mret))
(6)
G
where ARTT computed with the normalization strategy defined in Equation 2 using the RLT reward function from Equation Unlike for correctness-basee rewards, our learning signal is inherently dense, providing informative rankings to the RLT's output even before achieving any task expertise. This fundamental difference greatly facilitates optimization, akin how heuristically shaped rewards enabled RI. agents to learn entirely new behaviors for videogames and robotics tasks 2 21
Experiments
4.1 Training, distillation, and evaluation
We train RLT's on the set of questions and solutions selected by Li et al. 11 based on their level of challenge. This dataset comprises less than 17K math and coding problems originally used for distilling filtered and post-processed reasoning traces collected from QwQ [22] and DeepSeck RI [4]. In contrast, the RLTs we consider are orders-of-magnitude smaller models, all trained starting from the 1enn2.5-7BIInsruue LM (23). We precede our RL phase with a short supervised fine-tuning phase to familiarize RLT' with their new system prompt and input format using the open reasoning dataset released by Labs [13] During RL, we compute the reward for the RLT explanations using another small Qwen- model as the student. We train our main models for 125 steps, less than single epoch, with batch size of 1024, constant learning rate of 1 10-6, and a group size of 64. We note that we were also able to train RLTs with a smaller batch size of 256 and more steps for faster preliminary experimentation with only slightly inferior results.
We collect our distillation dataset with the learned RLTs using the same full set of 17K question- solution pairs from training. With the new reasoning traces, we then proceed to fine-tune our students either on this full data randomly sampled IK subset, equating the distillation budget and following the same recipes as our baselines 6 12 Unlike previous RL distillation pipelines, we do not apply extra postprocessing refinements to improve the quality of the RLT's reasoning traces, directly using our model's 's raw outputs for student fine-tuning. We note this is in contrast to prior distillation pipelines that make use of the ground-truth answers to verify the correctness of their reasoning traces and rely on multi-genene celliigenerations, filtering, and post-processing stages to improve data quality 24 13] We refer to Appendices A and for further details regarding our training and distillation phases with complete lists of hyprrparameters.

---

Table 2: RLTs and prior distillation pipelines for cold-starting traditional RL.
Model
Data size
AIME 2024
MATH 500
GPQA Diamond
Overall
Bwen2.5-7B-Instruuee
N.A.
10.00
74.20
33.30
39.17
Bespoke-7B
17K
20.00
82.00
37.80
46.60
RLT-7B (Ours)
17K
23.30
82.80
42.40
49.50
RL no cold-start
N.A.
13.30
74.20
34.80
40.77
RL cold-start (raw) + RL
17K
10.00
71.00
34.80
38.60
RI cold-start (GPT) + RL
17K
16.70
78.20
36.90
43.93
Bespoke-7B + RL
17K
16.70
82.80
45.40
48.30
RLI-7B RL (Ours)
17K
26.70
84.00
40.90
50.53
Following prior work 13 13
12 our main evaluation considers three popular and challenging tasks
from the literature: AIME24 25 the set of problems used for the American Invitational Mathematics Examination. MATH 500 [26]. the set of problems selected by (27] from the canonical competition math benchmark. GPQA Diamond [28], the set of diamond difficulty problems natural science topics from the Graduate-leve Google-prost Q&A benchmark. We report the completion accuracy of each of our students using Lighteval [99] When available, we use baseline results reported in prior work, which we found close to our early reproduction attempts. In Appendix[C we extend the experiments in this section by evaluating our models on additional tasks and analyzing the impact of teacher size, the base RL algorithm, and the student used during training.
4.2 Test-time reasoning across teachers and students
Our main experiments focus grounding the effectiveness of RLTs to obtain instructive reasoning traces beyond traditional distillation pipelines. As described in Section 4.1 construct the student distillation dataset, we use the same starting question-solution pairs as our recent state-of-the-arr baselines 12 13), with each sample only differing in terms of its reasoning trace. While RLTs could be inexpensively applied to provide explanations of larger corpora, this consistency serves to remove potential confounding factors, other than the quality of the reasoning traces, biasing our experiments and comparisons. For the same reason, we do not retune any hyperparameters for the distillation phase, training students following the same procedure as our baselines based on data size [2 6)
We compare the RLTs' explanations with prior approaches, evaluating students fine-tuned on both full 17K distillation samples and its 1K subset. Our recent baselines all follow a similar recipe of distilling datasets obtained by generating reasoning traces with expensive reasoning models or API calls and postprocessing them with closed-source LMs: sl 66) using traces from Gemini Flash Thinking 33). Sky-TI [12] using traces from QwQ (2). and Bespoke [13) using traces from
DeepSeek R1 [4. Since the Bespoke baseline obtained state-of-the-are results with our same question- solution corpus, we extend its evaluation with new results distilling its processed R1 traces only for
our same IK questions subset, equating its number of datapoints with the other s1 baseline.
As shown in Table I the raw output explanations of our small 7B parameter RLT outperform all the considered data-distillation pipelines involving teachers with orders of magnitude more parameters and additional ad-hoe postprocessing steps. We also find that the effectiveness of the RLT traces stays consistent across different data sizes. In contrast, the RI traces from the Bespoke pipeline appear significantly less effective when subsampled. Furthermore, even when distilling a Qwen-32B student, much larger than our 7B teacher, our RLT still outperforms all prior methods for both data sizes with considerable margins. We believe this result, in particular, shows how our framework could allow overcoming the current prohibitive costs of RL reasoning: shifting the burden of expensive RL procedures to small teachers, unable to effectively solve problems from scratch but highly specialized the simpler task of producing effective explanations for large, more powerful students.
4.3
RLTs to cold-start RL
Our next set experiments focuses on evaluating the effectiveness RLTs in providing cold-start data for traditional RI. For this new RI phase, we use our same GRPO implementation with the standard student format and correctness-basee rewards described in Section As compared to the RLT framework, we find that using a larger batch size of 1024 is significantly more beneficial to
7

---

Countdown (CD) performance
MATH500 performance
60-
73 RLT traces 56.5
Lange model distillation
79-
Student cistilation
Owen22 5---strrt baseline
50
47.5
49.2
78
78.0
77.2
77-
76,8
76.
76.0
40
76
75,8
30
74.6
74.5
74.4
74.4
Owen 78
Bespake-78 (R1)
Bespake RL
RC transfer ooD
RDT-70
Bespol e CD Rl
Rt
915
#
Dotaset Index by RIT rewsrrd rank
Figure 4: Left: -ittof--iitribution performance transferring RLTs to produce new distillation data as compared to students trained on the Li et al. [12 corpus and direct RL on the countdown task. Right: Performance after training on different distillation datasets ranked by the RLT reward.
better cope with the increased variance and reward sparsity of traditional RL. We train for a full epoch on the recent RL dataset from Li al. 3 collected by analyzing and selecting an effective subset of the competition math data based on the correlation of individual samples with overall performance improvement.
We compare performing this new RL phase on the Qwen-7B model cold-started from the reasoning traces of our 7B RLT and the postprocessed R1 traces from the Bespoke pipeline, strongest distillation baseline. Moreover, we also compare a 7B parameters baseline teacher trained with traditional RL as done in prior work [1] 4 effectively performing RL twice on the Qwen model and collecting a dataset at the end of the first iteration to cold-start the second. To construct the cold-starting dataset for this last baseline, we consider either taking the model's raw output traces, as done with RLTs, or postprocessing them with additional refinements using GPT4, 1-mini following very similar strategy to the other RI and QwQ traditional distillation pipelines
[32 and
12
As shown in Table 2 the reasoning traces from our 7B parameter RLT again display superior cold- starting effectiveness compared to all of our baselines. The performance gap is exceedingly noticeable with the cold-starting approaches that are also using a 7B teacher trained with traditional RL. In fact, only after improving the format and structure of the traces from these RL-trained teachers with GPT postprocessing, we were able to observe any improvements from the original Qwen-7B results. While our RLT was itself trained from the same 7B model, it again demonstrates superior cold-starting even when compared to postprocessed RI pipelines. Overall, we find these results to be compelling evidence indicating that RLT' have the potential to unlock new key avenues to democratize the RL reasoning framework beyond the current reliance on prohibitively large and closed-soure LMs.
4.4 Out-of-domain zero-shot transfer
Unlike problem-soleee solving from scratch, we posit that providing effective explanations to given solutions is much less task-specific skill. Thus, in this subsection, we evaluate how well RLT's can be applied to construct datasets and distill new specialized students out-of-distribution domains, without any expensive RL retraining. In particular, we focus on the canonical countdown task [3] asking student models to combine a set numbers to equal a given target using basic arithmetic operations. We train and test our models on distinct datasets of 16K and IK weatmmaticallyggnnentteee solution pairs. We compare zero-shot transferring our RLT with transferring the RLT-7B student and the Bespoke-7B baselines from the previous subsections. To ground our results, we also consider performing RL on the countdown task itself (CD RL), training both from the Qwen-7B model and the cold-started Bespoke-7B 7B baseline with the same setup described in Section
question and
As shown the left bar plot of Figure 4 applying RLT distillation zero-shot remarkably achieves even higher performance than direct RL on the countdown task. Interestingly, direct RI. appears to provide only marginally better scores than using students distilled from our original set of reasoning questions that do not include any examples of countdown problems (50.8 vs. 49.2). Furthermore,

---

R1 postprocessed reasoning
RLT explanation
(beggnn of_ouhht
<|baginn_of of _explanation])
[-]
[-]
Just check, If use calculator..
First, the unit circle definition..
[.]
Another angle (no pun intended]) using the derivative..
Altematively, using calculus. The derivative of...
[.]
[-]
Another way to check maybe using complex numbers or other identities, but think the method is solid. the final answer </end of__ofoo_hhoh_ r_thoughtt] >
Altemativell. use complex numbers. Let me try that Express the two angles compiex numbers the unit circle.. <[end_of_ _eed_of_expla_eeeeeeeedde____e_d of_explanation]
Figure 5: Examples comparing the contents from the post- pest-proeessd R1 traces 13 that were particularly improved by the corresponding RLT explanations as measured by our reward function
find there is stark overlap of over 98.5% in the final sets of solved problems between direct RL and the RL-free Bespoke-7B baseline. We find these results in line with prior analysis [12] (9. providing further evidence that the exploration challenge of traditional RL might make most of its benefits come from steering the base model's distribution toward long-context generation. In contrast, by simplifying the task and foregoing sparse rewards, our RLT appears much more effective providing countdown-specing traces that allow students to learn new knowledge and solve unseen questions, yielding higher improvements than direct RL even without any teacher training in this new domain.
4.5 Explanation reward analysis
To analyze the design of the RLT reward function, we start by examining the relationship between the traces' rewards and the effectiveness of student distillation. In particular, use our RLT's checkpoint right before RL training to generate 16 completions for each question-solution pair in our data. We then score all completions with our reward and divide them into groups based on their relative rank for each prompt. Thus. we obtain 16 datasets with different reasoning traces for each question, which we use to train 16 new 7B students from Qwen. As illustrated in the right bar plot of Figure 4 ordering student performance by the respective dataset rank shows clear correlation between the two, with a Pearson coefficient over 0.89, validating the efficacy of the RLT objective. Additionally, the highest ranked traces of our 7B teacher before any RL remarkably already yield 90% the performance gains of our baseline R1 distillation pipeline [3], showing how even small models already possess latent teaching skills unlocked by our new reward and simplified task formulation.
We also inspect qualitative examples chosen by selecting samples where the reward of the RLT explanations is particularly improved from the baseline RI distillation pipeline. As shown in Figure [ we find that the RI traces with low rewards often try rely on external tools, such calculators. and employ language patterns likely idiosyncratic to the training data of the DeepSeck-Ve LM, as sentences with brief humorous comments [16]. Instead, the corresponding RLT explanations appear much more grounded and even manage to add new alternative verification steps not considered by RI to check the final solution. In Appendix D we provide additional examples showcasing further qualitative differences of our framework with R1 traces and also specific failure cases from training RLT's without proper balance between each reward component, such as repetitions and overly-long explanations.
Related work
Inspired by the unprecedented abilities of the OpenAl ol model (34] there has been a resurgence of RL approaches aimed at inducing a new kind of open-ended reasoming to scale test-time compute. The work from Guo et al. 4 was another milestone in this new domain, providing a first openly detailed example of what is possible with large models and RL. Other follow-ups considered smaller LMs and ways to decrease optimization costs via approaches such as explicit task breakdown [[]. exploration strategies [5], new RL objectives [35], and cold-start data scale [10]. However, it is still an open question RL on small models can go beyond cheaper supervised alternatives 4 and induce new skills beyond the pretraining corpus 19. In contrast to this work, RLT's break the traditional framework of maximizing one-hot accuracy with verifiable rewards turning the task on its head by feeding the model the correct solution input and avoiding RL's inherent exploration challenge.

---

A large part of the recent test-time scaling literature considering smaller LMs has focused on inducing reasoning with "teacher-student student" supervised distillation [36.. a widely validated technique in traditional LM development [23 37.. This approach's 's popularity to induce I.M reasoning dates earlier than the RL paradigm, with older methods harnessing verifiers and prompting for self- Part of this earlier exploration showed preliminary signs of how teachers could be considerably steered to provide better student data [40 41] and how improvements in teaching also led to improvements in question-answering domains 42 Lately, by following common structure of generation, filtering correct responses, and postprocessing them, modern RL-based distillation has seen significant advances mostly driven by more capable teachers [3 12 and carefully curating targeted datasets 66, [7]. However, the effectiveness of current distillation pipelines was shown to be closely tied to the properties of the student itself 44 [0], and their ability to induce actual generalization remains unclear 44 Unlike these traditional distillation pipelines, the RLTT framework does not rely verifiers for filtering, directly optimizes the teacher for downstream distillation, and does not require any post-processing, allowing direct transfer of reasoning capabilities to arbitrary tasks and even larger student models.
improvement [8 39
6 Discussion and extensions
This work introduced a new class of Reinforcememn-Leareee Teachers trained with simpler dense- reward task that inputs both each problem's 's question and solution, and optimizes the LM provide instructive reasoning traces for distillation as outputs. Empirically, students trained or cold-started from the raw outputs of 7B RLT obtain higher performance than prior distillation pipelines using orders-of-mapee magnitude larger LMs. Furthermore, RLTs maintain their effectiveness also when providing reasoning traces for -iitiii-iittittiionn tasks beyond their training corpus, and even for distilling much larger students than the teacher itself. These remarkable properties can have significant implications for the scalability of developing reasoning models, due to the disproportionate cost of RI. compared to all other post-training stages: while distilling a 32B student on fixed traces took less than day on a single H100 compute node, training this larger model using RL on the same questions would have taken months with the same hardware. Nonetheless, our work has only begun to study the design space of our new framework, with many exciting directions yet to be explored. One example is training RLT' and their students in tandem, allowing the teacher's explanations to adapt to the student's learning dynamics live. Pushing this further, the same model could even take both roles, iterating RL with task formulation, providing access to each problem's 's solution, to obtain instructive step-by-step reasoning traces, and self-distillation (38) to revise its own explanation and learn how to solve questions from scratch unifying the open-endedness of RL. with the stability of supervised optimization.
References
[1] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[2] Tim Salimans and Richard Chen. Learning montezuma's 's revenge from a single demonstration. arXiv preprini arXiv: 18/2.03381, 2018
[3] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv: 10995, 2019.
[4] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
[5] Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao. Juanzi Li. Jie Tang, and Yuxiao Dong. Advancing language model reasoning through reinforcement learning and inference scaling. arXiv preprint arXiv:2501. 2025,
[6] Niklas Muennighoff Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.
10

---

[7] Takeshi Kojima. Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 15:22199-22213, 2022.
[8] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint 0xX3:22408.03314, 2024.
[9] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model?, 2025.
[10] Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Jiaming Ji, Yingying Zhang, et al. Redstar: Does scaling long-cot data unlock better slow-reasoning systems? arXiv preprint arXiv:2501. /284, 2025.
[11) Maohao Shen, Guangtao Zeng. Zhenting Qi. Zhang-Wei Hong. Zhenfang Chen, Wei Lu, Gregory Wornell, Subhro Das, David Cox, and Chuang Gan. Satori: Reinforcement learning with -wwpt-------cii--tooooe enhances IIm reasoning via autoregressive search. arXiv preprint arXiv:2502.02506 2025.
[12] Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G Patil, Matei Zaharia, et al. Llms can easily learn to reason from demonstrations structure, not content, is what matters! arXiv preprint arXiv:2502.07374, 2025.
[13] Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distilla- tion. www.bespokelabs.co awt.aa//bbbbppp// wwt/a/llo/bbspoo-sssrrttttsossee wtt.e//twws..l.....ooooooooeeeee distillation, 2025.
Accessed: 2025-01-22.
[14] Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:2312.08893, 2023.
[15] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:240..03300, 2024.
[16] DeepSeek-Al. Deepseek-v3 technical report, 2024. URL httpp::/arxiv.org/bbba.t. 19437
[17] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less more for reasoning. arXiv preprint 2502::2000..3387, 2025.
[18] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.0655 2017.
[19] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback 11ms. arXiv preprint arXiv:2402. 14740, 2024.
[20] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1937. PmLR, 2016.
[21] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 11X1v:13 2013.
[22] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenIm. github. / boo/blo//// -wg-gwwgbbaa

---

[23] Binyuan Hui, Jian Yang. Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang. Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. /2186, 2024.
[24] NovaSky Team. Sky-tl: Train your own ol preview model within S450. https//novasky- higithubbio/postt/kkkytll 2025. Accessed: 2025-01-09.
[25] Mathematical Association of America. Aime, February 2024.
URL /ttps://
artofprooleemsolving wwtpsiki/wwowiiioeeee http://wwiiiiiiie..eeee// php/ http/AIMM_PPoooleee and Solutions/
[26] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
[27] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint 230.::3305.200500 2023.
[28] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
[29] Clémentine Fourrier, Nathan Habib, Hynek Kydliček, Thomas Wolf, and Lewis Tunstall. Lighteval: A lightweight framework for I1m evaluation, 2023. URL https://github.. com/ huggingface/ Sangginngaa///ghttaaal
[30] Google. Gemini 2.0 flash thinking mode (gemimi220--lsss-tiininnn--x>-2- De-
cember 2024. URL httpss//clout: gowgle..oog....oooooo com htt.gg/w.w...eteee.aaaaaaaaataaa
thinking-mook -ode
[31] Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling. arXiv preprint arXi::2502./ 18866, 2025.
[32] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
[33] Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D Goodman. Stream of search (sos): Learning to search in language. arXiv preprint arXiv:2404.0368: 2024.
[34] Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, al. Openai ol system card, arXiv preprint 2rXiv:2412.16720) 2024.
[35] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective, 2025. URL
//arxiv. orgg www.:og///aaaaaeeeee htsp//bbb2200.... http:/553..2010 20783
[36] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1033:2150..0253,, 2015.
[37] Makoto Shing, Kou Misaki, Han Bao, Sho Yokoi, and Takuya Akiba. TAID: Temporally adaptive interpolated distillation for efficient knowledge transfer in language models. In The Thirteenth International Conference on Learning Representations, 2025. URL httpss
//openreview net//forum?2d-oqsw2bbbb
[38] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022. httptppps//a.iiv.oor/a/a//////// 14465
[39] Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet-star: Language models can teach themselves to think before speaking, 2024. URL
https: :taps:////xivv org/ ab////bs//...// 09629
12

---

[40] Zaid Khan, Elias Stengel-Eskin -skinn Jaemin Cho, and Mohit Bansal. Dataenvgym: Data generation agents teacher environments with student feedback. arXiv preprint arXiv:2410.06215, 2024.
[41] Xiaochuan Li, Zichun Yu, and Chenyan Xiong. Montessori-instruet:: Generate influential training data tailored for student learning, arXiv preprint arXiv:2410. 14208, 2024.
[42] Xuefei Ning, Zifu Wang, Shiyao Li, Zinan Lin, Peiran Yao, Tianyu Fu, Matthew Blaschko, Guohao Dai, Huazhong Yang, and Yu Wang. Can lims learn by teaching for better reasoning? preliminary study. Advances in Neural Information Processing Systems, 17:71188-712399 2024.
[43] Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ra- masubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. arXiv preprint arXiv:2502. 12143, 2025.
[44] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong. Saining Xie, Dale Schuurmans, Quoe V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025.
[45] I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
[46] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforce- ment learning. htttps://gite gitggtthhuu..... com/ huggingface/tr// 2020.
[47] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[48] Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev. Yaroslav Ak- senov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. arXiv preprint vrXiv:2404.0966 2024.
[49] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. urXiv preprint arXiv:2403.07977 2024.
[50] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
[51] Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprini arXiv:2501.03266, 2025.
[52] Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024.
[53] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: (001:080000000 2018
[54] Ilge Akkaya, Marcin Andrychowicz. Maciek Chociej. Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint 191007111 2019.
[55] Rati Devidze, Goran Radanovic, Parameswaran Kamalaruban, and Adish Singla. Explicable reward design for reinforcement learning agents. Advances in neural information processing systems, 34:201 18-20131, 2021.
[56] Edoardo Cetin, Tianyu Zhao, and Yujin Tang. Large language models to diffusion fine- tuning. In Forty-second International Conference on Machine Learning, 2025. URL
13

---

Table 3: Hyperparameter listing for the RLT training optimization and reward.
Hyperparameter name
Value
RLT training
Fine-tuned model
Qwen2.5-7B-instruct 23
Number training steps
125
Batch size
1024
Learning rate
1 10-6
Learning rate decay
Constant
Final learning rate
1 x 10-6
Weight decay
0
Optimizer
AdamW 45
Adam betal
0.9
Adam beta2
0.999
Adam epsilon
1e-8
Warmup steps
0
Maximum gradient norm
1.0
Maximum generation context size
16384
Generation temperature
0.7
Generation top-p
1.0
Generation top-k
No
Generation min-p
0.0
Generation repetition penalty
1.0
GRPO group size
64
GRPO |
0.04
Reference model syne. steps
32
Reference model sync. mixup
0.9
Dtype
b6
Gradient checkpointing
true
RLT reward
Student model
Qwen2.5-7B-instruet 23
3
0.01
Format penalty
-1
A Implementation details
A.1 RLT training phase and reward
Our experiments conducted single compute node comprising 8 Nvidia Hopper H100 GPUs, 1.8 TB of memory, and 208 Intel Xeon Platinum 8481C CPUs. Due to the efficiency of training with small 7B models, note this setup is significantly less resource-intensing than prior RL and even SFT work, often relying on multi-node settings 6
[2) J As described in Section for its new RL phase, train Reinforcement Learned Teachers the set of questions and solutions selected by Li et al. 12 which is available under an Apache 2.0 License, comprising less than 17K math and coding problems. All our teachers are trained from (0wen2.5-7B-Instruu LM 23) also available under an Apache 2.0 License, together with the other models from the Qwen family. Before RL, shortly fine-tune on pre-collected samples using example traces from Labs [ formatted using our new system prompt and tags. This short, inexpensive phase is conducted with the same distillation hyperparameters used for the IK data subset following Muennighoff et al. ( detailed in Appendix B but with double the number of epochs, and serves to quickly familiarize our teacher
14

---

Table 4: Cost to train and generate datasets with RLT models as opposed to DeepSeek RI 4
Model/GPU-hours
Training (GPU model)
Data generation (GPU model)
DeepSeek RI 7B RLT teacher
>68800 (H800)
>1067 (H100)
280.4 (H100)
6.7 (H100)
with the new RLT input format. We also use the output checkpoint at the end of this phase for our correlation analysis experiment conducted in Section 4.5 For the same reason and also to limit the requirements for reproducibility, our lightweight Qwen2.5-7B-inste compute the RLT reward function is initialized with the checkpoint provided by Labs [3 available under an Apache 2.0 License, which is already familiar with their system prompt without needing further training. However, we found that the RLT reward is robust to the specific LM choice for the student, yielding numerically close values when using our own Qwen student finetuned on only IK samples.
[23] student model used to
We use simple coefficients regulate the terms in the RLT rewards, with X = 3 to scale RT. and 0.01 to scale both the min term in r and the max term in y. ,k. Our choice was based making each individual component the RLT rewards have approximately the same expected magnitude over the model's initial output completions. As we observed the overall rankings to be quite robust after testing small changes to these initial choices, we did not find extensive sweeps necessary. We also add 11 penalty to any completion that does not use the explanation tags that exceeds generation context length to limit training time and disincentivize overly long and expensive reasoning traces. For our training runs, make use of a custom GRPO implementation with the specifications from Shao et al. [1). extending the TRL library 46 with faster distributed VLLM generation [47]. Our RL phase is short, comprising only 125 steps, less than single epoch, with batch size of 1024, an AdamW optimizer 455 with a constant learning rate of 1 x 10 6 and a group size of 64. We synchronize the reference model every 32 steps as popularized by 48 with mixup ratio of 0.9. For the post-cold-start RL phase employed in Section 4.3 we use our same GRPO implementation with the standard student format and correctness- -based rewards described in Section 2 The only difference is that we train for one epoch in total on the very small LIMR (31) dataset, containing less than IK samples. We provide a full list of hyperparameters ensure reproducibility in Table All main libraries used in our implementation are again available under an Apache 2.0 Licence.
maximum
A.2 Traditional reasoning and RLT formats
As detailed in Section 3.2 and illustrated in Figure 2 RLTs are prompted with a new formatting style that provides both the question and solution to each problem as inputs. Then, their system prompt instructs to output instructive, detailed step-by-step explanations, connecting the dots between the two. In contrast, traditional formats used for reasoning datasets employed for RL and distillation forego any information about each problem's 's solution and task the model to solve each problem from scratch. We provide specific comparison between the two, contrasting the system prompts used for traditional RI and distillation from Li et al. [12] in Figure A.2 and our own new RLT input format in Figure A.2 As shown, our prompting format's ''s design strives to make minimal changes to the prompts used for traditional reasoning frameworks, replacing surrounding expl explanation in with think tags, and simply appending the solution tokens after the user's provided input question, to allow our teachers to make full use of this information before generating each completion. This design also practically avoids the need to re-prompt our teacher multiple times for each question and manually check the answers to filter out incorrect solutions.
A.3 Cost comparison
In Table 4 we report the GPU cost of training and collecting distillation data with our RLT. We compared these times with an estimated cost of traditional RL post-training and data collection using the DeepSeek R1 model. To compute this estimate, used official figures for the GPUs used to train this DeepSeck model (2,048 NVIDIA H800 GPUs (16)) and online communication from the DeepSeek team for the training time. We calculated a lower bound of the GPU hours for
http:r//waggggtttecccc//epeeeeee Strrtos--7
15

---

Reasoning input format
System prompt
[limstarr| (>aystem
Your role as an assistant involves thoroughly exploring questions through a systematic long thinking process before providing
final precise and accurate solutions. This requires engaging in comprehensive cycle of analysis, summarizing, exploration,
reassessment, reflection, backtracing, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution. In the Thought section, detail your reasoning process using the specified format: ofeounnnt_thougtt/>>> {thought with steps separated with '\n'n'} Each step should include detailed considerations such analisying questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The solution should remain a logical,
lend_of_thoughee
accurate, concise expression style and detail necessary step needed to reach the conclusion, formatted as follows: < (final formatted, precise, and clear solution} <lend_of_solution
Now, try to solve the following question through the above guidelines: imden|/>
Generation prefix
/>ueer
Return your final response within \boxed(}. Positive integers $a$ and
$b$ are such that the graphs of $y=ax+5$ and $y=3x+b$ intersect the
- $x$-axis at the same point. What is the sum all possible $x$-coordinates of these points of intersection?
$ltextbf( (=20N\gquadtexxtbf
{-18) (ext)
{-12) (egeegguuaatttttbaaae
{-8) im im__nd|
< <lim_start Passistant
< <lbegin_off oftogghtttt/>
Figure 6: Reasoning input format employed in traditional RL and student distillation, using example question and the system prompt from Li et al. [12), providing the model instructions first to present step-by-step rationale and then the problem's solution, deriving it from scratch.
generating the distillation dataset by multiplying the minimum amount of GPU resources required host DeepSeek RI with half precision and the "roof inference speed" for generating each completion in our dataset, without taking into account slowdowns from the growing context size. However, we note that both these estimates should be considered large approximations as we did not even consider time/resources for multiple generations, filtering, and post-processing stages conducted in our baselines 12 13
B Student distillation
B.1 Reasoning traces generation
As detailed in Sections 3 andd we collect reasoning traces for each sample in the student distillation dataset by feeding RLTs both the problem's 's question and its solution input. Moreover, for
16

---

RLT input format
System prompt
1lim_starti s (saystem
Your role as an assistant involves providing precise and accurate solutions before providing detailed explanations with your full work
showing your systematic thinking process leading to each solution.
Your explanations should show how you engaged in a comprehensive cycle of amalysis, summarizing, exploration, reassessment, reflection, backtracing, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Solution and Explanation. In the Solution section, present your well-thought solution that accurately answers the question. The solution should remain a logical, accurate, concise expression style and detail necessary step needed to reach the conclusion, formatted as follows: dobegin_of_solutinnn
(final formatted, precise, and clear solution) [and_of_solutoon>>>> In the Explanation section, comprehensively detail your reasoning process using the specified (explanation with steps separated with 'nnn' <lend_of _es____f_xxpllaaiioonono Each step should show detailed considerations leading to your solutions such as analisying questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. <lim_end|>
format: ofbegin__o__xxplaattii
Generation prefix
< <lim_start/
Return your final response within \boxed(}. Positive integers $a$ and
$b$ are such that graphs of $y=ax+5$ and $y=3x+b$ intersect the $x$-axis at the same point. What is the sum of all possible $x$-coordinates of these points of intersection?
(=IB\gguuddttoxtbfff (C)) $ ((textb((()) (=2x))g
{-15>\gquad < inn_sarrt>assistaan
{-12) tex) {-8}$< im end|>
-
[bogin_
the of all possible \( x \) -)coordiisttes of these points of intersection is (V)))
< <fend_of_ ofllutiion/
Figure 7: RLT Input format using an example question from Li et al. [12, providing the model instructions produce step-by-ster explanation given the solution each problem as input.
cold-starting experiments in Section 4.3
we also collect reasoning traces by feeding the RL-trained Qwen models each question and postprocessing each output with GPT4.1 (32 similarly Li al.
12 We provide the hyperparameters used to collect these traces across all our datasets and settings Table (5 In particular, we use standard generation hyperparameters for reasoning Qwen- models [2], that are aligned with the online generation parameters during our new RL phases, as detailed in Appendix A The only main difference is that we allow for a longer maximum context size to avoid collecting incomplete traces for downstream distillation, which is possible thanks to the fact that test time we are subject the same training-time memory constraints dictated by backpropagating through long traces.
17

---

Table 5: Hyperparameter listing for the RLT generation pipeline.
Hyperparameter name
Value
Distillation data generation setting
Maximum generation context size
32764
Generation temperature Generation top-p
0.7
1.0
Generation top-k
No
Generation min-p
0.0
Generation repetition penalty
1.0
Generation dtype
bíloat16
Table 6: Hyperparameter listing student distillation phases on the datasets from the set of questions from Li al. [12], and the countdown dataset.
Hyperparameter name
Full fine-tuning
IK subset fine-tuning Countdown fine-tuning
Student training
Distilled model
Qwen22.577B332B-iisttuuu
23
Number training samples
16710
1000
16000
Number epochs
3.0
5.0
3.0
Batch size
96
16
96
Learning rate
x 10-5
1 x 10-5
1 10-5
Learning rate decay
Cosine
Cosine
Cosine
Final learning rate
10
1 X 10-6
10-6
Weight decay
1 x 10-4
0
Optimizer
AdamW 45
AdamW 45
AdamW 45
Adam betal
0.9
0.9
0.9
Adam beta2
0.999
0.95
0.999
Adam epsilon
x 10
1 x 10-8 8
10
Warmup ratio
0.1
0.05
0.1
Maximum gradient norm
1.0
1.0
1.0
Dtype
6
bfloat16
6
Gradient checkpointing
true
true
true
The RLT distillation datasets used for training all of our 7B students were collected by generating single completion for each question- question-solution solution pair, directly placing it in the student format. After preliminary experiments, we also found our 32B students can be particularly sensitive to cropped reasoning traces that exceed the 16384 maximum context length specified the SFT hyperparameters from Li et al. [12], which they purposefully limited for computational efficiency. Thus, to avoid this mismatch potentially affecting our results, we simply collected up to 16 reasoning traces for each question-solution pair and selected anyone below 16384, otherwise resorting to SS for selection.
B.2 Student distillation specifications
Our main experiments from Section usee the RLT traces collected either from the full set training questions from Li et al. [12] or its randomly selected 1K data subset. For the student distillation phases of these experiments, we re-use the same hyperparameters from et al. [22 and Muennighoff et al. (6) for the full data and 1K subset, respectively. The purpose of not re-tuning based on our new data is to attempt isolate the reasoning traces used for student distillation as the only degree of variation in our comparison between RLTs with traditional reasoning distillation pipelines.
18

---

Table 7: Hyperparameter listing for the traditional RL on the Li et al. 31
dataset.
Hyperparameter name
LIMR data traditional RL
Countdown data traditional RL
Traditional RL student training
Fine-tuned model
Qwen2.5-7B-inntruut
23
Bespokes-777
13
Number of training samples
1389
16000
Number of epochs
1.0
1.0
Number of training steps
86
250
Batch size
1024
256
Learning rate
1 x 10-6
x 10
Learning rate decay
Constant
Constant
Final learning rate
1 10-6
10-6
Weight decay
Optimizer
A 45
A 45
Adam betal
0.9
0.9
Adam beta2
0.999
0.999
Adam epsilon
1 10-8
10-8
Warmup steps
0
0
Maximum gradient norm
1.0
1.0
Maximum generation context size
16384
16384
Generation temperature
0.7
0.7
Generation top-p
1.0
1.0
Generation top-k
No
No
Generation min-p
0.0
0.0
Generation repetition penalty
1.0
1.0
GRPO group size
64
64
GRPO 3
0.04
0.04
Reference model sync. steps
32
32
Reference model sync. mixup
0.9
0.9
Dtype
6
6
Gradient checkpointing
true
true
Furthermore, in our experiments in Section 4.4 we also compare transferring students learned on our original set of datapoints and direct RI with transferring the RLTS themselves zero-shot to the countdown task [33]. For these experiment, we use a set of 16K automatically-goneet geneeraee countdown question and solution pairs with or 4 numbers and, after obtaining the corresponding RLT traces, distill our students using the hyperparameters from Li al. [12] once again, which we found to work well in practice without further tuning, given the two similar dataset sizes.
We provide full list the hyperparameters used for all our distillation experiments in each setting in Table 6 where we highlight the key differences across the three. As detailed, the considered approaches mostly differ terms of the number of epochs, the batch size, and the optimizer parameters. These differences reflect the total number of samples and relative variance in each data batch. In line with the findings from the relative prior works 6, 12, 24), we note that training on these small datasets was very inexpensive and could be completed within hours. We note that the learning rate of our reinforcement learning phases matches the final learning rate during our SFT optimization, a simple choice which we found to work well in practice.
B.3 Student RL details
For our stad-ita experiments in Section 4.3 we also implement and perform a phase traditional RL training optimizing the model with conrectness-basel rewards under the "student's perspective."
19

---

Table 8: Student evaluation hyperparameter listing for the RLT generation pipeline, matching the hyperparameters from Li al.
Hyperparameter name
Value
Student evaluation
Maximum generation context size
32764
Generation temperature
0.7
Generation top-p
1.0
Generation top-k
No
Generation min-p
0.0
Generation repetition penalty
1.0
Generation dtype
b6loat16
This phase is done atop a Qwen2.5-7B-Insttuue the cold-started Bespoke-7B. and the RLT-7B models using our same GRPO implementation on the open-source LIMR dataset [31]. We re-use most of the hyperparameters from the RLT training phase with batch size of 1 1024, something particularly important to cope with the increased variance and reward sparsity of traditional RL. Additionally, we also conduct traditional RL as baseline for our out-of-distrioon diddstriibution transfer experiments in Section (4.4 that directly optimizes correctness the countdown task starting again from the Qwen2.5-7BB -wen2.5-7B-Instruce model, and the cold-started Bespoke-7B models. For this particular task, we found using a larger batch size to be not strictly necessary, as for RLT training, and obtained better results optimizing the model for more steps (250 total) with batch size of 256. We provide a full list of the distillation hyperparameters used for our RL phases on these tasks in Table 7 where we highlight the key differences between these two settings.
B.4 Student evaluation
As described in Section 4.1 our main evaluation consists of three graduate and competition-leve tasks on math and natural science domains. In particular, these include AIME24 25 the of problems used for the American Invitational Mathematics Examination; MATH 500 26 the set of problems selected by (27] from the canonical competition math benchmark; and GPQA Diamond 288 the set of diamond difficulty problems on natural science topics from the Graduate-level Google-proof Q& A benchmark. Additionally, in Appendix [ we also extend our set of experiments include additional challenging coding and multilingual domains. In particular, contamination- free set of coding challenge problems continuously collected from several prominent online hosting platforms; and OlympiadBench [50), set of olympiad-leve bilingual problems in English and Chinese from past math and physics competitions.
consider LiveCodeBench [49],
We evaluate on all the above benchmarks using Lighteval [29]. library available under the MIT license. In all our results, we report the completion accuracy of each of our students for a single generated completion, consistency by re-using the task implementation code, including the system prompt, provided by our baselines (24], which is available under an Apache 2.0 License. For the same reason, we do not modify any of the existing evaluation generation hyperparameters from the suggested settings used in their evaluation, which we report in Table
also reported in prior work 6
12 13 Furthermore, we also ensure
Additional experiments
C.1 Coding and muitilingual reasoning
We extend our main set of experiments from Section 4 focusing on graduate and competition-level tasks on math and natural science domains, comparing the effectiveness the reasoning traces from our 7B parameter RLT with traditional distillation pipelines. We consider challenging coding and multilingual domains, which are less aligned with the training and distillation set of questions employed. In particular, these new benchmarks include LiveCodeBench (LCB) [49]. contamination-
20

---

Table 9: RLTs and prior distillation pipelines across model (7B and 32B) evaluated on coding and multilingual reasoning benchmarks. Overall computed by taking the average between LCB-Averae and OlympiadBench scores.
Model
Data size
LCB-Average
LCB-Hard
OlympiadBench
Overall
Qwen2.5-7B-Insteee
N.A.
31.88
3.30
35.90
33,89
Bespoke-7B
17K
36.10
1.60
43.30
39.70
RLT-7B
17K
34.63
3.30
46.10
40.37
Qwen2.5-32BB-Inttun
N.A.
48.94
9.80
46.70
47.82
Sky-T1-32B
17K
57.94
17.90
57.30
57.62
Bespoke-32B
17K
71.06
26.20
60.30
65.68
RLT-32B
17K
71.24
32.50
64.00
67.62
free set of coding challenge problems continuously collected from several prominent online hosting platforms across three difficulty categories; and Olympiadßench [50], set of olympiad-level bilingual problems English and Chinese from past math and physics competitions.
In Table 9 we provide a comparison of reported results from our baselines [2] and our fine-tuned student models using our 7B RLT traces. We also re-collected the performance on OlympiadBench of the Qwen2.5-7B -nInntruct model and the Bespoke- 7B baseline fine-tuned from the postprocessed R traces, as they were omitted in the reference results from prior work. For LiveCodeBench, we report both the performance on the "hard" difficulty set of problems and the average weighted performance. This was obtained by weighting the performances of the models in the "easy." "medium," and "hard" sets of LiveCodeBencc problems, by the relative number of problems in each.
We find the performance of RLT distilled students these new tasks be consistent with the performance from our Section experiments. In particular, the overall performance exceeds the performance of the baseline distillation pipelines using orders of magnitude larger models across student sizes, Furthermore, the RLT performance is also best across the individual settings, with the sole exception of LCB- -aaaaeeee only for the 7B model, where comes as close second. However, as shown by the experiments in Section 4.4 by equating the pool of initial questions from which to perform distillation, we believe these experiments could still be underplaying the true potential enabled by the zero-shot transferability of RLTs. In particular, transferring RLTs, rather than their students, to construct reasoning traces to include more coding and Chinese- problems could allow downstream distillation to develop further domain-speciin expertise and reasoning, without the need run expensive pipelines requiring large and closed-source models.
C.2 The generality of RLT across RL algorithms
Table 10: Student performance from RLTs trained with GRPO 15 and RLOO 19
Evaluated model
Data size
AIME 2024
MATH 500
GPQA Diamond
Overall
Dwen2.5-7B. -
N/A
10.00
74.20
33.30
39.20
Bespoke-7B
17K
20.00
82.00
37.80
46.60
RLT-7B
17K
23.30
82.80
42.40
49.50
RLT-7B (RLOO)
17K
20.00
83.60
42.90
48.80
To validate the generality of the RLT framework, we extended our implementation by training our 7B RLT model using the RLOO [99 algorithm instead of GRPO. We provide this comparison in Table above. In line with recent empirical findings about the similar empirical effectiveness of the different reasoning RL algorithms [5T]. the performance of our RLT trained with RLOO appears very close to our results obtained using GRPO in the main text. While there exists small gap, we believe this is mostly due to the fact that we did not re-adjust any of the main hyperparameters from our GRPO implementation (e.g., batch size, number of generations per question). These results provide further concrete evidence confirming that the RLT paradigm's is performance is not tied any specific RL algorithm.
21

---

Table 11: Student performance from RLTs with 3B and 7B parameters.
Evaluated model
Data size
AIME 2024
MATH 500
GPQA Diamond
Overall
Owen2.5-7B-Instrenn
N/A
10.00
74.20
33.30
39.20
RLT-7B
17K
23.30
82.80
42.40
49.50
RLT-7B (3B teacher)
17K
20.00
80.60
38.90
46.50
Qwen2.5-32B- Instruutt
N/A
26.70
84.00
49.00
53.20
RLT-32B
17K
66.70
93.40
59.60
73.20
RIT-32B (3B teacher)
17K
46.70
91.40
53.50
63.90
C.3 The effects of teacher scale
To validate the effects the RLT teacher's scale, we extended our implementation by training an RLT model with 3B parameters. The only change to our hyperparameters was double the number of supervised finetuning iterations, but kept the same number of RL training steps. As shown in TT expectedly find that larger teachers yield better explanations and downstream results. However, we find that most of the performance gap between our 3B and 7B teachers occurs when distilling the 32B student LMM where the mismatch between teacher and student capabilities tends to the extreme. However, even in this case where the student is over 10x larger than the teacher, our 3B RLT is still able to provide considerable improvements to the initial Qwen 32B student. The reason that we did not observe performance degradation appears to be due to the role of the r reward term plays during the optimization. Even in cases where the initial 3B teacher is unable to provide logical explanations to a specific question and answer, optimizing ,' will naturally guide the teacher's s output distribution (with both question and answer context) toward converging to what was most likely from the student's 's distribution (with only the question in context).
C.4 Stronger students make stronger teachers
Table 12: Student performance using RLT's trained with combination of 7B and 32B students, and following 7B student updated midway through training.
Evaluated model
Data size
AIME 2024
MATH 500
GPQA Diamond
Overall
Qwen2.5-7B-Instruu
N/A
10.00
74.20
33.30
39.20
Bespoke-7B
17K
20.00
82.00
37.80
46.60
RLT-7B
17K
23.30
82.80
42.40
49.50
RLT-7B (7B + 32B)
17K
23.30
83.80
41.90
49.70
RLT-7B (2-stage)
17K
26.70
84.00
41.40
50.70
We extended implementation analyze how scaling up student capabilities to compute the terms in the RLT reward affects the training of our new models. First, we considered ensembling both the 7B and the 32B students, averaging their probabilities, thus also introducing degree of diversity into the RLT reward. Second, we considered conducting training in two stages, each optimizing both teacher and student for half the total number of optimization steps. In particular, we paused the teacher RL optimization midway, distilled a student with this preliminary RLT's explanations, and resumed our teacher optimization for the remaining steps with this updated student compute the RLT reward. This second extension has the added benefit of better aligning the teacher rewards to the student's learning dynamics, and can considered an initial step toward online optimization of both models, which we believe to be an interesting future direction, as outlined in Section 6 As shown Table 12 above, both these extensions provide noticeable improvements the RLT distillation performance. Overall, these results, together with the ones showing the effects of teacher scale from Table further highlight the complementarity our new framework to harness future models and LLM advances across different beneficial axes.
22

---

Table 13: The effects of data size from training with RLT-generated data from different numbers of questions from Li al. 12 (1K and 17K) and its NuminaMath split [2
Evaluated model
Data size AIME 2024
MATH 500 GPQA Diamond Overall
Qwen2.5-7B-Instruus
N/A
10.00
74.20
33.30
39.20
Bespoke-7B-IK
IK
13.30
80.00
33.80
42.40
Bespoke-7B
17K
20.00
82.00
37.80
46.60
Bespoke-7B 7B (IK. NuminaMath)
1K
16.70
79.00
33.30
43,00
Bespoke-7B (10K NuminaMath)
10K
16.70
79.60
40.90
45,70
RLI-7B-IK
IK
20.00
80.40
41.90
47.40
RLT-7B
17K
23.30
82.80
42.40
49.50
RLT-7B (1K NuminaMath)
1K
16.70
80.40
40.40
45.80
RLT-7B (10K NuminaMath)
10K
20.00
81.00
43.40
48.10
9000
Number of think tokens
8000
Gureted 11 tranes
Rawer 78 ACT tracce
HBT ablit traces
7000
8000
4545.6
4000
3000
2000
1000
Pracessed traces
Full RET reward
No minimar returcionn
thought takens torm
Figure 8: Average number think tokens the reasoning traces generated by our baseline pipeline, Our 7B RLT, and other 7B teachers after ablating some of the key components in the RLT reward function.
C.5 Student performance and scale of RLT dataset
We collected additional results by changing the amount of student distillation data, only focusing on the NuminaMath dataset [52]. We constructed new datasets with 1 and 10K samples using the AIME, MATH, and Olympiads subsets of NuminaMath from Li al. [12], enabling us directly compare with the available reasoning traces from our best baseline using postprocessed reasoning traces from DeepSeek R1, used for Bespoke-7B. As shown in the [3]aove,,,, we find that the performance of the RLT and RI traces from the Bespoke pipeline positively scales with the amount of student training data. While we removed all science questions from STILL-2 present in our original data, we find performance across different reasoning domains. We note these results are consistent with our previous random subsampling of the questions from Li al. [12), with the RLT traces outperforming the postprocessed RI traces across all tested data sizes and configurations.
GPQA does not decrease substantially, evidencing positive transfer
D Extended analysis
D.I RLT reward and RI traces analysis
We analyze the empirical consequences of ablating the components in our RLT reward function, described in Section 3.3 In particular, we focus on the effects of removing two key components of design. First, we examine ablating the think tokens KL reward ,KL quantifying whether the
23

---

Table 14: Teachers ablated of individual reward components compared with RLTs and prior distillation pipelines on our main set of challenging reasoning tasks.
Model
Data size
AIME 2024 MATH 500 GPQA Diamond Overall
Owen2.5--BBInstten
N.A.
10.00
74.20
33.30
39.17
Bespoke-7B (R1 traces)
17K
20.00
82.00
37.80
46.60
Full RLT reward
17K
23.30
82.80
42.40
49.50
No thought tokens KL term
17K
6.70
63.80
31.80
34.10
No min/max reduction
17K
23.30
79.00
40.00
47.43
Postprocessed R1 reasoning distillation example
lepie
Twat cole co da ussistent ierolves thoroughly suploring question....
Pyshon generated glow proept...
rontning
<lbagin_ef_san ongggne
Chay, lat's aes. The peoblen to the aisimun nuztez of abeuts aveded peint It pagez double-sided. Boch abest cas hold two pagea. So, tut need rigere
Em., lot'u think about uome eumples. Like the ssaple ispal in 6, uxigus is Becouse divided 2 is 2.6, whish coundo up to 3, maybe the general
Veit, but vuis, fomre up to Sn that corract.
cleck unstasz cxenple. Suppeue 4. Ther, 4/2 exartly abouts. That makez sunue. 12 H zu then 1/2 0.5, which
staply cellingit
cospote
Python.
In Python, if you do teoeger dtstatee with H/, het shen you need to narnd up, pon can do OE 12 /f twb wew, For Weds OS +13//9 -5//2w0. Conavet. For 3-4: (0-1)/2-6/222 5, integus divasico siver 2. Mass us. Mask, 5/72 in Python iu 2. do 06411/72 la 5//2/2 Masch is soureci. pelt (111)//De). Corract. So pen. the formiia in
oongese (ll +13//9, corgur st. Testing the sapple tegee: tepes 5. /246)//246//240 conrect.
foncstion streightforrund. This the
So the code would be sceething 1ste::
tegas, coepote 01-13//2, print Bo the Pythos code would bei
Lnpptt peizet 413
But wait, paclupo laure's au vige care, Dec'u sen. acourding to comstesinto, at lows uo su problum.
thac's the antettinn The code steple. 200t read N, catentate 413//9, print.
<leoll.o_,sbogght
Iistilation sodution
<lbugin, aclatioul>
groblen, noed deternisu cisinus mabee ubeela of papez requiced print decamusi with pagez using Undilessided pristing.
held
The insighs here is that each theet of poger can hold tuo yageo. Therafore, the sixince trmber of ateste required in desercined by dividing efficiently computed using integer aritinstic by adding effectively up
Ipproarh
nubeer pagas by and rounding the nearest integez. appecact.
dividing
by 2.
pytbes
(1s5ingut Ergeniia 1/
1. wabteuiatoon The
Taputes The tepes in read as integer H. which represents the makker of pagee.
2" zemuiniur The the divinien casutia is annding up. to the sust shole munlez
before divinton that
of shaste meaded.
This spgrocch affict witty corputes the ramite, in conetaes timw, 001), waking it very afficient own the opper constrates Mast..
<lend,of_
Figure 9: Full formatted distillation data sample using a postprocessed DeepSeek RI reasoning trace from the Bespoke dataset [33..
think tokens fo, themselves interpretable logical continuations from the student's perspective as compared with the teacher's. s. Second, we examine ablating the min/max reductions terms, which serve ensure the rewards do not forego any individual token, avoiding introducing bias to the reward values based on the solution length or the number of think tokens in the teacher's explanations.
For these experiments, we train entirely new 7B RLTs following the hyperparameters from Table 3 but setting À = (see Equation 5) or a 0 (see Equations andd respectively for the thought tokens KI. reward and min/max reduction ablations. We then construct student distillation datasets, using the same full set of starting question-solution pairs as our recent state-of-the-ar baselines [ as considered in main Section experiments, We compare the generated datasets and students after

---

Raw 7B RLT distillation example
Ingie
<lsa start bappates
Tose rolk
(raysten
unscotent iorokves tauroughly saplecing questiot....
enmoutable Pyshon function geserated givm proept...
(1s and
nosening
<Ibogan, of. I>
Okay, let's probdex to sinimus neater of abeeta noeled to peins H pages double-sided. Book abees our hold two pageo, Bo,
The, stink short some ameples so. tike the expple tepat, to orsget te Tecaura dieided by 1x. 2.5, shicc rornde up to 3. mayhe the 'p-
general formll
Veit, but vals, Les check mntherr exeeple. Suppono 4. Ther. 4/2 is exacsly sheoto, Thes wakes sease. If R in thex 1/2 is 0.5, which
+ rounds up to 3. Bo that's oserect.
stsply caslingis
conpete
Python.
In Python, if you ds teseger atviation with I/, bet shen you need to rornd up. you can do ON + 13 21 2. Let'n ose. For Meb: 05 +13//9 -6//203. Cocruct. For pot: (0-1)/2-9/222.5 5, integez divisicn Rives 2. West, as. Mass, 5//2 in Python iu 2. do (6-11//2 La 5//2/2. Masck is correct. For pol: (1/1)//2/1. Cocruct. So yeu, the cormzia is
funcsion streightsorward. Dake the ingus ornguse +13//2, output it. Tweting the exppl tegee: tepus 5. (240)//266/2400 connect.
Zo the code vould be acesthing Lake:
ingus, coepute 0H-12/72, print the resuls.
Bu the Pytbus code would be:
Lsscinput (2)
peicst
but wait, perbapa taaze's an vdge sure. Leots sus. For acorrding to comsteuints, at loast uo au probium. Tex, that's the zolctten. The code steple. 2ust rend N. calculate (Nl +1)//2, print.
<leud,of, Lbonghul>
Dieiiaation solition
<lbegin_ef, of >sluttoo/>>
paper regitied print
pager uning
printing. share sheat hold pager.
typreach
The amsigh beee is thit eush abeut of puper can bold tup paavu. Iberufore, the misinse nambes of abeetu zequized detersined dividins meber pages by and romnding up to the nearact integer, This can atficisntly corpured uxing integer aritimeste by wdstng This uppcouch
pages peztorming
elleetivelyy up divading
by 2:
*e* rytbin
grtent
// 2)
walaacing Imputes: The tegee ds read as integer H. which represente the mamber pagse. enlatione:: csnsinius Leua the divinion cesulia is conniing up to the aust shole muasez ot abeuta ateded.
2"
before disttttion that
This apgeonch effiolently computes the resolt in conatess timo, 001), making it very officisnt eves for the apper constreias limit.
<luad, aoweetii thd
Figure 10: Full formatted distillation data sample using the raw reasoning trace from our 7B RLT.
distillation with these ablated teachers to our original 7B RLT trained with the full RLT reward and
our strongest distillation baseline using postprocessed RI reasoning traces [12 [3]
First, we focus on the effects of these ablations in terms of the traces' content as compared to our original RLT's traces and the postprocessed R1 pipeline. As shown in Figure g the length of the reasoning traces is greatly affected by our ablations. When looking at our original RLT with full rewards, the average length of the produced reasoning trace is 39% higher than the original curated RI traces in the Bespoke dataset [13. This is consistent with our analysis from Section 4.5 and further highlighted by the full examples shown in Figures and showing how our 7B RLT often includes alternative verification steps and approaches not considered by traditional pipelines that do not optimize directly for downstream distillation. Furthermore, ablating each term in our reward leads to the concrete unwarranted effects described in Section [.3] allowing the RI. optimization procedure to find "shortcuts" to maximize ,RLT that hurt the quality of the reasoning traces. In particular, without the thought tokens KL reward KL our RLT cannot differentiate between explanations that guide the student step-by-step and those that increase the solution's likelihood without a logical path that can be learned from. Thus, as shown in Figure [T] this ablation leads to teacher that only learns to repeat the solution tokens themselves its explanation to exploit the repetition tendency of pretrained student LMs, with the average length of its output reasoning traces dramatically dropping. Moreover, our second ablation of the min/max reward reduction terms, the rewards become effectively biased by the length of the reasoning trace, leading the teacher to prefer long explanations only to reduce the influence on KL of hard but necessary individual logical steps. As a consequence of this bias, the average number of think tokens in the teacher's 's reasoning traces almost doubles from
25

---

Thought tokens KI. reward ablation distillation example
lopie
<lta, start bayatas
Tuse tolk
(raysue
ussiotent ierokves tauroughly saplecing questiot....
ennortable Pyshon function geserated givm proept...
(1sx, rnd
<bbegan, 1
groblen,
doternise huld pages.
mober sheeto of paper reguiced print documess pager uning drchle-side
pristing.
The innight hare is. that each theet of peger can hold tuo pegne. Therafore, the stxince mamher of abeats required in desereined by dividing nober pages by and romdinn up to neareat integer, This can afficimnlly corputed using integer aritherste by adeing appeoect.
Ipprearh
iffectively up
dividing
by 2:
#63 Bolation p/Lbee (escinput))
prassiia
Taputees The tegus in read as Integer N. chiic represente maker pegre. idting before divisien that
waslouistonee:: The
2' zensinder feom the diviatio resuita is couning up to the sest shele maneer sharts Seaded.
ststeue
This appeveck eifacientiy computea the resuls in cocatess taw, 0(1), muking it very ufficisnt eves for the upper consteniat limit.
thought>>
(bagtn.offesturtmm) mn
solne this problem, need to dateratee the nistese memher abeats of + pristing. wbece wach short cea hold two pages.
docrment with
desble-sided
The 2ay insiate bace that each shret paper can hold tup zagen- Therstore, efficlently computed aritteetic
namber abeets zequized determined dividias
ste omber pager before gorforming latagor divinion by 2, This apprssst: vorks becanse it effectively rounde up the result of dividing by
#68 Solation Gode
(ipput)))) pejea((s
*en Rep' anactin Peoiooolaiionn The Cormila (S 1) /f 2 is noed to conput the siximu nuaker of steets, Adling to before divinion enouces that any
Tnputes: The
chich
namber
+edetpures: The varait ie peinted, which the afstnum memser of chasce neaded.
edficiently computea
very
the upper
Full formatted distillation data sample using a reasoning trace collected after training our teacher ablated from the thought tokens KL reward ,k in the RLT rewards defined in Equation 5
Figure 11:
the postprocessed RI traces and, as shown in Figure 12
their content starts including many additional unnecessary steps that are just semantical repetitions of each other as learning progresses.
Then, we also quantify and compare the effects that each of our ablations has on downstream student performance. As shown in Table [4] the disruptive effects of ablating the thought tokens KL reward entirely reflect on the capabilities of the learned students, with their performance being lower than even the original Qwen-7B model they are fine-tuned from. This result validates our reward design, showing how regularizing for the reasoning traces to be "natural" continuations from the student's 's own perspective is of key importance for effecttve distillation. On the other hand, ablating the min/max reward reduction terms produces a more moderate reduction in performance, with the new output traces of this 7B teacher still remarkably outperforming our strongest baseline pipeline using the RI LM with orders of magnitude more parameters. However, we note that by preventing the increase in the lengths of the reasoning traces with the min/max reduction terms, our full RLT reward also yields faster training, distillation dataset generation, and student fine-tuning, with non-trivial benefits contributing to our framework's efficiency.
E Extended discussion and limitations
E.1 Traditional RL and the importance of overcoming reward sparsity
In traditional RL for robotics, neural network policies can learn to solve tasks from random initializa- tions thanks to being guided by dense reward functions designed and tuned to provide domain-specific guidance on the level of progress [53] [54]. Without this guidance, they would be faced with an infeasible exploration challenge made exponentially more difficult by the task horizon (55). This is
26

---

because dense rewards allow the policy gradient optimization to rank the relative progress obtained across its initial suboptimal actions, allowing the policy to bootstrap from partial solutions and extrapolate far beyond its initial capabilities. However, in the context of traditional LM reasoning, as correctness-basee based rewards are inherently sparse, this extrapolation is not possible. In particular, an LM is too weak to provide the optimal solution for a task, all its rewards (and, thus, the policy gradient) will be zero. In contrast, we note this limitation would not apply to our RLT, which can make use of a dense reward function and obtain a learning signal to iteratively improve even when all sampled initial explanations are suboptimal, as with the traditional RL framework.
E.2 Limitations and unexplored directions
This work's purpose was to introduce a new class of Reinforcement- Learned Teachers designed to avoid the exploration challenge of sparse rewards and align the optimization of RL-trained LMs with the test-time goal of downstream distillation. However, there are still several limitations and improvements which we hope will be tackled in future extensions. First, output explanations, the RLT framework relies on access to the ground-truth solutions. Hence, when used with datasets and domains where this information might not be available or not be practical to recover without querying LMs, small RLTs might have to again rely on larger models, even though still to a lesser extent than prior distillation pipelines. Similarly, as described in Section 4.1 and Appendix A our current training recipe does not exclusively involve RL, with an initial phase to familiarize our small model with its new teaching format and role, again relying on some level of initial access to collecee reasoning examples that can be used accordingly. We also note that RLT training makes use of an additional student model for computing the rewards. In practice, however, we note the downsides from this were minimal, as simple parameter offloading removed any potential memory burden of having this model on GPU memory during backpropagation, and actual training time was dominated (> 90%) by the costs of long-context autoregressive generation with our computational setup. Recent work has shown early promise in overcoming this inherent bottleneck of RL methods by grounding the model's 's reasoning process away from self-geenratee tokens, and using separate spaces such as diffusion [56. Yet, while this complementary approach can be more efficient, it has yet to have the same impact as RL reasoning at larger scales.
As described in Section we also did not explore the potential of sharing the teacher and student roles with the same optimized model, nor concurrent training of the two. This leaves two open questions yet unanswered: whether there can be an effective transfer between the two roles, potentially improving training efficiency by harnessing the relatedness of their objectives, and whether RL. can devise online curricula tailored to the student's 's specific learning dynamics. Furthermore, due to computational constraints, we had to limit the maximum context size during our RL-training and student distillation phases to 16384, half the maximum context available with Qwen-based models. For the same reason, the considered RLTs only comprised small, inexpensive 7B models and did not consider further scaling. Breaking these constraints, thus, also remains another outstanding immediate direction for future improvements and further pushing the capabilities of our framework. Lastly, this work did not explore increasing the number and breadth of starting distillation datasets, beyond the ones considered in prior work that made use of much more expensive pipelines. This leaves the potential extent of another key feature of our method unexplored, as with our small 7B RLT, it could be much more feasible cheaply collect data for student distillation that matches the level of scale and efficacy of dlosedssuuree state-of-the-an data sources such as the one from Guo et al.
cersstion-souttion pairs used for obtaining
E.3 Broader impact
Our work introduced new class of models that enable small LMs to generate better synthetic distillation datasets beyond prior, much more expensive pipelines. Rather than introducing new application, this work's contribution was foundational in nature, thus, its broader implications are bound to the effect of improving the capabilities and democratizing the training of large language models. To this end, as the accessibility and capabilities of LMs improve, there is an increasing chance of misuse for potentially harmful goals, such as influencing public opinion or obtaining access to sensitive information. Moreover, with increasing demand, the carbon footprint of LMs and their potential social impact might become increasingly relevant. However, we believe these risks are currently offset by the potential upsides of advancing the Al field, such as empowering humanity to better face upcoming environmental and economic challenges.
27

---

Min/max reduction reward ablation distillation example
Disiilation
<lbegtn_of__toeee I>
Okay, weed in this problum wiere
vasia print pager dusble-aidedd wilh eoch atrut bolding two pages, The task to Let's thine through this by
+ ststere namber chesse
stts double-sided,
orranged offects how Shey Ttb, last, so, dolieesidde printing ing just moans that too pages printed the onee shset, fcons and lork, de
for ecuppla, zare pare on the tront and page 2 on the back, shat's eze sheet. Bot if there's first
maber pages, the sbeet seciod,
ispat those stird, de each abees excrps potocbly the lest con have pages.
For pages, the mmbbrr of to the totel mraber of abeeta would be (5 // + E1 it Ht 2 slze ot. Vuit, but in the sumple inges, 5. 5//2 in 2, and 612 in en tel, Which ratcher the exeple anteet. in that corvect.
the
calestate hes
w//2. net ie ald, there'e wetra
4. Then 4//2 in 2, and 422 in O. Bo 2 O 2. Which wolld be twp abeeta. Each shrot hew to tro pages. Libe 1-2 in stret, 3-4 on sacond. statts correct.
Vuit, bat vals. Les's chrck anotier examgl.. It
another exteple: ped, 3/22 in t, a20 in to So 1+502. So san theete. Like 1-2 on first, 3 on the eacond. Cirrect. theat. tet thatle tagoortble backezs sheet, Neak, but uvcarding to the pooblea statemes, the comstriint way thet in at lsust 1. Bat the probles suss thes two peavo can be du for dei, the usever La beceuse taaze's only page, su sua't be have use page. But is that ceas, last's io ulloved Wass, but for single pags, you cua't it peint touble-zided 20 murte the probiem's 's leve H-1, accordins to the compiciiats givm bers. pertape Thes prace, the shout use 5//2 in lhich to otd. the fermals (1-12//2. Wast, conrect, For Ded, 3rsms, 4//202, Correct. for tei, 111e2, 2//201, to chat correct. Wast, 1-1e2, 2//20i. do the fornia de (I 1) // 2.
What about He1? Then 1//2 te r8p0), So nitel.
can't peint stiegle
derbile-etdde
peiateed ou sestded
io there dosble-aidedd Weit, wais. peoblem meber
tast will
constzninia stateennt,
probiom allows 8-1. Bet uccording to the 300. rnderrttddingg the problee.
shatle iaponetble. abeet hold pages, supple isput which works ozt to sheete, do eaybe
for
Wuit, probleen
the peoblen
Vait, for 1-1: (1-13//22 L. Which coreect. For 192: (2/12/2e1. 5, integer division in 1. Maich correct. For Rhd: (3/2)/2x2, Dorrect. 1o4: connect. leit, het sate, Eatte see. For tel, we use 08-13/25, that's (1-1)//201, So yea. the fomata Of 433//2- Bet uxit, in the prablen (543)/2263,
61/500
gloee tn statieent, the sspple whitch strided by gives tnsnger stvistor correct. the correct nember
of 113//2.
aditng give
nomber
agsin,
(213)//241
to yes,
would (1-13//22
cocmulat
holds plua one if there's ond samber. do the of puico
pousitly
Bo
int 11/2, and is 9 is ood, thecela ta us uxtev shaet, so the votel is (M/22) (1 if Il 32 t-0 slae o1. Which ous be wrsstvs us de11//2.
totel aumber abeets the aumber of peicu feuch pair
Teu. Beceuse Jor y even, 01432/2 is (8/2) +0.5 but integer divisien wolld be 3/2, For esupli,, Hik: (4+3)2242.6-2 Which in corrvet. Por 1-6: Sopp (5-1)/23d. do thec'e correct.
So the formuin is +17 //
(1-13/2-1. Curcuct. H-2: (241)/241.611 Ourrect. 3-d: (3-11/2-2. Oocsuct. Bo4;
Hr5:3, Doenuct.
sorse.
So the seoblex reduces to calcuiating (8 //
But waik, the ispes is H, which in betuves i and 100. Bo the Fanction needa to avad 3 teem at standard ixpes, compute (H-13//2, and cutput that. Vuil, bat vall. Lus's is take 1-1, The supple iugu in 6, whish gives cutpoi 3, Pur H-t, the outgut would by 1. Bas occording to the probles tut stateeent, the cinseal mebher cheete. do for Hel, you need sheet, Bacauze aven thonge you can't fill fell sbeat, you pust hare to una wacks.
in stroightfervedd Bend R, conpute O8-1//72, print shat.
Vuit, bas let zs theek usother sxmple.. Les's say Sid. (6/1)//2-0 161. Mait, 5//205: Coeruct. Bo that's correct. vate, but mate. last, Vatt, (0-13//2 se 1//200. Natt, sterded by 3.5, tateger stviaton 3. (613)//2 de 3.5 i. Se Bo - pages would zuqaize alevia. Buch aboet has two pages Bu 1-2, 9-4, 6-6. Bo thew shaute. Cucseet. cailing divided 2. Nhich in +11//2.
to the in staple reading then the Pyshon tinction unild bwr
from inpui, This copptte i8 +5) /1 Peiss thes cesuls. whil,, but saie, Leste tess led again. (1433//2e1. Correct. The. in the code in straagerfrreett.
Bu the code would be:
Real integer froe isput. Then print Of 13 2. But wsit, vuis, Les's check (242)//241. the For 3n5:3. Cuzcees. For 1e5:3. Ooenect.
For Me2: (2-13//2211. Bo 1 sheet. Maick is correct. For B-3: abeets. Corruct. For B:4: 2.
Thecotore, Pythou fusotion
from sepes. thes compute C8 +2) // snd peint 18. So the zode soild ba:
Woit, vois. me think again, For Hrt, exswez 1. For N-g, Bo thas'e is correct. do the code corzecs. But wait, the problem suys that in the number of pagen. Bo thst's correct. the code is simple as that. I chink thac'e st. So the antoor te to conpate (1 -13 // 2. <land,ot, ot ttoogght Distilation solution
<lbugin, pelatioul>
tolce this groblem, need to deteratea the mistese mibhar sheete of paper reguired to print docimert with pager uning denble-at pages.
ineighe hare that each sheat of pagar con hold tup pagne- Therafora, the sistere rabbrr shaets regetred deteretned diristig tntager.
papes by the nadber by
acficiantly coeputtd using bufore pezlomnng divinion by This uppcouch vuska it silvelively cvunda the soalt ei divading
by seiing
*e* Botution Dode
Fejan 11 7/ 2)
se* anation
as 2: cuiiitittteee The formila is 1) 2 it Seed to compote stsinue nember of Stasts, aditng to before diviaton ansacee that any
Inputer
reprenents
3. -iolppass, The swalt is pe peanted, shich as tis aitinae nusier of aleevsa avedel.
mumser
This apgeosch effiolently conputes the resule in conatass time, O013, making very officient eves for She spper coostreias limit.
Figure 12: Full formatted distillation data sample using a reasoning trace collected after training our teacher ablated from the min/max reduction term in the RLT rewards components defined in Equations 4 and System prompt and Assistant message omitted for space.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our experiments in Section 4 validate the superiority of our new framework compared to prior reasoning distillation pipelines, and its claimed properties.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[Yes]
Justification:
We discuss unexplored directions in Section 6 and provide further discussion of the limitations related to computational constraints in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
29

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Details and hyperpurameters for experiments and tasks provided in Section 4 and Appendices B C and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
30

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Full code is provided for re-training our models, and also provide open checkpoints to facilitate reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
poplic/guides. wtt.///euuuiiiissooooo......
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpss.ccppubblc gwwdeg/ddeeeee wtw.oodeeSubmmsssoooooo.ooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The main details and hyperparameters for training and our evaluation settings are provided in Sections 3 and 4.1 Remaining implementation details and hyperpa- rameters are provided in Appendices and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the computational cost of training large language models, performed training each of our models only once, in line with the prior work. Thus, we could not report error bars for our main experiments. However, do report some statistical measures (c.g., Pearson correlation coefficient) for our analysis in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
31

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details about our compute resources in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttp::/neeurpp:: http.////c....ccccciieeeeeeeeeee
Answer: [Yes]
Justification: There is no violation with the code of ethics as the nature of this submission focuses on introducing and evaluating, using existing benchmarks, a new methodology.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: While this work focuses on foundational research, we still discuss the potential impact of future advancements and misuses of our framework Section E
Guidelines:
The answer NA means that there is no societal impact of the work performed.
32

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release large-scale models or data with misuse potential.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credit the models, frameworks, and datasets used for training and evalua- tion.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
33

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide instructions together with our training and evaluation code to ensure reproducibility.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
34

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
35


Reviews:
[{"review_id": "A2NDny5yR6", "reviewer": "Reviewer_YrSD", "summary": "The authors introduce the RLT paradigm where the teacher's task is to simply provide explanations for a student given a question-answer pair, rather than prior approaches where the teacher takes in only the question to give out explanations and the solutions. This approach is intuitive and cleverly circumvents the exploration problem from correctness-based sparse rewards that are typically used. The authors demonstrate multiple use cases and advantages of the RLT paradigm: efficiently training larger students with a smaller teacher model, showing out-of-domain transfer for the teaching task, applying RLT to cold-start future RL iterations. The RLT paradigm is instantiated in the GRPO setting where the sparse based reward is a replaced with a dense-reward modelling two aspects: student understanding of the the solution given the question, and agreement of the teacher logical explanation under the student model and teacher model.", "questions": "1. The authors state that the paradigm of solely correctness-based rewards does not allow the model to explore beyond its initial latent abilities (line 38). With the RLT's dense reward paradigm, can there be exploration beyond the initial LM's latent abilities? To me, this answer still seems to be \"no\" -- but since the teacher does not have to solve the task, there are no sparse-reward problems. If this is indeed the case, the authors must clarify this in the paper.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The RLT paradigm cleverly avoids the exploration problem by modeling the teaching task with a dense-reward objective, rather than the usual correctness based rewards that are sparse.\n2. Intuitively, the task of providing an explanation given a question-answer pair seems much simpler and makes a lot of sense. This is reflected in a smaller teacher able to effectively train a much larger student. \n3. Out-of-domain zero-shot RLT outperforming direct RL on the countdown task is a strong and interesting result.\n4. The authors also provide the code and many details of their approach in the paper, so the work seems to be reproducible.\n\nWeaknesses:\n1. The authors state that the RLT framework can be used with any RL algorithm, but the experiments performed solely use GRPO. Although I find this believable, the paper could benefit from an experiment to demonstrate and verify this claim.\n2. The authors do not discuss what happens in cases where there is a large mismatch in the abilities of the teacher and student LM. For example, a small teacher LM might output explanations that are not sound (while still maintaining a high logprob under its own distribution). This can be exacerbated in scenarios where the student LM is much larger (or a same-size student that is trained to solve a domain extremely well). Will the RLT paradigm then resort to reward-hacking (i.e., will the student over-trust the teacher's explanations regardless of its own initial abilities)? An ablation on \"small teacher LM\", with increasingly more capable student LMs (beyond 32B) would greatly improve the analysis of the paper.\n3. In (i) of section 3.3, the authors state: _This first reward term is computed with the student’s log probabilities over the solution tokens, reduced with both average and minimum operations_ and then provide equation 3 which seems to imply that the reward is the average log prob + $\\alpha$ min logprob. From the text, I interpreted that the logprobs are normalized (divided by) the avg and min logprobs. This hinders clarity. Similar comments apply to also (ii) in section 3.3, and equation 4. The authors can consider modifying text to improve clarity.\n\nOverall, I remain positive about the paper and believe it provides many new insights.", "was_revised": true, "final_justification": "I have read the rebuttal and my questions/concerns have been sufficiently addressed. Since my original score was already high and recommending acceptance, I will keep my score (5).", "rebuttal": "We would like to thank Reviewer YrSD for their feedback and the time they dedicated to our review. We worked to address each of their comments and hope they will not hesitate to let us know of any further suggestions or questions.\n\n**weaknesses**\n\n> 1) The authors state that the RLT framework can be used with any RL algorithm, but the experiments performed solely use GRPO. Although I find this believable, the paper could benefit from an experiment to demonstrate and verify this claim.\n\nFollowing the reviewer’s suggestion, we extended our implementation and added a new set of experiments training our 7B RLT model using the RLOO [1] algorithm instead of GRPO [2]. We added these new results and analysis to Appendix C, which we summarize below: \n\n|Evaluated model|Teacher model|AIME 2024|MATH 500|GPQA Diamond|Overall|\n|-|-|-|-|-|-|\n|Qwen2.5-7B-Instruct|N/A|10.0|74.2|33.3|39.2|\n|Bespoke-7B|DeepSeek R1|20.0|82.0|37.8|46.6|\n|RLT-7B|RLT teacher (GRPO)|23.3|82.8|42.4|49.5|\n|RLT-7B (RLOO teacher)|RLT teacher (RLOO)|20.0|83.6|42.9|48.8|\n\nIn line with recent empirical findings about the similar empirical effectiveness of the different reasoning RL algorithms [3], the performance of our RLT trained with RLOO appears very close to our previous results obtained using GRPO. We believe the small gap is mostly due to the fact that we did not re-adjust any of the main hyperparameters from our GRPO implementation (e.g., batch size, number of generations per question). We hope these new results will provide concrete evidence confirming that the RLT paradigm's performance is not tied to any specific RL algorithm.\n\n> 2) The authors do not discuss what happens in cases where there is a large mismatch in the abilities of the teacher and student LM [...] An ablation on \"small teacher LM\", with increasingly more capable student LMs (beyond 32B) would greatly improve the analysis of the paper.\n\nWhile our current experiments do show that even with a significantly smaller 7B teacher, our framework remains effective for teaching a much larger student, we understand the reviewer’s interest in examining the effect of pushing the gap between teacher and student to the extreme. However, while we ran all our experiments on a single H100 node (8 GPUs), we note that training models beyond 32B would require multi-node training and be computationally too expensive for this project’s resources.\n\nThus, to still try to address the reviewer’s suggestion, we instead focused on further decreasing the size of the RLT teacher down to 3B parameters. We added these new results to Appendix D of our current revision, which we summarize in the Table below:\n\n|Evaluated model|Teacher model|AIME 2024|MATH 500|GPQA Diamond|Overall|\n|-|-|-|-|-|-|\n|Qwen2.5-7B-Instruct|N/A|10.0|74.2|33.3|39.2|\n|RLT-7B|RLT teacher(7B)|23.3|82.8|42.4|49.5|\n|RLT-7B(3B teacher)|RLT teacher(3B)|20.0|80.6|38.9|46.5|\n|Qwen2.5-32B-Instruct|N/A|26.7|84.0|49.0|53.2|\n|RLT-32B|RLT teacher(7B)|66.7|93.4|59.6|73.2|\n|RLT-32B(3B teacher)|RLT teacher(3B)|46.7|91.4|53.5|63.9|\n\nAs shown by our new set of results, we expectedly find that larger teachers yield better explanations and downstream results. However, we find that most of the performance gap between our 3B and 7B teachers only occurs when distilling the 32B student LM, where the mismatch between teacher and student capabilities tends to the extreme. However, even in this case where the student is over 10x larger than the teacher, our 3B RLT is still able to provide considerable improvements to the initial Qwen 32B student.\n\nThe reason that we did not observe performance degradation appears to be due to the role of the $r^{KL}$ reward term plays during the optimization. Even in cases where the initial 3B teacher is unable to provide logical explanations to a specific question and answer, optimizing $r^{KL}$ will naturally guide the teacher's output distribution (with both question and answer in context) toward converging to what was most likely from the student's distribution (with only the question in context). We added in Appendix D an analysis of this phenomenon, which we validated by inspecting at different training stages the 3B teacher's output explanations for challenging questions with low values of $r^{SS}$.\n\nWe believe these results further validate how the RLT framework enables even small, inexpensive models to be effective teachers, but also confirm that larger teacher models are indeed able to provide better explanations, highlighting the potential future implications of scaling with more training compute. \n\nWe are open to collecting additional results for future revisions of our work by pushing the gap between teacher and student even further by training and evaluating a 0.5B teacher model, in case the reviewer finds this further necessary. \n\n> 3) In (i) of section 3.3, [...] I interpreted that the logprobs are normalized (divided by) the avg and min logprobs. [...] Similar comments apply to also (ii) in section 3.3.\n\nFollowing the reviewer’s feedback, we went through the paper and extended the text in several cases where we previously used concise language (for saving space) to improve clarity. For example, in the specific instances flagged by the reviewer, we rewrote lines 175-176 and 178-181 to:\n\n“This first reward term is computed with the student’s log probabilities over the solution tokens. We reduce this vector to a scalar by applying both average and minimum operations over the different per-token values:” (ex lines 175-176)\n\n“This second reward term is computed with the KL divergence over the same think tokens [...]. \nSimilarly to $r^{SS}$, we reduce this vector to a scalar by applying both average and maximum operations over the different per-token values:” (ex lines 178-181)\n\n**questions**\n\n> 1) The authors state that the paradigm of solely correctness-based rewards does not allow the model to explore beyond its initial latent abilities (line 38). With the RLT's dense reward paradigm, can there be exploration beyond the initial LM's latent abilities? To me, this answer still seems to be \"no\" -- but since the teacher does not have to solve the task, there are no sparse-reward problems [...]\n\nFollowing the reviewer's question, we realized that the term \"exploration\" could have different interpretations and added an explicit explanation in Section 1 and Appendix E for our statement that “correctness-based rewards do not allow the model to explore beyond its initial latent abilities,” which we summarize below:\n\nIn traditional RL for robotics, neural network policies can learn to solve tasks from random initializations thanks to being guided by dense reward functions. Without this guidance, they would be faced with an infeasible exploration challenge made exponentially more difficult by the task horizon [4]. This is because dense rewards allow the policy gradient optimization to rank the relative progress obtained across its initial suboptimal actions, allowing the policy to bootstrap from partial solutions and extrapolate far beyond its initial capabilities.\n\nHowever, in the context of LM reasoning, as correctness-based rewards are inherently sparse, this extrapolation is not possible. In particular, if an LM is too weak to provide the optimal solution for a task, all its rewards (and, thus, the policy gradient) will be zero. In contrast, this limitation would not apply to our RLT, which can make use of a dense reward function and obtain a learning signal to iteratively improve even when all sampled initial explanations are suboptimal, as with the traditional RL framework.\n\nWe hope this new discussion will clarify what we mean by “extrapolation beyond the LM’s initial latent abilities” (lines 37-38) and hope the reviewer will not hesitate to let us know if they believe anything else requires additional contextualization.\n\n**references**\n\n[1] Ahmadian, Arash, et al. \"Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.\"\n\n[2] Shao, Zhihong, et al. \"Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\"\n\n[3] Hu, Jian, et al. \"Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models.\"\n\n[4] Devidze, Rati, et al. \"Explicable reward design for reinforcement learning agents.\""}, {"review_id": "Xhf98jxxJA", "reviewer": "Reviewer_Esvg", "summary": "This paper addresses the current challenges in reinforcement learning for training LLMs on reasoning tasks, such as exploration difficulties and poor distillation performance, by proposing a novel Reinforcement-Learned Teachers (RLTs) framework. \nUnlike traditional RL models that solve problems from scratch, RLTs are explicitly provided with problems and their corresponding answers during training, generating explanatory processes that are pedagogically valuable for student models. The framework is optimized based on feedback from the student models.", "questions": "1. There appears to be a disconnect between the paper's core contribution and its title. From my perspective, the work primarily addresses the challenges of cold starts and sparse rewards in reinforcement learning. Could you please clarify the connection between these aspects and the concept of \"Test-Time Scaling\" as highlighted in the title?\n\n2. Could you provide a more detailed explanation as to why the proposed method is considered scalable? What specific properties or mechanisms enable its scalability?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The proposed method effectively addresses the critical problem of sparse rewards in reinforcement learning. A significant advantage is its compatibility with any existing RL algorithm, making it a flexible and practical contribution to the field. \n\n\n\nWeakness\n1. The RLT teacher model requires the standard answer to each problem as input in order to generate explanations. This means the method cannot be directly applied to unknown problems or real-world reasoning tasks without provided answers. During the testing phase, the student model still relies on explanations from the teacher to produce answers, lacking independent problem-solving ability. Additionally, the teacher's explanations may already include the answer, effectively revealing the solution to the student model—naturally leading to better performance.\n2. Although the authors designed a reward function based on student model's feedback to optimize the teacher model, the training process remains a one-way offline optimization. The student model remains fixed during the teacher's training phase and cannot dynamically adapt or learn based on the explanations generated by the teacher. This may lead to issues such as the reward function overfitting to the initial parameters of a specific student or the teacher's training objectives failing to align in real time with the student's learning curve.\n3.  No  ablation study was conducted to evaluate the independent contributions and sensitivity of different reward terms(r^SS vs. r^KL)", "was_revised": true, "final_justification": "I previously had some misunderstandings about the role mechanism of teachers during the testing phase. I appreciate the author's response, which has now clarified the issue. I will improve my rating", "rebuttal": "We would like to thank Reviewer Esvg for their feedback and the time they dedicated to our review. We worked to address each of their comments and hope they will not hesitate to let us know of any further suggestions or questions.\n\n**weaknesses**\n\n> 1) During the testing phase, the student model still relies on explanations from the teacher to produce answers, lacking independent problem-solving ability. Additionally, the teacher's explanations may already include the answer, effectively revealing the solution to the student model—naturally leading to better performance.\n\nWe would like to clarify that the teacher’s explanations are only used during the training of our students. During testing, our student models do not rely in any way on the explanation produced by the teacher and can effectively solve problems from scratch independently. In fact, our procedure for distilling and testing our students is exactly the same as prior distillation approaches [1, 2]. The only difference is how we obtain the distillation dataset: \n- Prior approaches [1, 2] using expensive LLMs not only make use of the ground-truth solutions to verify the correctness of their reasoning traces but also rely on filtering and post-processing stages to improve the quality of their distillation data.\n- In contrast, we show RLTs are more effective without requiring any filtering and post-processing stages, using the raw outputs from orders-of-magnitude smaller 7B teacher LMs to generate the distillation data.\n\nFollowing the reviewer’s feedback, we expanded the text in Section 4.1 by including the above explicit comparison with the traditional RL distillation pipeline, which we hope will make it clearer to readers that our RLT framework actually has lower requirements and that our students do possess stronger independent problem-solving ability.\n\n> 2) [...] The student model remains fixed during the teacher's training [...] This may lead to issues such as the reward function overfitting [...] or failing to align in real time [...]\n\nThroughout our experiments in Section 4, we found that keeping the student model fixed was sufficient for our 7B teacher to outperform distillation/cold-starting with LLMs such as DeepSeek R1. While optimizing both models at the same time would be more costly in terms of computation, we do agree with the reviewer that concurrent optimization is a potential further direction to scale up our framework. In fact, we mention this exact idea when discussing future extensions in Section 6:\n\n“One example [of future directions] is training RLTs and their students in tandem, allowing the teacher's explanations to adapt to the student's learning dynamics live” (lines 361-363).\n\nNonetheless, to still try to address the reviewer’s feedback, we collected and analyzed two new sets of more expensive experiments in Appendix D, taking a first step in this direction:\n1. First, we considered conducting training in two phases, each optimizing both teacher and student for half the total number of optimization steps. In particular, we paused the teacher's RL optimization midway, distilled a student with this preliminary RLT’s explanations, and resumed our teacher optimization for the remaining steps with this updated student to compute the RLT rewards. The objective of this experiment is precisely to lessen the risk of potential overfitting to the student’s initial parameters, as suggested by the reviewer.\n2. Second, we also considered using both the 7B and the 32B students for computing the RLT reward, averaging their probabilities. The objective of this experiment was instead to lessen the risk of potential overfitting to a single model.\n\n|Evaluated model|Teacher|AIME|MATH 500|GPQA|Overall|\n|-|-|-|-|-|-|\n|Qwen2.5-7B|N/A|10.0|74.2|33.3|39.2|\n|Bespoke-7B|DeepSeek R1|20.0|82.0|37.8|46.6|\n|RLT-7B|RLT teacher|23.3|82.8|42.4|49.5|\n|RLT-7B(2-stage teacher)|RLT teacher(2-stage training)|26.7|84.0|41.4|50.7|\n|RLT-7B(7B + 32B $r^{RLT}$)|RLT teacher(7B + 32B $r^{RLT}$)|23.3|83.8|41.9|49.7|\n\nAs summarized in the Table above, both these extensions provide some moderate additional improvement to the RLT's distillation performance. However, we also note that these improvements came at a substantial additional cost, taking approximately 1.8x and 1.4x more GPU hours than our original RL training pipeline.\n\n> 3) No ablation study was conducted to evaluate the independent contributions and sensitivity of different reward terms.\n\nWe would like to clarify that we did perform an in-depth ablation of our reward terms in Section D of our Appendix:\n1. First, we examined ablating the think tokens KL reward term $r^{KL}$, quantifying whether the think tokens $t_{o_i}$ themselves are interpretable logical continuations from the student's perspective.\n2. Second, we examined ablating the min/max reductions terms, which serve to ensure the rewards do not forego any individual token, as detailed in Section 3.3.\n\nWe provided these results in Table 7 and summarize them again in the Table below for easier reference:\n\n|Evaluated model|AIME|MATH 500|GPQA|Overall|\n|-|-|-|-|-|\n|Qwen2.5-7B|10.0|74.2|33.3|39.2|\n|Bespoke-7B(R1 traces)|20.0|82.0|37.8|46.6|\n|Full RLT reward|23.3|82.8|42.4|49.5|\n|No thought tokens KL term $r^{KL}$|6.7|63.8|31.8|34.1|\n|No min/max reward reduction|23.3|79.0|40.0|47.4|\n\nFurthermore, beyond performance, in Figures 8-12, we also compared the lengths of the produced teacher explanations and provided several qualitative, concrete examples showing the effects of each reward term. As detailed in Appendix D.1, ablating $r^{KL}$ leads to a teacher that only learns to repeat the solution tokens themselves in its explanation to exploit the repetition tendency of pretrained student LMs, with the distillation performance and average length of its output reasoning traces dramatically dropping. Furthermore, ablating the min/max reward reduction terms makes the rewards effectively biased by the length of the reasoning trace. While this second ablation only minimally influences performance, it makes the number of think tokens in the teacher's reasoning traces almost double, unnecessarily increasing the downstream student distillation cost.\n\nFollowing the reviewer’s feedback, we extended lines 198-199 at the end of Section 3.3 to reference more explicitly the presence of this set of important results that validate our reward design, and avoid readers potentially missing them:\n\n“For further discussion, we refer to Appendix D, where we empirically analyze and validate each design choice in our reward function in terms of final performance (Table 7) and concrete qualitative differences of the resulting explanations (Figures 8-11).”\n\n**questions**\n\n> 1) Could you please clarify the connection between these aspects and the concept of \"Test-Time Scaling\" as highlighted in the title?\n\nBy “test-time scaling,” we are referring to the RLT’s ability to produce explanations for training students and providing them the ability to use additional compute to generate lengthy reasoning traces and improve performance. We note that this same nomenclature is used in the  abstracts, introductions, and titles of many prior important papers in the LM post-training field, such as DeepSeek-R1 [3], THUDM’s T1 [4], and \"s1: Simple test-time scaling.\" [5].\n\nFollowing the reviewer’s comment, we tried making the connection more evident for our readers by modifying lines 26-29 and adding direct citations to the above example works:\n\n“With the rise of RL for open-ended reasoning (RL reasoning), inducing a new form of language model (LM) test-time scaling [3, 4, 5]...”\n\n > 2) Could you provide a more detailed explanation as to why the proposed method is considered scalable? What specific properties or mechanisms enable its scalability?\n\nBy simplifying the RL task and directly aligning the RL objective with downstream student learning, we show how a 7B RLT provides better distillation and cold-starting data than the 671B DeepSeek R1 model and other closed-source LMs. These results provide concrete evidence that the RLT framework has the potential to become a more scalable framework by greatly reducing the cost of training advanced models. Instead of relying on large models at every stage, our framework allows training small, specialized teachers that can be used to teach much larger models efficiently. This flips the traditional scaling paradigm: the heaviest work (RL training) can now be handled by compact, more affordable models.\n\nTo better put the scalability and efficiency of our framework into perspective, we added a section comparing the GPU cost of training and collecting reasoning traces for distillation with our RLT model compared with an estimated lower-bound cost of RL training and data collection using DeepSeek R1 (see our first response to reviewer UdmA for details):\n\n|Model/GPU-hours|Training(GPU model)|Data generation(GPU model)|\n|-|-|-|\n|DeepSeek R1|>688000(H800)|>1067(H100)|\n|7B RLT teacher|280.4(H100)|6.7(H100)|\n\nWe hope that adding this new direct comparison and discussion to our paper will help better contextualize the scalability and potential of our new RLT framework, allowing for cheaply distilling large reasoning models while performing expensive RL training with much smaller and efficient ones to counteract the exponentially growing training costs from the scaling laws of LLMs [6].\n\n**references**\n\n[1] Li, Dacheng, et al. \"LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!.\"\n\n[2] Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distillation.\n\n[3] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\"\n\n[4] Hou, Zhenyu, et al. \"Advancing language model reasoning through reinforcement learning and inference scaling.\"\n\n[5] Muennighoff, Niklas, et al. \"s1: Simple test-time scaling.\"\n\n[6] Hoffmann, Jordan, et al. \"Training compute-optimal large language models.\""}, {"review_id": "GJ0a7EXMjW", "reviewer": "Reviewer_UdmA", "summary": "This paper introduces Reinforcement-Learned Teachers (RLTs), a new framework for training language models to be effective teachers for knowledge distillation. Instead of conventional approach of training models to solve problems from scratch, the RLT framework tasks the teacher model to generate detailed explanations for problems given the solution. The authors demonstrate that a 7B RLT can produce raw reasoning traces that are more effective for distilling student models than the traces from much larger LMs (e.g., DeepSeek-R1, Gemini). The paper shows the effectiveness of RLTs for distilling students of various sizes, for cold-starting traditional RL, and for zero-shot transfer to new reasoning tasks.", "questions": "1. The analysis of cost: Could you provide a more direct comparison of the total computational cost (e.g., in GPU-hours) required to (a) train a 7B RLT and distill to a student, versus (b) generating a 17K dataset with a large model like DeepSeek-R1 and then distilling?\n2. Change one student to diverse students: Recent work [1] provides empirical evidence that teaching a diverse set of students is more effective for improving the teacher. Have you ever considered changing the single student to diverse students(e.g. different model sizes or from different model families)? This can be implemented by either teaching a diverse group of students concurrently or by teaching different students at different training stages. \n3. The scalability of the RLT framework: Have you ever tried the training of larger RLTs, for example, a 14B RLT? Does the effectiveness of the RLT framework continue to scale with the teacher's model size?\n4. Why is $r^{KL}$ computed under the RLT format of the teacher? Since KL-Divergence naturally exists for RLT format and regular QA format.\n\n[1] Can LLMs Learn by Teaching for Better Reasoning? A Preliminary Study", "limitations": "Yes, the authors have addressed the limitations of their work. In Section 6 (\"Discussion and extensions\"), they acknowledge that their research is an initial exploration of the RLT framework and they propose several avenues for future work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. **Innovative and intuitive framework.** The paper introduces an innovative and intuitive framework that reframes the conventional approach to training reasoning models. The core idea is training a RLT to create instructive explanations given question and its solution, instead of solving a question from scratch. This cleverly avoids the exploration challenge in traditional RL approaches. Besides, the intuition that explaining knowledge is more important than solving questions for real-world teachers provides a strong conceptual grounding for this work.\n2. **Sufficient and compelling empirical results.** The paper shows strong results in many experiments. These experiments support the paper's claims. It demonstrates that raw outputs from a 7B RLT are more effective for distillation than processed traces from larger models, providing a superior cold-start for subsequent RL training and enabling zero-shot generalization to new domains. The authors also validate their method by showing a strong correlation between their reward scores and the performance of students trained on the corresponding data.\n\n**Weakness**\n\n1. **Lack of cost analysis.** The paper highlights efficiency gains by avoiding distillation from large models like deepseek-r1. However, the RLT training loop itself seems computationally intensive, requiring initial SFT, online generation from the teacher and forward passes through the student to calculate rewards for every sample and finally the distallitation process. A clear cost analysis is missing.\n2. **Lack of analysis of the scaling behavior of the RLT.** The paper shows that a small (7B) RLT can effectively teach a larger (32B) student. However, it does not explore the scaling behavior of the teacher model.\n3. **Minor:** \n   1. Writing style. Some sentences are too long to easily comprehend. The author is recommended to break these down for better and clearer understanding.\n   2. Figure clarity. In Figure 1, unclear whether it is the student size or teacher size", "was_revised": true, "final_justification": "RLTs offer a well-motivated way to distill reasoning traces by letting the teacher model *reason with answer*. The rebuttal resolves my main concerns on cost analysis, writing clarity and figure notions. Although the scaling-up experiments are not done, a scaling-down experiments provide acceptable orcle of the scaling behavior. Thus, I maintain positive and give boarderline accept for the paper. The addtional scaling-up experiment is still advised for the final version.", "rebuttal": "We would like to thank reviewer UdmA for their feedback and the time they dedicated to our review. We worked to address each of their comments and hope they will not hesitate to let us know of any further suggestions or questions.\n\n**responses**\n\n> 1) A clear cost analysis is missing [W1]. Could you provide a more direct comparison of the total computational cost [...]? [Q1]\n\nFollowing the reviewer’s suggestion, we added a subsection to Appendix A, reporting the GPU cost of training and collecting distillation data with our RLT (recorded using a single H100 node with 8 GPUs, see Appendix A.1 for full hardware specs). We compared these times with an estimated minimum cost of traditional RL post-training and data collection using the DeepSeek R1 model. To compute this estimate, we used official figures for the GPUs used to train this DeepSeek model (2,048 NVIDIA H800 GPUs [1, 2]) and online communication from the DeepSeek team. Moreover, we also calculated a lower bound of the GPU hours for generating the distillation dataset by multiplying the minimum amount of GPU resources required to host DeepSeek R1 with half precision and the “roof inference speed” for generating each completion in our dataset, without taking into account slowdowns from the growing context size.\n\nHowever, as we do not have the resources to train/host DeepSeek R1 ourselves, we note that both these estimates should be considered large underestimations (we did not even consider time/resources for multiple generations, filtering, and post-processing stages conducted in our baselines [3, 4]). We provide a summary of this cost comparison in the Table below:\n\n|Model/GPU-hours|Training(GPU model)|Data generation(GPU model)|\n|-|-|-|\n|DeepSeek R1|>688000(H800)|>1067(H100)|\n|7B RLT teacher|280.4(H100)|6.7(H100)|\n\nWe would also like to note that the training time of RLTs is largely dominated by the autoregressive generation loop during the RL phase, with the initial SFT and the extra forward passes through our student network adding a negligible amount of compute cost. This is also precisely the case in the traditional RL reasoning framework, as autoregressive generation comprises a large number of forward passes up to the maximum output context length specified for each single batch sample (16384 in our case).\n\nWe hope that adding this new direct comparison and analysis to our paper will help better contextualize the trivial overheads and great potential of our new RLT framework to cut future requirements for training and distilling RL reasoning models.\n\n> 2) Recent work [5] provides empirical evidence that teaching a diverse set of students is more effective [...] This can be implemented by either teaching a diverse group of students concurrently or by teaching different students at different training stages. [Q2]:\n\nFollowing the reviewer’s suggestions, we added a direct reference to [5] and collected two new sets of experiments in Appendix D to increase student diversity for computing the RLT rewards:\n1. First, we considered using both the 7B and the 32B students, averaging their probabilities, thus introducing student diversity into the RLT reward.\n2. Second, we considered conducting training in two stages, each optimizing both teacher and student for half the total number of optimization steps. In particular, we paused the teacher RL optimization midway, distilled a student with this preliminary RLT’s explanations, and resumed our teacher optimization for the remaining steps with this updated student to compute the RLT reward.\n\nWe note that this second extension has the added benefit of better aligning the teacher rewards to the student’s learning dynamics, and can be considered an initial step toward online optimization of both models (a future direction outlined in Section 6, lines 361-363).\n\n|Evaluated model|Teacher|AIME 2024|MATH 500|GPQA|Overall|\n|-|-|-|-|-|-|\n|Qwen2.5-7B-Instruct|N/A|10.0|74.2|33.3|39.2|\n|Bespoke-7B|DeepSeek R1|20.0|82.0|37.8|46.6|\n|RLT-7B|RLT teacher|23.3|82.8|42.4|49.5|\n|RLT-7B(7B + 32B $r^{RLT}$)|RLT teacher(7B + 32B $r^{RLT}$)|23.3|83.8|41.9|49.7|\n|RLT-7B(2-stage teacher)|RLT teacher(2-stage training)|26.7|84.0|41.4|50.7|\n\nAs summarized in the Table above, both these extensions provide some moderate improvements to the RLT distillation performance. However, we also note that these came at a substantial additional training cost, taking approximately 1.4x and 1.8x more GPU hours than our original pipeline.\n\n> 3) The paper [...] does not explore the scaling behavior of the teacher model. [W2] Have you ever tried the training of larger RLTs, for example, a 14B RLT? [Q3]\n\nAs detailed in our previous response, we ran all our experiments for a 7B RLT using a single H100 node. However, as the cost of training and deploying LLMs does not scale linearly with the number of parameters, our same experiments with a 14B RLT would have taken several weeks and required expensive multi-node setups, beyond the project's allocated resources. \n\nThus, to still try to address the reviewer’s feedback, we instead focused on decreasing the size of the RLT teacher down to 3B parameters. We ran these experiments and added these new results to Appendix D:\n\n|Evaluated model|Teacher|AIME 2024|MATH 500|GPQA|Overall|\n|-|-|-|-|-|-|\n|Qwen2.5-7B-Instruct|N/A|10.0|74.2|33.3|39.2|\n|RLT-7B|RLT teacher(7B)|23.3|82.8|42.4|49.5|\n|RLT-7B(3B teacher)|RLT teacher(3B)|20.0|80.6|38.9|46.5|\n|Qwen2.5-32B-Instruct|N/A|26.7|84.0|49.0|53.2|\n|RLT-32B|RLT teacher(7B)|66.7|93.4|59.6|73.2|\n|RLT-32B(3B teacher)|RLT teacher(3B)|46.7|91.4|53.5|63.9|\n\nAs summarized in the Table above, we indeed confirm that larger teachers yield better explanations and downstream results. However, most of the performance gap between our 3B and 7B teachers appears to occur only when distilling the 32B student LM, where the mismatch between teacher and student capabilities tends to the extreme. However, even in this case where the student is over 10x larger than the teacher, our 3B RLT is still able to provide considerable improvements to the initial Qwen 32B's performance. \n\nWe believe these results further validate how the RLT framework enables even small, inexpensive models to be effective teachers, but also confirm that larger teacher models are indeed able to provide better explanations, highlighting the potential future implications of scaling with more training compute. \n\n> 4) Why is $r^{KL}$ computed under the RLT format of the teacher? [Q4]\n\nThe $r^{KL}$ reward term is computed with the KL divergence over the distribution of think tokens outputted by our RLT during training (thus, under the RLT’s format with both the question and solution in context) and the student’s distribution over the same think tokens (under the distillation format with only the question in context). The purpose of $r^{KL}$ is to ensure that the think tokens in the explanations outputted by the teacher are interpretable logical continuations from the student’s perspective.\n\nInstead, if $r^{KL}$ was computed using the same student format also for our teacher (thus, seeing only the question in context), there would be no regularization preventing our teacher from outputting explanations simply copying and repeating the solution tokens to increase likelihood with the RLT format. As shown by our experiments in Appendix D.1, this is precisely what happens empirically when only optimizing the $r^{SS}$ term: resulting in a teacher that only learns to repeat the solution tokens themselves rather than providing general examples of reasoning steps that can be applied to new problems.\n\nFollowing the reviewer’s comments, we tried to make our motivation clearer by extending Section 3.3 as summarized above, explicitly explaining what would occur if we did not use the RLT format for $r^{KL}$:\n\n“[...] If we instead compared two distributions conditioned on the question alone, the KL would vanish and would fail to penalize the RLT steps that the teacher could not have generated without the solution in context.”\n\n> 5) Minor: Some sentences are too long to easily comprehend [W3.1]\n\nFollowing the reviewer’s feedback, we scanned through the text and broke down overly long sentences to improve readability. Two examples include:\n\nLines 57-59 in our introduction, which we modified to:\n\n“We train RLTs with dense rewards using the student’s log **probabilities. Intuitively, our rewards measure both the student’s**” understanding of each problem’s ground-truth solution from the teacher’s explanations, and the interpretability of the logical leaps in the explanations themselves.”\n\nLines 254-256 in Section 4, which we modified to:\n\n“We also find the effectiveness of the RLT traces stays consistent across different data **sizes. In contrast, the R1 traces from the Bespoke pipeline appear** significantly less effective when subsampled.”\n\nIf the reviewer has any other specific example where text clarity could be improved that we might have missed, we hope they will not hesitate to let us know.\n\n> 6) Minor: In Figure 1, unclear whether it is the student size or teacher size [W3.2]\n\nFollowing the reviewer’s feedback, we added a second sentence to the caption in Figure 1 to explicitly refer to the teacher/student sizes in each subplot:\n\n“A 7B RLT provides better distillation and RL cold-starts [...] **We show this holds both when distilling 7B students of the same size (Left) and also 32B students, much larger than the RLT itself (Right).**”\n\n**references**\n\n[1] Liu, Aixin, et al. \"Deepseek-v3 technical report.\"\n\n[2] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\"\n\n[3] Li, Dacheng, et al. \"LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!.\"\n\n[4] Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distillation.\n\n[5] Ning, Xuefei, et al. \"Can LLMs learn by teaching for better reasoning? A preliminary study.\""}, {"review_id": "ncbpLmFMU5", "reviewer": "Reviewer_YMHP", "summary": "The goal of this paper is to create better teachers for distillation, specifically distillation of long-chain-of-thought \"reasoning\" models. The core concept is to only have the teacher rationalize existing answers to given questions, rather than come up with new answers to existing questions from scratch or come up with wholly new questions. To train LLMs to be better teachers for distillation, the authors add two objectives to RL training for the teacher. The first loss rewards the teacher for providing explanations (chains-of-thought) that increase the student's log-probability of the solution tokens. The 2nd loss rewards the teacher for providing explanations (chains-of-thoughts) that also have a high probability under the student policy. A series of experiments shows that that models trained with these losses make better trainers for distillation and better starting points for RL training.", "questions": "Please see the \"Weaknesses\" section for my questions.  I think an ablation study for the impact of the two losses would be the most useful. I think the other experiments are likely too compute intensive to be done during the rebuttal period, thought I think the scaling experiment is still useful to see, especially with a dataset like NuminaMath-1.5. I'm interested in seeing, for example, how the performance of a student distilled from the teacher changes with / without the RLT teacher over differing amounts of NuminaMath data (perhaps 1k, 10k, 100k, 1M). Finally, see my comments about the related work.", "limitations": "There wasn't much discussion of limitations, but I also don't think anything very specific was needed here.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The central idea of the paper is well-motivated and implemented elegantly with two easy-to-compute losses that both make sense at a glance.\n- The experimental results are strong; it's clear the RLT-trained teacher is a better teacher than the baselines across multiple benchmarks. \n- Given the interest indicated by the community in distilling reasoning models, I think the technique described here + the strong results could be built on by others.\n- I thought the out of distribution analysis was _really interesting_ and backs up one of the core motivations of the paper, that focusing on making better teachers (provided verifiable answers are present) is maybe a better use of compute than doing RL with sparse rewards. \n\n# Weaknesses\n- I didn't see an ablation study for the two losses. Do we really need both of them?\n- I think distillation is something very data dependent and conclusions might change a lot with scale and where questions + answers are being sourced from. I didn't see an analysis of this in the paper.\n- There are no comparisons with models that are designed for distillation, like Nvidia's Nemotron-4. But this might be beyond the compute constraints of the authors. \n- The idea of the paper can be summed up as \"do RL to make better teachers for distillation\". But the Related Works section includes few discussions or mentions of papers that are working on \"better teachers\". Examples are [Montessori-Instruct](https://arxiv.org/abs/2410.14208) and [DataEnvGym](https://arxiv.org/abs/2410.06215), which both have similar motivations but different ways of achieving them. I think this paper is different enough that it doesn't need to justify it's novelty, but it should situate itself relative to prior work better.", "was_revised": true, "final_justification": "This paper makes an important contribution to data-centric research involving distillation. This is an important part of many data pipelines for open LLMs. The proposed approach is a technically ambitious one (of which simpler variants have been previously attempted) to sort of \"meta-learn\" how to improve the data (e.g. by training the teacher). This is pretty hard to do, and the authors propose a method that clearly seems to work, and is relatively elegant. \n\nI had questions about an ablation, but the authors have done the ablation. I also had an issue with a missing discussion of papers that are attempting the same meta-learning approach, but the authors have also addressed this. The remaining issue is that the authors only experiment on smaller data scales, but I think this is unavoidable in the academic setting. \n\nI have read the other reviews. I am not convinced by their weaknesses — some of them either seem irrelevant to how distillation is used in practice (e.g. \"cases where teacher is smaller than student\") or miss the aim of the paper (e.g. \"method only improves teacher instead of student\") or very out of scope (e.g. \"method does not allow solving problems with unknown answers\").\n\nI think there are interesting scientific insights in this paper regarding how reinforcement learning on LLMs works on tasks with a very complex reward like learning to give better explanations, and there are practical applications here. Generally, I think this will be quite useful to the community and many people will be interested in the idea.", "rebuttal": "# Reviewer 4 YMHP\n\nWe would like to thank reviewer YMHP for their feedback and the time they dedicated to our review. We worked to address each of their comments and hope they will not hesitate to let us know of any further suggestions or questions.\n\n**responses**\n\n> 1) I didn't see an ablation study for the two losses. [...] I think an ablation study for the impact of the two losses would be the most useful.\n\nWe would like to clarify that we did perform an in-depth ablation of our reward terms in Section D of our Appendix. \n1. First, we examined ablating the think tokens KL reward term $r^{KL}$, quantifying whether the think tokens $t_{o_i}$ themselves are interpretable logical continuations from the student's perspective as compared with the teacher's.\n2. Second, we examined ablating the min/max reductions terms, which serve to ensure the rewards do not forego any individual token, as detailed in Section 3.3.\n\nWe provided these results in Table 7 and summarize them again in the Table below for easier reference:\n\n|Evaluated model|AIME 2024|MATH 500|GPQA Diamond|Overall|\n|-|-|-|-|-|\n|Qwen2.5-7B-Instruct|10.0|74.2|33.3|39.2|\n|Bespoke-7B(R1 traces)|20.0|82.0|37.8|46.6|\n|Full RLT reward|23.3|82.8|42.4|49.5|\n|No thought tokens KL term $r^{KL}$|6.7|63.8|31.8|34.1|\n|No min/max reward reduction|23.3|79.0|40.0|47.4|\n\nFurthermore, beyond performance, in Figures 8-12, we also compared the lengths of the produced teacher explanations and provided several qualitative, concrete examples showing the effects of each reward term. As detailed in Appendix D.1, ablating $r^{KL}$ leads to a teacher that only learns to repeat the solution tokens themselves in its explanation to exploit the repetition tendency of pretrained student LMs, with the distillation performance and average length of its output reasoning traces dramatically dropping. Furthermore, ablating the min/max reward reduction terms makes the rewards effectively biased by the length of the reasoning trace. While this second ablation only minimally influences performance, it makes the number of think tokens in the teacher's reasoning traces almost double, unnecessarily increasing the downstream student distillation cost.\n\nFollowing the reviewer’s feedback, we extended the end of Section 3.3 to reference more explicitly the presence of this set of important results validating our reward design, and avoid readers potentially missing them. In particular, we modified lines 198-199 to:\n\n“For further discussion, we refer to Appendix D, where we empirically analyze and validate each design choice in our reward function in terms of final performance (Table 7) and concrete qualitative differences of the resulting explanations (Figures 8-11).”\n\nWe note that we did not consider ablating $r^{SS}$ to devolve our compute resources to the other experiments, as without this term our RLT would simply collapse to the student’s initial distribution to minimize $r^{KL}$ with no incentive to make its explanations useful. In fact, this is precisely what we also observed in early runs where the weighting coefficient $\\lambda$ of $r^{KL}$ was too high. However, we hope the reviewer will not hesitate to let us know in case they believe we should still collect these results and add them to our ablation study in future revisions.\n\n> 2) Distillation is something very data dependent and conclusions might change a lot with scale and where questions + answers are being sourced from [...] I'm interested in seeing, for example, how the performance of a student distilled from the teacher changes.with / without the RLT teacher over differing amounts of NuminaMath data.\n\nWe definitely agree with the reviewer that distillation performance can be very dependent on the scale and quality of the initial question dataset. As described in Sections 4.1 and 4.2, this is precisely the reason that we distilled our students using the same starting set of 17K question-solution pairs from the recent state-of-the-art distillation baselines used in our comparison [1, 2]. This allowed us to remove potential confounding factors other than the quality of the reasoning traces between the ones generated from our RLT and much more expensive LLMs such as DeepSeek R1 [3] (we note that we also matched our baselines’ exact distillation hyperparameters and evaluation code rather than making our own).\n\nFollowing the reviewer’s feedback, we collected and added new results in Appendix C, changing the amount of student distillation data, focusing on the NuminaMath dataset. While we did not have time to experiment with hundreds of thousands of samples, we constructed new datasets with 1K, and 10K samples from the AIME, MATH, and Olympiads subsets of NuminaMath as done in [1], enabling us to directly compare with the available reasoning traces from our best baseline using postprocessed reasoning traces from DeepSeek R1 (i.e, Bespoke-7B [2]):\n\n|Evaluated model|Distillation data|AIME 2024|MATH 500|GPQA Diamond|Overall|\n|-|-|-|-|-|-|\n|Qwen2.5-7B-Instruct|N/A|10.0|74.2|33.3|39.2|\n|Bespoke-7B-1K|1K subset from [1]|13.3|80.0|33.8|42.4|\n|Bespoke-7B|17K data from [1]|20.0|82.0|37.8|46.6|\n|Bespoke-7B(1K NuminaMath)|1K subset from NuminaMath|16.7|79.0|33.3|43.0|\n|Bespoke-7B(10K NuminaMath)|10K subset from NuminaMath|16.7|79.6|40.9|45.7|\n|RLT-7B-1K|1K subset from [1]|20.0|80.4|41.9|47.4|\n|RLT-7B|17K data from [1]|23.3|82.8|42.4|49.5|\n|RLT-7B(1K NuminaMath)|1K subset from NuminaMath|16.7|80.4|40.4|45.8|\n|RLT-7B(10K NuminaMath)|10K subset from NuminaMath|20.0|81.0|43.4|48.1|\n\nAs summarized in the Table above, we find that the performance of the RLT and R1 traces from the Bespoke pipeline positively scales with the amount of student training data. While we removed all science questions from STILL-2 present in our original data [4], we find performance on GPQA does not decrease substantially, evidencing positive transfer across different reasoning domains. We note these results are consistent with our previous random subsampling of the questions from [1] (from Table 1 of our paper), with the RLT traces outperforming the postprocessed R1 traces across all tested data sizes and configurations. \n\nWe hope these new results will provide further insights to our future readers regarding the scalability and robustness of our new framework.\n\n> 3) The Related Works section includes few discussions or mentions of papers that are working on \"better teachers\". Examples are Montessori-Instruct and DataEnvGym [...] I think this paper is different enough that it doesn't need to justify it's novelty, but it should situate itself relative to prior work better.\n\nFollowing the reviewer’s feedback, we extended our Related Works section to include prior methods working toward similar goals of designing/training “autonomous data generation agents,”  including the two referenced works [5, 6] and others such as [7]. We made sure to better contextualize the contributions of our paper in relation to these papers in terms of its different methodology and its application to the domain of LM reasoning.\n\n**references**\n\n[1] Li, Dacheng, et al. \"LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!.\"\n\n[2] Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distillation.\n\n[3] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\"\n\n[4] Min, Yingqian, et al. \"Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems.\"\n\n[5] Li, Xiaochuan, Zichun Yu, and Chenyan Xiong. \"Montessori-instruct: Generate influential training data tailored for student learning.\"\n\n[6] Khan, Zaid, et al. \"Dataenvgym: Data generation agents in teacher environments with student feedback.\"\n\n[7] Ning, Xuefei, et al. \"Can LLMs learn by teaching for better reasoning? A preliminary study.\""}]

Author rebuttal:

Meta-review: Summary of the paper: The paper proposes Reinforcement-Learned Teachers (RLTs), a new training paradigm in which a teacher language model is not asked to solve a problem from scratch, but only to generate detailed, pedagogical explanations (CoT) for a given question–answer pair.  Under this framing the teacher’s reward is dense: it measures (i) how much the explanation raises the student’s log-probability of the correct answer and (ii) how probable the explanation itself is under the student’s current policy.  Trained with these two lightweight losses, a 7 B-parameter RLT produces raw reasoning traces that (1) outperform distilled traces from much larger teacher models (DeepSeek-R1, Gemini) when used to train students of any size, (2) serve as a superior cold-start for subsequent sparse-reward RL, and (3) generalise zero-shot to new reasoning domains (Countdown, Game-of-24, etc.).  The authors also show that a small teacher can efficiently supervise a far larger student, and that the method can be bolted onto any existing RL algorithm.


Strength of the paper:
1. Conceptual novelty and clarity: RLTs reframe the teacher’s job from “solve” to “explain given the solution” which is simple, intuitive, and pedagogically grounded. RLTs elegantly sidestep the well-known exploration difficulty of sparse rewards in reasoning tasks.
2. Empirical strength and breadth: RLTs demonstrate that a smaller teacher can teach a larger student, overturning the usual distillation assumption. RLTs also provide multiple use-cases: standard distillation, cold-starting RL, and zero-shot transfer, all supported by strong quantitative results. Out-of-domain transfer (Countdown task) shows generalization beyond the training distribution.
3. Practical impact and reproducibility: RLTs are compatible with any existing RL algorithm, making them an immediate plug-in improvement. Authors release code and detailed experimental protocols, satisfying reproducibility standards.

Weaknesses of the paper: After reading the rebuttal, I think the authors' response addresses most of the reviewers' concerns and questions. No major weaknesses are identified for this paper. Please incorporate the content during rebuttal into the camera-ready version of the paper.

Reasons for the decision:  
1. I think this paper offers a fresh perspective on how to unlock the reasoning power of LLMs. RLT attacks a central pain-point in LRM training—sparse rewards and poor exploration.
2. The proposed losses are simple, theoretically grounded, and empirically validated across diverse settings. Specifically, a 7B RLT beats traces from much larger models, and zero-shot transfer to new tasks is demonstrated.
3. All four reviewers are positive, citing strong novelty, clarity, and experimental evidence, giving a clear consensus to accept.
Decision: Accept (poster)