Title: Reinforcement Learning Teachers of Test Time Scaling
Venue: NeurIPS 2025
Accepted: yes
Keywords: RL

reasoning

LLMs

language

math

reinforcement learning

language models
Abstract: Training reasoning language models (LMs) with reinforcement learning (RL) for one-hot correctness inherently relies on the LM being able to explore and solve its task with some chance at initialization. Furthermore, a key use case of reasoning LMs is to act as teachers for distilling new students and cold-starting future RL iterations rather than being deployed themselves. From these considerations, we introduce a new framework that avoids RL's exploration challenge by training a new class of Reinforcement-Learned Teachers (RLTs) focused on yielding the most effective downstream distillation. RLTs are prompted with both the question and solution to each problem, and tasked to simply "connect-the-dots" with detailed explanations tailored for their students. We train RLTs with dense rewards obtained by feeding each explanation to the student and testing its understanding of the problem's solution. In practice, the raw outputs of a 7B RLT provide higher final performance on competition and graduate-level tasks than existing distillation and cold-starting pipelines that collect and postprocess the reasoning traces of orders of magnitude larger LMs. Furthermore, RLTs maintain their effectiveness when training larger students and when applied zero-shot to out-of-distribution tasks, unlocking new levels of efficiency and re-usability for the RL reasoning framework.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe goal of this paper is to create better teachers for distillation, specifically distillation of long-chain-of-thought \"reasoning\" models. The core concept is to only have the teacher rationalize existing answers to given questions, rather than come up with new answers to existing questions from scratch or come up with wholly new questions. To train LLMs to be better teachers for distillation, the authors add two objectives to RL training for the teacher. The first loss rewards the teacher for providing explanations (chains-of-thought) that increase the student's log-probability of the solution tokens. The 2nd loss rewards the teacher for providing explanations (chains-of-thoughts) that also have a high probability under the student policy. A series of experiments shows that that models trained with these losses make better trainers for distillation and better starting points for RL training.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The central idea of the paper is well-motivated and implemented elegantly with two easy-to-compute losses that both make sense at a glance.\n- The experimental results are strong; it's clear the RLT-trained teacher is a better teacher than the baselines across multiple benchmarks. \n- Given the interest indicated by the community in distilling reasoning models, I think the technique described here + the strong results could be built on by others.\n- I thought the out of distribution analysis was _really interesting_ and backs up one of the core motivations of the paper, that focusing on making better teachers (provided verifiable answers are present) is maybe a better use of compute than doing RL with sparse rewards. \n\n# Weaknesses\n- I didn't see an ablation study for the two losses. Do we really need both of them?\n- I think distillation is something very data dependent and conclusions might change a lot with scale and where questions + answers are being sourced from. I didn't see an analysis of this in the paper.\n- There are no comparisons with models that are designed for distillation, like Nvidia's Nemotron-4. But this might be beyond the compute constraints of the authors. \n- The idea of the paper can be summed up as \"do RL to make better teachers for distillation\". But the Related Works section includes few discussions or mentions of papers that are working on \"better teachers\". Examples are [Montessori-Instruct](https://arxiv.org/abs/2410.14208) and [DataEnvGym](https://arxiv.org/abs/2410.06215), which both have similar motivations but different ways of achieving them. I think this paper is different enough that it doesn't need to justify it's novelty, but it should situate itself relative to prior work better.\n\nQUESTIONS:\nPlease see the \"Weaknesses\" section for my questions.  I think an ablation study for the impact of the two losses would be the most useful. I think the other experiments are likely too compute intensive to be done during the rebuttal period, thought I think the scaling experiment is still useful to see, especially with a dataset like NuminaMath-1.5. I'm interested in seeing, for example, how the performance of a student distilled from the teacher changes with / without the RLT teacher over differing amounts of NuminaMath data (perhaps 1k, 10k, 100k, 1M). Finally, see my comments about the related work.\n\nLIMITATIONS:\nThere wasn't much discussion of limitations, but I also don't think anything very specific was needed here.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Reinforcement-Learned Teachers (RLTs), a new framework for training language models to be effective teachers for knowledge distillation. Instead of conventional approach of training models to solve problems from scratch, the RLT framework tasks the teacher model to generate detailed explanations for problems given the solution. The authors demonstrate that a 7B RLT can produce raw reasoning traces that are more effective for distilling student models than the traces from much larger LMs (e.g., DeepSeek-R1, Gemini). The paper shows the effectiveness of RLTs for distilling students of various sizes, for cold-starting traditional RL, and for zero-shot transfer to new reasoning tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. **Innovative and intuitive framework.** The paper introduces an innovative and intuitive framework that reframes the conventional approach to training reasoning models. The core idea is training a RLT to create instructive explanations given question and its solution, instead of solving a question from scratch. This cleverly avoids the exploration challenge in traditional RL approaches. Besides, the intuition that explaining knowledge is more important than solving questions for real-world teachers provides a strong conceptual grounding for this work.\n2. **Sufficient and compelling empirical results.** The paper shows strong results in many experiments. These experiments support the paper's claims. It demonstrates that raw outputs from a 7B RLT are more effective for distillation than processed traces from larger models, providing a superior cold-start for subsequent RL training and enabling zero-shot generalization to new domains. The authors also validate their method by showing a strong correlation between their reward scores and the performance of students trained on the corresponding data.\n\n**Weakness**\n\n1. **Lack of cost analysis.** The paper highlights efficiency gains by avoiding distillation from large models like deepseek-r1. However, the RLT training loop itself seems computationally intensive, requiring initial SFT, online generation from the teacher and forward passes through the student to calculate rewards for every sample and finally the distallitation process. A clear cost analysis is missing.\n2. **Lack of analysis of the scaling behavior of the RLT.** The paper shows that a small (7B) RLT can effectively teach a larger (32B) student. However, it does not explore the scaling behavior of the teacher model.\n3. **Minor:** \n   1. Writing style. Some sentences are too long to easily comprehend. The author is recommended to break these down for better and clearer understanding.\n   2. Figure clarity. In Figure 1, unclear whether it is the student size or teacher size\n\nQUESTIONS:\n1. The analysis of cost: Could you provide a more direct comparison of the total computational cost (e.g., in GPU-hours) required to (a) train a 7B RLT and distill to a student, versus (b) generating a 17K dataset with a large model like DeepSeek-R1 and then distilling?\n2. Change one student to diverse students: Recent work [1] provides empirical evidence that teaching a diverse set of students is more effective for improving the teacher. Have you ever considered changing the single student to diverse students(e.g. different model sizes or from different model families)? This can be implemented by either teaching a diverse group of students concurrently or by teaching different students at different training stages. \n3. The scalability of the RLT framework: Have you ever tried the training of larger RLTs, for example, a 14B RLT? Does the effectiveness of the RLT framework continue to scale with the teacher's model size?\n4. Why is $r^{KL}$ computed under the RLT format of the teacher? Since KL-Divergence naturally exists for RLT format and regular QA format.\n\n[1] Can LLMs Learn by Teaching for Better Reasoning? A Preliminary Study\n\nLIMITATIONS:\nYes, the authors have addressed the limitations of their work. In Section 6 (\"Discussion and extensions\"), they acknowledge that their research is an initial exploration of the RLT framework and they propose several avenues for future work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the current challenges in reinforcement learning for training LLMs on reasoning tasks, such as exploration difficulties and poor distillation performance, by proposing a novel Reinforcement-Learned Teachers (RLTs) framework. \nUnlike traditional RL models that solve problems from scratch, RLTs are explicitly provided with problems and their corresponding answers during training, generating explanatory processes that are pedagogically valuable for student models. The framework is optimized based on feedback from the student models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The proposed method effectively addresses the critical problem of sparse rewards in reinforcement learning. A significant advantage is its compatibility with any existing RL algorithm, making it a flexible and practical contribution to the field. \n\n\n\nWeakness\n1. The RLT teacher model requires the standard answer to each problem as input in order to generate explanations. This means the method cannot be directly applied to unknown problems or real-world reasoning tasks without provided answers. During the testing phase, the student model still relies on explanations from the teacher to produce answers, lacking independent problem-solving ability. Additionally, the teacher's explanations may already include the answer, effectively revealing the solution to the student model—naturally leading to better performance.\n2. Although the authors designed a reward function based on student model's feedback to optimize the teacher model, the training process remains a one-way offline optimization. The student model remains fixed during the teacher's training phase and cannot dynamically adapt or learn based on the explanations generated by the teacher. This may lead to issues such as the reward function overfitting to the initial parameters of a specific student or the teacher's training objectives failing to align in real time with the student's learning curve.\n3.  No  ablation study was conducted to evaluate the independent contributions and sensitivity of different reward terms(r^SS vs. r^KL)\n\nQUESTIONS:\n1. There appears to be a disconnect between the paper's core contribution and its title. From my perspective, the work primarily addresses the challenges of cold starts and sparse rewards in reinforcement learning. Could you please clarify the connection between these aspects and the concept of \"Test-Time Scaling\" as highlighted in the title?\n\n2. Could you provide a more detailed explanation as to why the proposed method is considered scalable? What specific properties or mechanisms enable its scalability?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce the RLT paradigm where the teacher's task is to simply provide explanations for a student given a question-answer pair, rather than prior approaches where the teacher takes in only the question to give out explanations and the solutions. This approach is intuitive and cleverly circumvents the exploration problem from correctness-based sparse rewards that are typically used. The authors demonstrate multiple use cases and advantages of the RLT paradigm: efficiently training larger students with a smaller teacher model, showing out-of-domain transfer for the teaching task, applying RLT to cold-start future RL iterations. The RLT paradigm is instantiated in the GRPO setting where the sparse based reward is a replaced with a dense-reward modelling two aspects: student understanding of the the solution given the question, and agreement of the teacher logical explanation under the student model and teacher model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The RLT paradigm cleverly avoids the exploration problem by modeling the teaching task with a dense-reward objective, rather than the usual correctness based rewards that are sparse.\n2. Intuitively, the task of providing an explanation given a question-answer pair seems much simpler and makes a lot of sense. This is reflected in a smaller teacher able to effectively train a much larger student. \n3. Out-of-domain zero-shot RLT outperforming direct RL on the countdown task is a strong and interesting result.\n4. The authors also provide the code and many details of their approach in the paper, so the work seems to be reproducible.\n\nWeaknesses:\n1. The authors state that the RLT framework can be used with any RL algorithm, but the experiments performed solely use GRPO. Although I find this believable, the paper could benefit from an experiment to demonstrate and verify this claim.\n2. The authors do not discuss what happens in cases where there is a large mismatch in the abilities of the teacher and student LM. For example, a small teacher LM might output explanations that are not sound (while still maintaining a high logprob under its own distribution). This can be exacerbated in scenarios where the student LM is much larger (or a same-size student that is trained to solve a domain extremely well). Will the RLT paradigm then resort to reward-hacking (i.e., will the student over-trust the teacher's explanations regardless of its own initial abilities)? An ablation on \"small teacher LM\", with increasingly more capable student LMs (beyond 32B) would greatly improve the analysis of the paper.\n3. In (i) of section 3.3, the authors state: _This first reward term is computed with the student’s log probabilities over the solution tokens, reduced with both average and minimum operations_ and then provide equation 3 which seems to imply that the reward is the average log prob + $\\alpha$ min logprob. From the text, I interpreted that the logprobs are normalized (divided by) the avg and min logprobs. This hinders clarity. Similar comments apply to also (ii) in section 3.3, and equation 4. The authors can consider modifying text to improve clarity.\n\nOverall, I remain positive about the paper and believe it provides many new insights.\n\nQUESTIONS:\n1. The authors state that the paradigm of solely correctness-based rewards does not allow the model to explore beyond its initial latent abilities (line 38). With the RLT's dense reward paradigm, can there be exploration beyond the initial LM's latent abilities? To me, this answer still seems to be \"no\" -- but since the teacher does not have to solve the task, there are no sparse-reward problems. If this is indeed the case, the authors must clarify this in the paper.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-04"}