Title: RAST: Reasoning Activation in LLMs via Small-model Transfer
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Reasoning

Reinforcement Learning

Decoding-time Strategy

Scaling
Abstract: Reinforcement learning (RL) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs), as evidenced by recent successes such as OpenAI's o1 and Deepseek-R1. However, applying RL at scale remains intimidatingly resource-intensive, requiring multiple model copies and extensive GPU workloads. On the other hand, while being powerful, recent studies suggest that RL does not fundamentally endow models with new knowledge; rather, it primarily reshapes the model's output distribution to activate reasoning capabilities latent in the base model. Building on this insight, we hypothesize that the changes in output probabilities induced by RL are largely model-size invariant, opening the door to a more efficient paradigm: training a small model with RL and transferring its induced probability shifts to larger base models. To verify our hypothesis, we conduct a token-level analysis of decoding trajectories and find high alignment in RL-induced output distributions across model scales, validating our hypothesis. Motivated by this, we propose RAST, a simple yet effective method that transfers reasoning behaviors by injecting RL-induced probability adjustments from a small RL-trained model into larger models. Experiments across multiple mathematical reasoning benchmarks show that RAST substantially and consistently enhances the reasoning capabilities of base models while requiring significantly lower GPU memory than direct RL training, sometimes even yielding better performance than the RL-trained counterparts. Our findings offer new insights into the nature of RL-driven reasoning and practical strategies for scaling its benefits without incurring its full computational cost. The project page of RAST is available at https://ozyyshr.github.io/RAST/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes RAST, a method to transfer reasoning capability from a small RL-trained model to a larger base LLM without retraining the large model. The authors aim to mitigate the cost of RL training, which only adjusts output distributions on a small subset of reasoning-critical tokens. The authors show that adjustments can be extracted as logit differences between a small model and its RL-enhanced version. At inference time, injecting this difference into the logits of a larger base model can improve reasoning performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. RAST elegantly bypasses costly RL on big models by reusing small-model RL adjustments.\n\n2. The paper verifies an interesting insight: RL only perturbs output probabilities of reasoning-critical tokens.\n\nWeaknesses:\n\n1. The authors mainly focus on the mathematical reasoning problems. However, the **Hypothesis** in the paper does not constrain itself to the math problems. For example, whether the reasoning for general tasks also shares this property is still unknown. Thus, for a logical argument, either experiments in other domains or constraining the argument to math is necessary.\n\n2. Another concern is whether such an argument is influenced by the RL algorithm. For example, whether GRPO/PPO/REINFORCE/RAFT  share the same property is still unknown. The **Hypothesis** should be refined from this perspective.\n\n3. Section 2 is overall too coarse. The presented experimental results cannot support the hypothesis. For example, the different tokens of the base and RL models have not been statistically studied. Figure 10 is hard to understand and interpret, since the normalization procedure is not presented.\n\nQUESTIONS:\nSame as weaknesses. The additional questions are:\n\n1. The method proposed in the paper is interesting. An related quesiton is that how the relative size of LLMs influences the performance. For example, whether the difference in a 100M LLM can be used to improve the performance of a 70B model. If not, what is the failure relative size.\n\n2. For robustness of temperature and $\\lambda$, it is better to change them jointly. Changing them separately may hide some sensitivity.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose RAST, a decoding-time method that performs a reasoning knowledge transfer from a smaller to a larger model. The subject of the transfer is what the authors hypothesize to be small, selective adjustments of LM head probabilities. The authors claim that RAST enables large models to match/exceed performance they would otherwise gain by RL on math and coding tasks.\nThe experiments of the paper are designed to demonstrate that RAST improves decoding diversity and pass@k accuracy, overcoming what the authors claim to be known limitations of RL-based approaches. RAST is positioned as a \"new avenue for decoding-time reasoning enhancement\".\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. Clear formulation of the cornerstone hypothesis of the paper.\n2. Comprehensive, extensive evaluation (Table 2, Table 3, Figure 3), supplemented by clear interpretation of results (Sections 3.2 and 3.3).\n3. Contemporary relevance of the work, especially on the front of resource efficiency (1-2x memory and 2x compute reduction with 60-80% average performance recovery; Section 4.4).\n\n**Weaknesses**\n1. The authors put the review of the related work at the end of the paper but do not use this to their advantage to make concrete comparisons and links to related work. On this front, I find the current state of Section 5 lacking. \n2. I question the relevance of the inclusion of Figure 5 (referenced vaguely from Section 4.2). Perhaps the authors could elaborate on the connection between the Figure 5 and lines 237-245. The current formulation is largely anecdotal and borderline irrelevant, in stark comparison to some other parts of the paper.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nNo on the limitations and no on the potential negative societal impact, though the latter is not particularly important given the nature of the work.\n\nOn the topic of limitations, the paper does include some discussion of the trade-offs in transfer effectiveness, there is no dedicated part of the paper making conceptual outline of the work's/experimentation's limits.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new framework Reasoning Activation in LLMs via Small-model Transfer (RAST), an efficient alignment technique that transfers reasoning ability from a small reasoning model to larger reasoning model. The method is started from clear motivation and observation that the majority of token probabilities remain even after the base model has been trained with reinforcement learning. It just selectively adjusts the probabilities of certain tokens which influences the model’s deeper reasoning such as branch out, self-verification, and backtracking. Based on the observations, the authors propose RAST to transfer the reasoning capability from smaller reasoning models to larger base models by adding probability differences between rl trained and base model in small model to larger base model. Experiments conducted across various math and coding domains, using models of different sizes and architecture shows that RAST consistently improves the performance without heavily training the large model. Additionally, the paper shows some interesting analysis about the reasoning behavior of LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n[S1] The paper is well-written and easy to read with clear visualizations.\n\n[S2] The motivation and the result of observations are intuitive and provide a meaningful way to understand the nature of the reasoning of LLMs.\n\n[S3] The method is well-motivated and demonstrates great performance.\n\n[S4] RAST has the potential to significantly enhance the model efficiency by reducing the training cost.\n\n[W1] Currently, RAST can only be used between models that have the same tokenizer which limits flexibility of the method.\n\n[W2] The method requires 3 models simultaneously during inference time, which necessitates heavy gpu memory and may increase latency. \n\n[W3] Some typos. L150: meaningless hyperlink on text,  L224: (MBase, MRL), L743: (MAHT500)\n\nQUESTIONS:\n[Q1] Can the method be combined with speculative decoding to accelerate the inference speed? Discussing this could make the method more efficient in terms of both latency and training costs.\n\n[Q2] Is there a way to apply RAST between models that use different tokenizers? For example, from Deepseek-R1 to Llama.\n\n[Q3] Please discuss about the concept of RAST-aware training, which could even further enhance model performance training small models\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed a method called RAST,  trying to enhance the reasoning capabilities of large language models (LLMs) by transferring reasoning behaviors from smaller models trained with reinforcement learning (RL). The authors assume that the changes in output probabilities induced by RL are largely model-size invariant, allowing for efficient reasoning activation in larger models without the need for full-scale RL training, however, this assumption does not have any theoretic support.  RAST involves injecting logit adjustments from a small RL-trained model into larger models, the experimental results show performance improvements on various mathematical reasoning benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n1.The core idea of transferring knowledge from smaller models to larger ones is not entirely new. Similar concepts have been explored in other domains, such as knowledge distillation and model compression. The paper does not provide a clear distinction or significant advancement over these existing methods. \n2.The experiments are limited to mathematical reasoning tasks and specific datasets. While the results are promising, the paper does not demonstrate the generalizability of RAST to other types of reasoning tasks or domains. The effectiveness of RAST on more complex or diverse reasoning scenarios remains unexplored.\n3.The paper lacks a formal theoretical analysis of why the logit adjustments from smaller RL-trained models can effectively activate reasoning behaviors in larger models.\n4.The performance of RAST is highly dependent on the quality of the small RL-trained models. If the small models are not well-trained or fail to capture robust reasoning behaviors, the transferred adjustments may not provide significant benefits. This dependency limits the applicability and reliability of RAST.\n5.While RAST avoids full RL fine-tuning on large models, it still requires access to a reasonably strong small expert model trained with RL, which can be computationally expensive to obtain. The paper does not provide a detailed analysis of the trade-offs between the computational cost of training the small expert models and the benefits gained from transferring their reasoning behaviors.\n\nQUESTIONS:\nI have listed all the weak points of the paper, which are the questions authors need to address.\n\nLIMITATIONS:\nSee the weak points listed above.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose RAST, which replaces RL post-training on large LLMs by RL post-training on smaller LLMs and a simple inference-time steering adjustment. The authors show empirically that RAST often allows large LLMs to perform on reasoning tasks as if they were RL post-trained (or even better), while only incurring the costs of RL post-training on smaller LLMs and a negligible steering adjustment. The authors also show that RAST increases trajectory diversity in the large LLM.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n* The method is clear, simple, and novel. \n* The authors' core insight - reasoning behaviors learnt during RL can be used to elicit better performance from a larger model - is plausible.\n* The authors demonstrate empirically that using the single additive logit difference term allows larger models to perform much better on reasoning benchmarks, without additional fine-tuning or weight edits. This works and scales consistently across model sizes and families.\n\n# Weaknesses\n\n* One of the core issues is that the paper never swaps out or ablates the specific behaviors it claims to transfer (e.g. branching, backtracking, self-verification). Experiments that study whether a small model trained to produce the right formatting of answers, or an s1 (https://arxiv.org/abs/2501.19393)-like approach where the small model can force the larger one to answer, or just testing alternative behaviors would be useful. Without such ablations, it is difficult to say whether RAST is truly improving reasoning skills.\n\n* The SimpleRL-Zoo method is used for the small RL teacher and uses MATH500 in its reward data. This also forms the test set, which makes empirical results on the same invalid. It is possible that the improvements could stem from just re-activating memorized solutions rather than transferring actual reasoning skills.\n\n* The experimental setup has additional issues: All comparisons use the base models rather than instruction tuned versions, and show improvements on these base models. Instruction tuning already elicits reasoning capabilities and performance similar to what RAST seems to provide. It would be important to either compare directly against Instruct tuned models or to use RAST on an Instruct model and see if it still improves performance (along with the ablations mentioned above).\n\n* The experiment measuring path-coverage does so on only 50 teacher forced traces from MATH500 on models of the same family. Qwen’s smaller versions contain distillation data from larger versions and have essentially the same training data, so it is unsurprising that these models have similar distributions on benchmark questions. Broad generalisation about “reasoning alignment” seems overstated.\n\n(The reviewer is considering increasing their score if the weaknesses are adequately addressed in the rebuttal)\n\nQUESTIONS:\n* It would be useful to have an ablation where the small model is RL trained for the right answer format, and see if RAST still works. Moreover, experiments around the Instruct models mentioned above would be important to understand the method’s validity. Would you be able to kindly provide this?\n\n* It would also be important to have ablations around individual reasoning behaviors to determine which ones are actually important.\n\n* Could the authors explain how RAST outperforms RL on the larger model in Figure 3?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-19"}