RAST: Reasoning Activation in LLMs via Small-model Transfer
Authors: Siru Ouyang
Xinyu Zhu
Zilin Xiao
Minhao Jiang
Yu Meng
Jiawei Han
Venue: NeurIPS 2025 poster ()
Keywords: LLM Reasoning
Reinforcement Learning
Decoding-time Strategy
Scaling
Primary area: applications

Abstract: Reinforcement learning (RL) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs), as evidenced by recent successes such as OpenAI's o1 and Deepseek-R1. However, applying RL at scale remains intimidatingly resource-intensive, requiring multiple model copies and extensive GPU workloads. On the other hand, while being powerful, recent studies suggest that RL does not fundamentally endow models with new knowledge; rather, it primarily reshapes the model's output distribution to activate reasoning capabilities latent in the base model. Building on this insight, we hypothesize that the changes in output probabilities induced by RL are largely model-size invariant, opening the door to a more efficient paradigm: training a small model with RL and transferring its induced probability shifts to larger base models. To verify our hypothesis, we conduct a token-level analysis of decoding trajectories and find high alignment in RL-induced output distributions across model scales, validating our hypothesis. Motivated by this, we propose RAST, a simple yet effective method that transfers reasoning behaviors by injecting RL-induced probability adjustments from a small RL-trained model into larger models. Experiments across multiple mathematical reasoning benchmarks show that RAST substantially and consistently enhances the reasoning capabilities of base models while requiring significantly lower GPU memory than direct RL training, sometimes even yielding better performance than the RL-trained counterparts. Our findings offer new insights into the nature of RL-driven reasoning and practical strategies for scaling its benefits without incurring its full computational cost. The project page of RAST is available at https://ozyyshr.github.io/RAST/.

RAST: Reasoning Activation in LLMs via Small-model Transfer
Siru Ouyang', Xinyu Zhu?, Zilin Xiao', Minhao Jiang', Yu Meng", Jiawei Han' University of Illinois Trbana-Champaln University of Virginia Rice University, 4 GE HealthCare
siruo2@illinois.o
Abstract
Reinforcement learning (RL) has become powerful approach for improving the reasoning capabilities of large language models (LLMs), evidenced by recent successes such as OpenAl's ol and Deepseek-RI. However, applying RL at scale remains intimidatingly resource- intenssvee requiring multiple model copies and extensive GPU workloads. On the other hand, while being powerful, recent studies suggest that RL does not fundamentally endow models with new knowledge; rather, it primarily reshapes the model's 's output distribution to activate reasoning capabilities latent in the base model. Building this insight, hypothesize that the changes in output probabilities induced by RL are largely model-size invariant, opening the door and transferring its induced probability shifts to larger base models. To verify our hypothesis, we conduct token-level analysis of decoding trajectories and find high alignment in RL-induced output distributions across model scales, validating our hypothesis. Motivated by this, we propose RAST, simple yet effective method that transfers reasoning behaviors by injecting RL-induced probability adjustments from small RL-trained model into larger models. Experiments across multiple mathematical reasoning benchmarks show that RAST substantially and consistently enhances the reasoning capabilities of base models while requiring significantly lower GPU memory than direct RL training, sometimes even yielding better performance than the RL-trained counterparts. Our findings offer new insights into the nature of RL-driven reasoning and practical strategies for scaling benefits without incurring its full computational cost. The project page RAST is available https: / /zyyshr......//. github. io/RAST/.
more efficient paradigm: training a small model with RL
Introduction
Reinforcement learning (RL) [25, 62] has emerged as powerful and prevalent paradigm for en- hancing the reasoning capabilities of large language models (LLMs) [16, 51, 70, 56). Notably, recent successes such as OpenAl's ol model [23] and Deepseek-RI [14] have demonstrated sub- stantial improvements through learning from oracle-verified feedback, employing advanced RL algorithms including Proximal Policy Optimization (PPO) [55] and Group Relative Policy Opti- mization (GRPO) [57]. However, RL is notoriously inefficient and resource-intensiee [58] [68]-in requires loading multiple copies of the same-sized models (e.g., policy, critic, reference, reward) with extensive training GPU memory workloads. Additionally, traditional RL algorithms PPO) typically require multiple iterations, each involving interdependent stages such as rollout, replay, and optimization [32].
PPO)
On the other hand, recently, there have been a bunch of works studying behaviors of RL-trained models, aiming answer the question, "how does RL elicit reasoning capabilities in L.MM??" [44, 57,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Math Problem: For S x S S y 50, find the minimum value the follawing
Model Scale
PCR
PCR 96.71% expression: vi++ 400 + +/5*+ + U y"
100g - -4100.
7B
96.03
Decoding path from A RL
this
( branch out
148
95.60
be the distance to the point (0,20), not (0,0). If
Instoad
of we consider the expression
95.22
500
(ii) self-verification
true
fiiil backtracking
(a) PCR for different model scales
Let's
check
the expression: x² +y² 80r (t-
20j
This is
incorrect
5o we need
(b) A case study
decoding
Jai decoding:
Figure 1: (a) PCR (path coverage rate) across different model scales. (b) A case study revealing the decoding path of Munse and its RI-ttrinned version MRL.. Only very small subset of tokens differ on the decoding path between Mass and MRL. which indicates particular reasoning behaviors.
4]. It is increasingly believed that RL does not endow LLMs with fundamentally new knowledge [74, 12J. Instead, it serves to elicit and amplify reasoning behaviors already present within base models [78, 41, 8]. For example, branching out for alternative solution paths, backtracking from incorrect steps [11, 52], and self-verification for generation [63]. The aforementioned studies lead to major hypothesis [34] in our paper as follows:
Hypothesis: RL activates latent reasoning capabilities in LLMs not by globally altering the entire output distribution, but by selectively adjusting the probabilities of a small subset of tokens that correspond to key reasoning behaviors. The majority of token probabilities - which encode core knowledge and reasoning content remain largely unchanged.
Specifically, if RL primarily teaches models reasoning behaviors (how to reason) by modulating output probabilities rather than imparting fundamentally new knowledge or concepts (what to reason), then the adjustments learned through RL should inherently reflect reasoning skills already latent within base models. This reasoning-centrie view suggests that these learned adjustments may not strongly depend on specific model scales or capacities. Consequently, this hypothesis presents significant opportunity: applying RL. to smaller, computationally efficient models and subsequently transferring the learned probabilistic adjustments to larger, more capable models. Such an approach could substantially mitigate the prohibitive computational and financial costs associated with directly performing RL on large-scale models.
We test our hypothesis via a preliminary study that compares the token-level decoding path shifts between base Mbsse and RL-trained models MRL of varying sizes. As illustrated in Figure 1(b). MRL introduces only minimal shifts in the decoding path, with around 96.71% of tokens remaining unchanged in this specific case. Notably, these shifts are highly localized to a few reasoning castninn-crrtical critical tokens, such as those triggering self-verification, branching out, or backtracking. This suggests that RL acts by amplifying latent reasoning behaviors rather than rewriting entire outputs.
Inspired by the above findings, we propose a simple and intuitive method, RAST, which leverages shifts in the output space to transfer learned "reasoning patterns" from a small RL-trained model (relative to its base) to larger base models across different scales. Extensive experimental results show that RAST substantially and consistently enhances the reasoning capabilities of base models, while requiring significantly lower GPU memory than direct RL training. Surprisingly, sometimes RAST yield even better performance than the RL-trained counterparts. Additionally, RAST also increases search space diversity compared to conventional RL training, exemplified by the superior pass performance. We further conduct detailed analyses on why RAST works and provide insights and practical guidelines for applying RAST, hoping to shed light on future works in this research line.
2 Methodology
Our goal is to enable a large base model Mussc to emulate the reasoning behavior of a smaller reasoner SRL. tuned with RL, without requiring expensive RL training at scale. To this end, we begin with preliminary study that motivates our core hypothesis and then introduce reasoning activation via amall-model transfer, termed as RAST. a simple yet effective decoding-ting time method that activates reasoning capabilities across model scales.
Unless otherwise specified, all mentions of RL- models in this paper refer to "RL from scratch" [76]. directly training models using RL from the base model without SFT warmup.

---

2.1 Preliminary Study
We begin with a preliminary study to empirically support our hypothesis from Sec. 1 RL activates reasoning capabilities in LLMs by adjusting the probabilities of a small set of key tokens that relate to particular reasoning behaviors. Concretely, we take the decoding path T [1, t2. ..., tn] from MRL. and feed it into Mhsse token by token to see if the next token prediction 11 aligns with 1,+1 (i.e., Musse generates the 1 token based on the first n tokens generated by MRL. regardless of the previous Mise generated tokens):
f arg max PM (t f1,..... 14) 1 1,41
(I)
This setup enables us to quantify how likely Miuse is to recover the RL path. This study is conducted under greedy decoding avoid potential randomness. To capture this alignment quantitatively, we define Path Coverage Rate (PCR) as the proportion of tokens in T for which the base model exactly matches the RL output:
(TR(T) - - II
Σ
1(4111441 4+1]
(2)
high PCR indicates that the Musse is already well-aligned with the RI. decoding path, with only minor adjustments needed to activate desired reasoning behaviors. In our implementation, Qwen-2.5-
92B-SimpleRL-Zoo serves as MRR and Qwen2.5-32DD is used as Maase. We randomly sampled 50 trajectories from MRI. from MATH500 [18] and took the sample-level average as the final results. As shown in Figure (a)) we observe that PCR remains remarkably high (> 95%) across all model scales, indicating that RL-induced distributional shifts are notable only on a small set of tokens, with the majority of tokens also predictable by M Additionally, we found that the disparities mainly come from tokens that reflect certain reasoning behaviors. This indicates that by steering around these key tokens, Muss is able to recover the reasoning path generated by the RI-trained model.
2.2 RAST: Reasoning Activation in LLMs via Small-model Transfer
Building on the findings from our preliminary study, hypothesize that minor, targeted adjustments to the Musse 's output distribution can effectively enable it to perform on par with its RL-trained counterpart, without the need for expensive RL optimization directly conducted upon the large base model. In other words, we aim to transform Mhuse into a stronger reasoner at inference time by activating the latent reasoning capabilities already present within it in the token/output space.
To this end, we propose RAST, a decoding-tinn time method that activates reasoning capabilities in large models by transferring logit-level adjustments from smaller RL-tuned models. Given the smaller model pair Stase and SgL. we propose leveraging their differences in logit distributions as reusable reasoning correction signals. Specifically, at the decoding time stamp f, with previous input as r<t. we compute the logit scores of M Spase. SKL, and define the final probability distribution over tokens for the enhanced model M as:
(X, x<)))= = softmax [Masse x<1)) + A(SRL x<(e) Soase (X, x<())]
(3)
where the adjustment terms represent the difference in token-level scoring between the RL-trained model and its base counterpart, denoted as AR. À controls the strengths of AR. These logits encode reasoning-oriented shifts that can be transferred to the larger base model, allowing it to mimic improved inference behavior without retraining. Figure 2 outlines the overview of RAST, showing how A R from a small RL-tuned model is injected to adjust the output distribution of Mause decoding time, selectively amplifying reasoning-relevant tokens (e.g., "instead"" while preserving base predictions (e.g., "of"" elsewhere. The additive formulation enables lightweight adaptation at inference time by altering the output distribution in a way that reflects reasoning preferences learned by the smaller model.
Our method is conceptually related to [36, 37, 31], which apply similar manipulation in logits spaces for generation steering. However, while these methods are often task-specific or used for controlling style/toxicity, our focus is on eliciting complex reasoning behavior, and we demonstrate that even small-scale reasoning signals can be reliably transferred across model sizes and domains.
3 Unlocking Reasoning Activation
3

---

Math
For 5 40 and I I 50, minimum value fallowing expression:
Problem:
Vx*+4000 + y 900
x² + y² 100y
Prefix tokens:
distance to the point (0,20), not (0,0).
(Decoding time stamp
Amplifying tokens reflect reasoning
behavior.
instead
this
instead
insteed
this
in
Instead
heur
d
Prefix tokens:
to the point (0,20), not (0,0). If instead
(Decoding time stamp t+1
Maintain base prediction for of
reasoning tokens.
AAL
S
Figure A concrete illustration of RAST: logit differences from a small RL-tuned model SRL guide large base model Mbsse at decoding time, amplifying reasoning-relevant predictions (e.g.. "instead"" while maintaining base outputs for non-reasoning tokens (e.g.. ".f").
3.1 Experimental Setup
Models and Tasks We systematically evaluate model performance across a comprehensive suite of mathematical reasoning tasks of varying difficulty, including standard benchmarks such as MATH500 [18], Minerva [30], OlympiadBench [17]. GSM8K [6], as well as competition-level benchmarks AIME24 and AMC23. Our primary models are from the Qwen-2.5 225 family (1.5B, 7B, 14B, and 32B) [71] alongside their corresponding RL-trained variants using SimpleRL-Zoo [76] (e.g.. SimpleRL-7B denotes the "RL from scratch" variant of Owen-2.5-7B). We further demonstrate the generalizability of RAST across different model architectures and downstream tasks. validate this, we conduct experiments using the Llama-3.1 series (8B, 70B) [13] and their zero RL-trained counterparts. Additionally, we assess model performance on coding tasks, utilizing zero RI.-trained models from Code-R1 [40] trained from Qwen-2.5-1M [72] coding benchmarks including Hu- manEval [2], MBPP+ [1], and LiveCodeBench [24]. For details of datasets and setup used in our experiments, please refer to Appendix A. We also present additional experiments on MMLU [18] and GPQA [54] and analysis results in Appendix C.
Evaluation Metrics Following previous work [19], and to ensure rigorous evaluation across models and tasks, we perform inference runs & up to for each experimental setup and report the following metrics calculated over the collected trajectories:
Pass@ Pass & evaluates whether at least one correct solution is found among sampled outputs per problem. Formally,
Pass @ k
Σ ( 94) 2 21)
where N the total number problems, Ut.j the prediction for the i-th problem in the j-th run, Y: the corresponding ground truth, and I(-) denotes the indicator function."
Recovery Rate: The recovery rate quantifies how much of the gap between the base model and stronger RL-tuned model recovered by the proposed method. Formally,
Recovery Rate AccuracyRAst Accuracy
(5)
AccuracyRI. -Accuracyy Basc
where "Accuracy" denotes the averaged pass @1 over 32 runs. Higher values of recovery rate indicate a more effective recovery of the performance gap.
Decoding Configurations To accelerate the inference speed, implemented revised vLLM [27] version to support RAST. For mathematical reasoning, decoding is performed using a temperature
WW use the same answer extraction and matching for performance evaluation as SimpleRL, following https:/ / /ggtttb.....tggggggg com coom/huutt-nlpp mwpplpppelllemlllleeemppmmmmmm www.alllrrreeooeeeeeeeeeeeeeee hat.s////rreee//e//aameeeee. wtt.eeeeeessseeeeeeeeeeeeeeeeeee lea_eelrr__ate__ll_____

---

Table 1: Experiment results of RAST on mathematical reasoning datasets with Qwen-2.5 model series. All numbers are computed across runs with sampling, except that all base models use greedy decoding. Avg. indicates the averaged pass@ over 32 runs and RR. denotes the recovery rate.
Math500
AIME24
AMC
Minerva
Olympiad
GSM8K
Models
Avg.
RR.
Avg.
RR.
Avg.
RR.
Avg.
RR.
Avg.
RR.
Avg.
RR.
Qwen-2.5-32B
68.6
3.3
52.5
21.0
33.1
93.1
+ R1.sm
73.7
40.2%
12.5
37.9%
54.7
12.9%
25.1
84.1%
36.7
30.5%
93.3
7.7%
+ RyB
79.0
81.9%
16.4
53.9%
58.4
34.5%
32.0
87.3%
41.7
72.9%
94.4
50.0%
+ R142
80.7
95.3%
18.3
61.7%
65.2
74.3%
34.2
104.8%
43.5
88.1%
95.3 84.6%
SimpleRL-32B
81.3
27.6
69.6
33.6
44.9
95.7
Qwen-2.5-14B
63.8
6.7
47.5
19.1
31.1
91.4
70.3
45.1%
10.9
54.5%
50.3
25.0%
22.9
29.0%
35.5
37.0%
92.4 31.3%
+ARzn
77.4
94.4%
16.2
123.4%
56.1
76.8%
28.8
74.0%
40.0
75.4%
93.3 59.4%
SimplcRL-14B
78.2
14.4
58.7
32.2
42.9
94.6
Qwen-2.5-7B
62.9
6.7
32.5
19.9
28.0
87.7
+ R1.52
68.4
41.0%
9.5
31.5%
41.5
38.5%
23.0
51.7%
34.6
57.4%
89.8
50.0%
SimpleRL-7B 76.3
15.6
55.9
25.9
39.5
91.9
indicates that the prompt used for RAST does not match the one used to train the small RL-tuned model (see Figure 7), due to the training inconsistency for available RL-tuned models (e.g.. from SimpleRL-Zooo) As result, the transferred AR R555 yields relatively smaller gains.
Table 2: Experiment results of RAST on mathematical reasoning datasets. AR borrowed from Llama-3.1-8B 1-1------iilleR--22 [76]. Numbers are computed on 32 runs with sampling.
Models
Math500
AIME24
AMC
Minerva
Olympiad
GSM8K
Llama-3.1 7.1-7-
26.2
0.0
12.5
10.3
5.9
56.1
+ARsn
33.1
2.8
16.9
12.6
7.1
82.2
Table 3: Experiment results of RAST on code reasoning tasks with the Qwen-2.5-14B-I1 model as Mbase and AR from Code-R1-Zero [40] using greedy decoding.
Models
HumanEval+
MBPP+
LiveCodeBench
Average
Qwen-2.5-14B-IM
82.3
69.6
34.3
62.1
RA
85.4
76.5
37.6
66.5
setting of 1.0 and nucleus sampling with a top-p of 0.95, allowing a maximum generation length of 16, 384 tokens, consistent with prior work [76]. We set in Equation 3 to 1.0 for all experiments. For code reasoning tasks, we follow previous evaluation settings [40, 24] and use greedy decoding. Specifically, we use EvalPlus [38, 39] for HumanEval+ and MBPP+. Our experiments are conducted over 8 NVIDIA A6000 GPUs on single node, with GPU utilization and tensor parallelism parameters dynamically adjusted based on the model size. Typically, inference requires approximately 30 minutes per run for larger datasets like MATH500 and GSM8K, whereas smaller datasets such as AIME24 and AMC23 complete within 3-5 minutes per run. For specific parameters used for each benchmark, please refer to Appendix B.
3.2 Main Results
RAST enables consistent and scalable reasoning gains. Table summarizes the performance of RAST on six mathematical reasoning benchmarks using the Qwen-2.5 model family at 1.5B, 7B, 14B, and 32B scales. Across all settings, RAST delivers substantial improvements over the base models in both averaged pass @1 and the corresponding recovery rate. Notably, with signals from smaller RL-trained models, RAST can even approach or beat the performance of RL-trained counterparts of the base model. For instance, applying A Rian to the 32B base model achieves approximate or superior results on MATH500, Minerva, and GSM8K compared with the 32B RI-trained model. These findings validate the effectiveness RAST enhancing reasoning capabilities at inference time, without any retraining RL on the target model.
5

---

100
95
55
Pass@k
92
89
0-55
92.8
92.5
91
86.8
88.5-4
90
90.1.4
91.3
89
91.
91.4
45
50
42.4
85
39.7
65.1
073.7
39.5
40.1
68
79
35
60
6.60
75
27.9
30
33.6
4
8
16
32
4
8
16
32
4
8
16
32
(a) AMC 2023
(b) MATH500
(c) Minerva Math
60
56.7
100
75
56.7
98.9
64.2
67.4
Pass@kk
39.7
43.7
O 43,3
97.5
56.3
60.6
O
65.8
30
37.7
97
96.97.1
97.3
97.5
97.9
51
55.6
58.1
25,3
97
96
43
(d) AMC 2024
(e) GSMBK
(f) Olympiad Bench
M RL
-0- Ml buse
Our methad
Figure 3: The illustration of pass k for different values of k on 6 mathematical reasoning datasets, where Mbse is Qwen-2.5-32B, RAST uses AR and MR, is the RL-trained version of M
AR from stronger experts yield greater gains. The effectiveness of RAST also depends the strength of the SRL and Snuse that generates the delta logit A R. For each base model, using larger delta sources (e.g., base model of 32B with A R7B A R14B compared with A AR.5B) leads greater improvement. Taking the 32B base on MATH500 as an example, accuracy increases progressively from 73.7 (with (.)))) 80.7 (with 4R14B), while the ceiling model, the upper bound reaches 81.3. Similarly, AR7B also works better than ARt.sB for the 14B base model across all datasets. This trend suggests that logit deltas encode richer reasoning signals the SRT. model scale increases, making RAST a flexible tool for knowledge transfer across various model scales.
Trade-off between Mbase and AR. The effectiveness of RAST also depends the capacity of the base model Mbass and its aligmment with AR. In general, stronger base models exhibit higher recovery rates, indicating greater receptiveness transferred reasoning signals. For example, when applying RAST to the 32B base, the recovery rate often higher than when using 14B or 7B. On GSM8K, RAST bridges nearly the entire gap between the base model (93.1) and the RL expert (95.7), achieving 95.3 with the delta logit from a 14B model. In contrast, for the 7B base, the gain is smaller (87.7 to 91.9), even though the same ARt is applied. This suggests that higher-capacity base models are more receptive to the transferred reasoning signal. However, increasing base model capacity alone does guarantee better outcomes. When applying ARTB both 14B and 32B bases, the 14B base yields higher recovery rate, suggesting that large capability gap between the base and A may hinder effective transfer. These observations highlight trade-off: while stronger base models benefit more from compatible deltas, excessively mismatched pairs may reduce the efficacy of reasoning activation.
3.3 RAST Boosts Reasoning Diversity
Figure 3 illustrates the @ accuracy trends across six mathematical reasoning benchmarks, using the Qwen-2.5-32B base model augmented with AR1B. For each problem, we randomly sample A outputs from 32-sample pool, repeat this process 10 times, and report the average accuracy. This setup allows us to assess how well RAST supports diverse solution trajectories under varying sampling sizes. Based the results, we have the following key observations:
Increasing ie consistently improves accuracy. Across all benchmarks, @ass@ k increases mono- tonically with larger k. confirming the benefit of sampling multiple decoding paths. This trend reflects that RAST promotes solution diversity-a larger k enables broader exploration plausible answers, increasing the likelihood of capturing correct responses even when individual generations are imperfect. We also noticed that on most benchmarks except GSMSK, pass@ & for MRL. is large than Musse which might result from GSM8K's simplicity.

---

@ surpasses the ceiling performance. Remarkably, in all benchmarks, RAST achieves pass@k l accuracy that equals or even exceeds the performance of MRI.- This stands in contrast to prior findings [57, 74], which reported limited or deteriorated performance in pass k under RI. training. We posit that this effect may stem from the implicit ensembling of knowledge across models, which enhances diversity in the search space. Additionally, RAST can sometimes beat the pass @ of Masse on benchmarks like AMC, MATH500, and Olympiad Bench. This surprising behavior suggests that RAST not only saves the costly training efforts of RL, but also introduces a distinct form of diversity or guidance in the sampling space that helps recover correct answers, a notable drawback of RL-trained models.
3.4 RAST Generalizes Well to Other Models and Tasks
We conduct experiments of RAST on another model family, Llama-3.1, and on additional code reasoning tasks. The results are shown in Tables 2 and 3. Applying RAST to Llama-3.1 3-70B using A Rgg from a smaller RL-tuned model [76] yields consistent gains across all six mathematical reasoning datasets. As shown in Table 2, we observe +228 absolute improvement on AIME24, +4.4 on AMC, and +2.3 on Olympiad. We also found that the improvement in MATH500 and GSM8K was particularly high. This might be due to the training recipe of SRI. where they are trained with problems from MATH500 and GSM8K. Nonetheless, the experiments demonstrate that RAST is effective for different model families apart from Qwen, reaffirming the hypothesis that reasoning-relewe relevant distributional shifts are transferrable and model-agnostic, evaluate RAST reasoning benchmarks. RAST achieves consistent improvements on all datasets, leading to an overall +4.4 absolute improvement in average performance. This indicates that the method is not limited to mathematical reasoning, but code-related reasoning tasks. These findings confirm the broad applicability of RAST across model architectures, parameter scales, and reasoning domains.
Table 3, we 1.cn-22.5-4B-1111 [72] using AR from its 7B counterpart [40] on three code
4 Understanding Reasoning Activation
0.944 AR v.s. AR
4.1 Similarity of AR as Signal for Transferability
As shown in Table model performance varies widely across settings, motivating the need to understand what governs effec- tive transferability. The delta logits AR (defined in Equation 3) capture the modification induced by the reasoning-enhancee model relative to its base. To quantify the alignment between these delta logits across different model pairs, we adopt cosine similarity as the measure. This choice motivated by prior work [61, 15], which demonstrates that cosine similarity ef- fectively captures directional alignment in high-dimensional spaces, independent of magnitude. Following Section 2, we randomly sample 50 examples from MATH500 and extract the decoding trajectory T (with tokens t) generated by SimpleRL- 32B. Each trajectory is then fed through Mhnse and MRI. token by token to compute AR with prefix Tet. We then log the average cosine similarity across the trajectory as:
0.53
AR v.5 A8
AR v.8 A8
Recovery Rate
Cosine Similritt
Figure 4: Cosine similarity vs. re- covery rate across delta logit pairs (AR) from varying model scales. E.g., "ARMA v.s. ARTg" denotes AvgCosineSim( (RLB, R.AB,AARRp AR7B).
1 T
T
(R(!)
AR
AvgCosineSim(ARR, AR2) =
(6)
A
(t)
!
The results are shown in Figure 4. We found that the recovery rate exhibits a positive correlation with cosine similarity-as AR between models becomes more aligned, transferability improves.
4.2 Empirical Token-Level Signals of Reasoning Activation
To further examine how RAST elicits reasoning behaviors during generation, we perform a token-level analysis focusing on linguistic traces that reflect three hallmark reasoning behaviors: (i) branching out, (ii) backtracking, and (iii self-verification as previously mentioned in Section 1. For each behavior,
7

---

we manually curated set of representative tokens based on prior qualitative observations and related work [11, 52, 63]. These tokens serve as behavioral signatures that may surface when the model engages in complex reasoning. The complete set of curated tokens is displayed in Figure 9. We also present a case study of token-level reasoning activation for more intuitive view in Appendix C.5.
We compute the frequency of these tokens in all 32 model output trajectories across six mathematical reasoning benchmarks, comparing the base model Masses RL-trained ceiling model MRL. and RAST with A applying to 32B base models. Figure 5 presents the normalized occurrence rates of each token category across different models. Two key trends emerge:
Dataset-specifie reasoning emphasis. Different benchmarks accentuate different types of reasoning behaviors. For instance, AIME 2024 and Olympiad Bench exhibit pronounced increase in branching out and backtracking tokens, indicating that these tasks may demand exploring alternative solution paths and revisiting previous steps. In contrast, datasets like AMC 2023 and MATH500 emphasize self-verification, with higher frequencies of verification-reltee tokens such as "check" or "confirm". This variation suggests that reasoning demands are not uniform across benchmarks, and token-level analysis can surface behavior-specific task signals.
RAST steers base models closer to RL-trained behaviors. Across all datasets and reasoning cate- gories, we observe that RAST consistently increases the frequency reasoning- related tokens relative to Mbase and more closely matches the RL-trained model MRLL- This alignment is particularly clear in Minerva, Olympiad Bench, and GSM8K, where RAST nearly mirrors the behavior of MRL in the self-verification dimension. These results support our central claim that the delta logit signal AR effectively induces reasoning traits without the need for full RL training on large-scale models.
Together, these findings offer empirical evidence that RAST not only activates latent reasoning capa- bilities within base models but also tailors such activation in a task-sensitive manner, approximating the behavioral signature of much costlier RL-trained experts.
4.3 Efficiency Analysis
GPU Memory Overhead. We first present the efficiency analysis in terms of estimated GPU mem- ory requirements, highlighting the computational advantage of RAST over conventional RL training. We summarize the results in Table 4. We estimate the memory overhead in terms of GPU memory used for RAST (including training SRL and inference cost), and MRL. The results are estimated considering tensor parallel and CPU offloading (since these are common tricks dur- ing training) based on the following dimensions: (i) model memory foot- print (FP16), (ii) optimizer states, and (iii) activations & buffers. Details of the computation for estimation could be found in Ap- pendix D. Despite using significantly fewer resources, RAST achieves high recovery rates across all settings (c.g., reaching over 84% in the most demanding 32B AR1AB configuration). This demon- strates that RAST retains most of the performance benefits of full-scale RL training while reducing the computational burden by up 50% in terms of GPU memory and hardware requirements.
Table 4: Comparison of memory overhead and inference speed between our approach RAST and the conventional RL training pipeline (i.e., GRPO). We also report the averaged performance recovery rate (RR.) for all mathematical reasoning benchmarks.
32B
32B
14B
Settings
Mose
MRL
Mosse
MRL
Muss
MRL
+ARAB
+AR7B
+AR7a
Memory Overhead
GPU memory # GPU cards
~ 160G
350G 1000 3550G 100G ~ 160G
4x 80G
8x 80G 2x 80G 80G 8x00G
80G
Inference Speed
Input Tokens
61.1
73.0
62.4
73.0
65.0
75.1
Output Tokens
200.3
243.1
202.7
243.1
216.1
249.5
Averaged RR.
84.8%
100%
63.4%
100%
83.9%
100%
Inference Speed. Table further offers the actual inference speed in terms of the number of input and output tokens processed per second for RAST'S practicality. We observe that RAST incurs around 13-16% latency at inference time compared to directly using MRR. for decoding at different scales. This is expected and acceptable due to the need to compute multiple model logits at each decoding step. We believe that this latency could be further optimized by efficient decoding methods such as speculative decoding [29, 69, 49].
8

---

1.33
3.20
12
32
14
0.87
2.30
2:33
3.6
0.46
0.46
1.6
1.10
1.41
0.79 0.90
08
0.430.2
0.26
0.25
saff-vorfy
self-vanfy
brandhouu
selfiverfyy
(a) MATH500
(b) AIME 2024
(c AMC 2023
0.94
0.50
2.40
0.04
0.6
1.5
0.18
1.235
0.97
3.
0.6
0.34
9.67
0.05
0.07
0.02
0.030.05
branch-out
backtrack
sol-verily
branch out
backcradk
soli-verify
branch-out
backtrack
seli-veriyy
(d)
Olympiad
GSMBK
Figure 5: Normalized frequencies of maasoning-relatee tokens across three models (Maase RAST, and M over six benchmarks. Each subfigure reflects different reasoning behavior category.
4.4 Robustness regarding and X
To evaluate the sensitivity of our method to decoding-time hyperparametters we perform a grid search over sampling temperature and used in Equation The results are summarized with performance trends visualized in Figure 6. We plot accuracy across varying values (with À = 0.5 fixed) and varying values (with across multiple runs. Note that use these fixed values for investigation since they represent the peak performance. In both settings, the accuracy remains consistently high within reasonable range. Specifically, with E [0.5, 1.0] and À E [0.3, 1.5]. the performance would be reasonably good within a certain range. These trends demonstrate that our method is robust to moderate fluctuations in decoding-time hyperparameters and does not rely precise tuning for strong performance.
1.0 fixed), including standard deviation error bars reflect stability
Related Work
5.1 Reinforcement Learning for LLM Complex Reasoning
Reinforcement learning (RL) has emerged as an important process of LLM post-training for hu- man preference alignment. Early successes such as reinforcement learning with human feedback (RLHF) [48] demonstrated the effectiveness of reward modeling from preference data [75, 7]. Build- ing upon this, follow-up works have explored RL as means to directly optimize for reasoning quality [23] with outcome supervision [28] or process supervision [33, 64] as verifiable rewards, specifically in the domain of math [57] and coding [66, 40].
More recently, the success of DeepSeek-R1 [14] introduced notable shift in training methodology with the "zero-RL." paradigm, where RI. applied directly to the base LLM, entirely bypassing intermediate supervised fine-tuning. Following its release, the open-source community has made significant strides in replicating [76, 50, 81], extending [73], and interpreting [78, 74, 42] the R1 algorithm and its behavioral consequences. Our work builds directly upon these open-source Zero-RL models and takes a step further by exploring whether the reasoning behaviors elicited through RL in small models can transferred larger base models without additional RL training. Specifically, we focus on leveraging the output distribution (logits) of small RL-trained reasoning models to activate similar behaviors larger models across scales.
5.2 Decoding-time Strategy in Language Models
Decoding-tima methods [59] have been explored largely text generation, typically by manipulating the logit distribution from base language models. Earlier research efforts focused on camling-bbsee

---

84
82
0.5
80.8
Accuracy
Accuracy
00.0
74.4
00.2
79.6
68
0.1
0.4
0.5
0.6
0.7
0.0
0.9
1.1
12
1.3
1.4
1.5
1.5
Parametor
Sampling Temperature
Figure 6: Experiment results with varying (left) and (right) on MATH500 dataset, where * denotes the peak performance. The position represents the accuracy, and the size of the circle denotes the standard deviation over 32 runs.
strategies aimed at improving generation quality through techniques like hierarchical decoding [10], nucleus sampling [20], and locally typical sampling [45]. More recent approaches incorporate the notion of contrastiveness. Specifically, Contrastive Decoding [31] exploits the disagreement between stronger (expert) and weaker (amateur) model to down-weight completions favored by the weaker model. DExperts [37] steers generation by contrasting models fine-tuned on desirable and undesirable attributes. Proxy-tuning [36] and emulated tuning [46] further leverage the idea for training-fiee alignment, while DoLA [5] contrasts different layers in pext-token-pe prediction for factuality. The philosophy of contrastiveness has since been extended
eext-toeen-preiiciion variety of downstream tasks, including machine translation [65], retrieval- augmented generation (RAG) [53], and conflict resolution in pewllede-intensivy tasks [60]. While these works primarily focus on stylistic control or factual consistency in open-ended text generation, our approach targets caasoningsspecffi improvements.
When it comes reasoning, [47] explores contrastive decoding for mathematical reasoning. while [35] identifies critical tokens in the reasoning trajectory using contrastive estimation. Our work builds on this growing line of decoding-time reasoning enhancement. Specifically, we leverage the divergence between Zero-RL-trained experts and their base counterparts to guide decoding, aiming surface and amplify reasoning behaviors. contrast to prior approaches that operate on instruction-tune or supervised models, our method explicitly targets RL-induced reasoning signals and explores their transferability across model scales.
Conclusion and Discussion
We introduce RAST, a decoding-time framework that enables scalable reasoning enhancement in LLMs by transferring logit-level guidance from smaller RI-tuned models. Comprehensive experiments show that RAST consistently boosts the performance, often approaching or surpassing the performance of much more expensive ceiling models. Further analysis confirms the robustness of our method across decoding hyperparameters and highlights the alignment between delta signals and reasoning activation. Our findings suggest a practical and efficient pathway for eliciting complex reasoning in LLMs, opening new avenues for decoding time reasoning enhancement and model behavior study for RL. We also present some insights toward future directions in Appendix E.
Acknowledgments and Disclosure of Funding
We sincerely thank the anonymous reviewers and the chair for their constructive feedback and valuable suggestions, which greatly helped improve this work. Research was supported in part by National Science Foundation IIS- 19-56195,,,,, NSF IIS 25-37827. the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 201 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Environment (I-GUIDE) by NSF under BRIES Program No. HR001 1-24-3-0333- 0-35.. The research has used the Delta/DeltaAl advanced computing and data resource, supported in part by the University of Illinois Urbana- Chaanaaaa and through allocation #250851 from the Advanced Cyberinfrastruuture Coordination Ecosystem: Services Support (ACCESS) program, which supported by National Science Foundation grants OAC 2320345, #2138259 #2138286, #2138307, #2137603, and #2 138296.. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of DARPA the U.S. Government.
10

---

References
[1] Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. ArXiv preprint, abs/2 0os/2108.07732, 2021.
[2]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Berbert-Voss,, William Hehgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
[3] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song. Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. ArXiv preprint, 10n//242.21187, 2024.
[4] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Sergey Levine, and Yi Ma. SFT memorizes, RI. generalizes: A comparative study of foundation model post-training. In The Second Conference on Parsimony and Learning (Recent Spotlight Track). 2025.
[5] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality large language models. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/21 10. 14168, 2021.
[7] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2024.
[8] Xingyu Dang, Christina Baek, J Zico Kolter, and Aditi Raghunathan. Assessing diversity collapse in reasoning. In Scaling Self-Improving Foundation Models without Human Supervision, 2025.
[9] Thomas G. Dietterich. Ensemble methods machine learning. In Proceedings of the First International Workshop Multiple Classifier Systems, MCS '00, page 1-15, Berlin, Heidelberg, 2000. Springer-Vettee Verlag.
[10] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Melbourne, Australia, 2018. Association for Computational Linguistics.
[11] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. ArXiv preprint, htbs/2503.01307, 2025.
[12] TNG Technology Consulting GmbH. Deepseek-rlt-chimers 2025.
[13] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783, 2024.

---

[14] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-ri: Incentivizing reasoning capability in Ilms via reinforcement learning. ArXiv preprini, abs/2501. 1tt.//111.1194,,.. 2025.
[15] Gyeongdo Ham. Seonghak Kim, Suin Lee, Jae-Hyeok Lee, and Daeshik Kim. Cosine similarity knowledge distillation for individual class information transfer. ArXiv preprint, abs/2311.14307, 2023.
[16] Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. ArXiv preprint, abs/2400.04642. 2024.
[17] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilin- gual multimodal scientific problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 3828-3850, Bangkok, Thailand, 2024. Association for Computational Linguistics.
[18] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021.
[19] Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. ArXiv preprint, abs/2504.07086. 2025.
[20] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[21] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.ner 2022.
[22] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic loRA composition. In First Conference on Language Modeling, 2024.
[23] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed BI-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. ArXiv preprint, abs/2412.16720, 2024.
[24] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025.
[25] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
[26] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics. 22(1):79-95) 1951.
[27] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph F. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large lan- guage model serving with pagedattention. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
12

---

[28] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T ulu 3: Pushing frontiers in open language model post-training. ArXiv preprint, 0/x/2411.11122 2024.
[29]
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via specu- lative decoding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 19274-19288 PMLR, 2023.
[30] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning prob- lems with language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LAA, USA, November 28- December 9, 2022, 2022.
[31] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceed- ings of the 61 Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Pupers), pages 12286-12312. Toronto, Canada, 2023. Association for Computational Linguisties.
[32] Eric Liang, Zhanghao Wu, Michael Luo, Sven Mika, Joseph E. Gonzalez, and Ion Stoica. Rllib flow: Distributed reinforcement learning is a dataflow problem. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Perey Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 5506-5517, 2021.
[33] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[34] Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khy- athi Raghavi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base Imm: Rethinking alignment via in-context learning, In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-1,, 2024. OpenReview.net, 2024.
[35] Zicheng Lin, Tian Liang, Jiahao Xu, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. Critical tokens matter: Token-level contrastive estimation enhence Ilm's 's reasoning capability. ArXiv preprint, 115/22411.19994 2024.
[36] Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A. Smith. Tuning language models by proxy. In First Conference on Language Modeling, 2024.
[37] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and anti-experts. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6691 6691-6706, Online, 2021. Association for Computational Linguistics.
[38] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information P Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
13

---

[39] Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang Evaluating language models for efficient code generation. In First Conference on Language Modeling, 2024.
[40] Jiawei Liu and Lingming Zhang. Code-rl: Reproducing rl for code with reliable rewards. 2025.
[41] Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang. Chao Du, and Min Lin. There may not be aha moment in rl-zero-like training a pilot study. https: sttps://oatlmm wattatimmm.ooooonnmmmmmmmmmoiiit site/ oat-zero, 2025, Notion Blog.
[42] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. ArXiv preprint, abs/2503.20783, 2025.
[43] Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. Cot-valve: Length-compressible chain-of-though tuning. ArXiv preprint, abs/2502.09601, 2025.
[44] Sara Vera Marjanović, Arkil Patel, Vaibhav Adlakha, Milad Aghajohari. Parishad BehnamGhader, Mehar Bhatia, Aditi Khandelwal, Austin Kraft, Benno Krojer, Xing Han Lü, et al. Deepseek-rl thoughtology: Let's< think> about Ilm reasoning. ArXiv preprint, abs/2504.071 122 2025.
[45] Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Locally typical sampling. Transactions of the Association for Computational Linguistics, 11:102-121, 2023.
[46] Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D. Manning. An emulator for fine-tuning large language models using small language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[47] Sean O' B'Brien and Mike Lewis. Contrastive decoding improves reasoning in large language models. ArXiv preprint, abs/2309. .091 17, 2023.
[48] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[49] Siru Ouyang, Shuohang Wang, Minhao Jiang, Ming Zhong, Donghan Yu, Jiawei Han, and Yelong Shen. Temperature-centréc investigation of speculative decoding with knowledge distillation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages USA, 2024. Association for Computational Linguistics.
Miami, Florida,
[50] Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. wttpaa//i////a///aaaaaaaa////e
2025. Accessed: 2025-01-24.
[51] Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Infor- mation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
[52] Tian Qin, David Alvarez-Melis Samy Jelassi, and Eran Malach. To backtrack or not to backtrack: When sequential search limits model reasoning, ArXiv preprint, abs/2504.07052. 2025.
[53] Zexuan Qiu, Zijing Ou, Bin Wu, Jingjing Li, Aiwei Liu, and Irwin King. Entropy-based decoding for retrieval-augmeened large language models. 2025.
14

---

[54] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-prose proof q&a benchmark. In First Conference on Language Modeling, 2024.
[55] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv preprint, abs/1707 100000000777.22 2017.
[56] Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In The Thirteenth International Conference on Learning Representations, 2025.
[57] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
[58] Guangming Sheng. Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 3 April 2025, pages 1279-1297. ACM, 2025.
[59] Chufan Shi, Haoran Yang. Deng Cai, Zhisong Zhang, Yifan Wang, Yujiu Yang, and Wai Lam. A thorough examination of decoding methods in the era of LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8601-8629, Miami, Florida, USA, 2024. Association for Computational Linguistics.
[60] Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zetteeoyer. and Wen-tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 2: Short Papers). pages 783-791, Mexico City, Mexico, 2024. Association for Computational Linguistics.
[61] Sungho Shin, Joosoon Lee, Junseok Lee, Yeonguk Yu, and Kyoobin Lee. Teaching where to look: Attention similarity knowledge distillation for low resolution face recognition. In Computer Vision ECCV 2022, pages 631 Cham, 2022. Springer Nature Switzerland.
[62] Richard S Sutton, Andrew G Barto, al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[63] Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning, ArXiv preprint, 0t//2503.01066. 2025.
[64] Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. ReFT: Reasoning with reinforced fine-tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 7601-7614, Bangkok, Thailand, 2024. Association for Computational Linguisties.
[65] Jonas Waldendorf, Barry Haddow, and Alexandra Birch. Contrastive decoding reduces hallu- cinations in large multilingual machine translation models. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22299, St. Julian's s, Malta, 2024. Association for Computational Linguistics.
[66] Junqiao Wang, Zeng Zhang, Yangfan He, Yuyang Song, Tianyu Shi, Yuchen Li, Hengyuan Xu, Kunyu Wu, Guangwu Qian, Qiuwu Chen, et al. Enhancing code Ilms with reinforcement learning in code generation. ArXiv preprint, abs/2412.20361, 2024.
[67] Yuyang WU, Yifei Wang, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of- though length in 1lms. ArXiv preprint, abs/2502.07266, 2025.
15

---

[68] Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. Tokenskip: Controllable thain-of-thought compression in 1lms, 2025.
[69] Zilin Xiao, Hongming Zhang, Tao Ge. Siru Ouyang, Vicente Ordonez, and Dong Yu. Parallel- spec: Parallel drafter for efficient speculative decoding. 2025.
[70] Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. ArXiv preprint, abs/2502.14768. 2025.
[71] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. ArXiv preprint, abs/2412.15115, 2024.
[72] Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, et al. Qwen2. 5-1m technical report. ArXiv preprint, abs/2501.15383, 2025.
[73] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. ArXiv preprint, abs/2503. 2025.
[74] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? ArXiv preprint, abs/2504. 13837, 2025.
[75] Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder automated test-case synthesis. ArXiv preprint, abs/2502.01718, 2025.
[76] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models the wild. ArXiv preprint, abs/2503. 188992, 2025.
[77] Jinghan Zhang, Xiting Wang, Fengran Mo. Yeyang Zhou, Wanfu Gao, and Kunpeng Liu. Entropy- based exploration conduction for multi-step reasoning. ArXiv preprint, 15s/2503.15888, 2025.
[78] Rosic Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RI post-training amplifies behaviors learned in pretraining, ArXiv preprint, abs/2504.07912. 2025.
[79] Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi-modality rl training framework. https: /ttpps//github.. com/hiyouga//asyl.. 2025.
[80] Ming Zhong, Chenxin An, Weizhu Chen, Jiawei Han, and Pengcheng He. Secking neural nuggets: Knowledge transfer in large language models from parametric perspective. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[81] Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsich. R RI-zero's' s" aha moment" in visual reasoning on a 2b non-sft model. ArXiv preprint, abs/2503.05 2025.
16

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims have been validated with extensive empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations in Section 6 and Appendix F.
17

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We released our code, and provided experimental details to reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All the datasets used our paper are open-access, and we also provide an anonymous GitHub repository for our code in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: ctaaps.c//ubbic//ute guwdee/eeeeeeee /CodeSubmis http:////gguieee/eeeuuuuuiiiiii ssi ssionPoiicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
19

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We presented experiment settings and details in Section 3.1.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All of our experimental results are computed on 32 runs, and we also provide the standard deviation for evaluation in Section 4.3 for statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer:
[Yes]
Justification: We mentioned our computation environment to reproduce the experiments in Section 3.1.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:
c csGuideliness ines?
Answer: [Yes]
Justification:
Our research was conducted in accordance with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is methodological paper to activate reasoning capabilities in a large model with small-model guidance, and does not have direct societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
functioning correctly, harms that could arise when the
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
21

---

Justification: Our work does not involve releasing data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited the creators of the assets used in the paper, and provide URLs to assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcom thoode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have released our code, used datasets, and inferenced trajectories, and provided detailed documentation in the anonymous GitHub repository mentioned in the Abstract.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
22

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowd-sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our work only uses LLMs for writing, editing, and formatting purposes that do not impact the core methodology, scientific rigor, and originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: htt////nerripp.pee http://uuppp.eeoooeeeeeeeeeeeeee for what should or should not be described.
23

---

A Implementation Details
A.1 Datasets
We provide the details for all the datasets used in our work as follows. All datasets or benchmarks used in this paper are publicly available online. For mathematical reasoning tasks, include 6 widely used datasets, detailed below:
MATH500 The original MATH collection contains 12,500 problems in total, with 8,000 training and 4,500 test problems, meticulously curated to cover wide range of topies and difficulty levels. Each problem in MATH has a full step-hy-step solution that can be used to teach models to generate answer derivations and explanations. MATH500 (could be found at https: htt//uggiiggaaccc datasets/ HuggingFaceei4 HA////MAT-OO0 is non-standard train/test split of the original MATH dataset [18 [18]. following [33] to avoid the risk of over-fitting and for more efficient testing configurations. These 500 test problems selected uniformly at random, and are representative of the test set as whole.
co/
GSM8K GSM8K (Grade School Math 8K) [6] is dataset of 8,500 high-quality linguistically diverse grade school math word problems. The dataset was created to support the task of ques- tion answering on basic mathematical problems that require multi-step reasoning. The test set of GSM8K (could be found at https: https://hugg h/thgggnggaaacce..c/mm//// httoddaatsett//peaa/ggaaaa includes 1,319 problems in total.
Olympiad Bench Olympiad Bench [17] is originally Olympiad-level bilingual multimodal scientific benchmark, which contains 8,952 math and physics questions from international Olympiads, Chinese Olympiads, Chinese college entrance examinations, and mock exams. To support our testing needs, we select a subset from the Olympiad Bench that is categorized as "open- ended", "text-only" and "competition- be found at https: h/thggggggggaccc co /daaaatts//ttttaaeeeeee Hoothhnnllyyaa maths en_P)MP) for this subset used in our paper.
Together, there are 675 test problems (could
The collection of AIME actually contains problems from the American Invitational Mathe- matics Examination (AIME). AIME is prestigious high school mathematics competition known for its challenging mathematical problems. In our work, we follow previous works and adopt all the 30 problems (could be found https:/ https://huggingae htth//ggggfaac.....////////////e co/datasets//ii wwwttaasttt/gggggggaeeeeeee hawngngFaeeH/aiinn / 2024) in AIME 2024 for testing purposes.
AIME
AMC Similar to AIME, AMC is another very challenging dataset that contain problems in competi- tions, specifically, the American Mathematics Competitions (AMC). The collection of AMC actually contains 40 problems (could be found in total the year of 2023 for our testing set.
co/datasets /matttss//aatataaeee ai ai/amc23)
The testing set of Minerva math is curated in [30], which consists of STEM problems at the undergraduate level. In total, there are 272 problems (could be found at datasets/matee aaw.aairiniaaaath.. 191 of which have numeric solutions and 81 have symbolic solutions.
Minerva
co/
For code reasoning tasks, we incorporate three datasets as following:
HumanEval+ HumanEval+ is an adapted version from the original HumanEval by [38]. Hu- manEval+ extends beyond HumanEval with additional high-quality and automatically generated test inputs to 80x, powered by both LLM- and mutation- based strategies. It contains 164 samples for testing (could be found at https:: http:///hgggaaaaae wttpe//nnan.e..naaaaaaeeaaaeeeee st/dattaset///ealllllsss htwpe////////.aaaaa///////////// httpa////eaapppapsuuuaauaaaappa htw.aaaalllllss).....
MBPP+ HumanEval+ is also an adapted version from the original MBPP by [38]. The construction
process is quite similar to HumanFval+, and it contains 378 samples for the testset (could be found at
/t//////////eee matasets/evalplnn mbppppuuss
24

---

Prompt (a)
<lim_ @im_sstrt>sssstem
You are a helpful assistant.<llmg <lim_endl>
jim_start/>user
(input)
Please reason step by step, and put your final answer within Wboxed(0)<lim endl>
<lim_ assssaaa
Prompt (b)
Ouestoorn//ippu/////sveeeevveess think step by step.
Figure 7: Prompt templates used for mathematical reasoning tasks.
Table 5: Detailed decoding configurations (hyperparameters) used in experiments.
Scttings
gpu_utiline uttliizaiion tensor, parallel_sizz
tensor_parallel
temperature
top_p max_seq len
Qwen-2.5-32B
0.6
0.0
1.0
0.6
0.75
1.0
16,384
1.0
0.95
+ARu
0.75
4
32B-RI.Zero
0.6
4
Qwen-2.5-14B
0.6
2
0.0
1.0
+A R5.5
0.6
1.0
16,384
Ran
0.75
0.95
14B-RLZero
0.6
2
Qwen-2.8 -25-7-
RA.s
0.6
1.0
16,384
1.0
0.95
7B-RL
0.6
LiveCodeBench LiveCodeBench [24] is recently proposed benchmark that aims compre- hensive and contamination- evaluation LLMs for code, which continuously collects new problems time from contests across three competition platforms, namely LeetCode, AtCoder, and CodeForces. In our experiments, there are 880 test samples in total.
A.2 Prompt Templates for Inference
Inference prompts used in this work generally following the common practice in the open-source community. Specifically for mathematical reasoning tasks, there are two kinds of prompts as shown in Figure 7. During our experiments, choose to use Prompt (b) following the training and inference setting in [76] since the model scale is too small to follow the complex chat template and output format instruction of "'\boxed". For both prompt templates, the "(input)" will substituted with the corresponding input problem for each data sample.
select experiments with respect to AR. For 5R,
B Decoding Configurations
Table 5 presents the detailed decoding configurations for our experiments, specifically, the hyperpa- rameters used.
25

---

Table 6: PCR results on various settings.
Backbone Models
AIME MATH500
GPQA
MMLU
Qwen-7B
96.03
96.67
93.73
94.54
Qwen-14B
95.60
96.28
93.67
93.91
Qwen-32B
95.22
96.58
92.93
93.32
Llama-88
91.15
93.16
93.23
90.44
More Results
C.1 Additional PCR Results
We provide a comprehensive view of PCR results on different datasets, model families, and scales used across this paper Table 6.
We can see that our conclusion still holds, with over 90 PCRs across all the different model families and different benchmark data points. The results prove the generalization of RAST.
C.2 Results on Additional Datasets
Apart from the mathematical reasoning and coding tasks explored in Section 3. we also experiment more general reasoning settings. Specifically, we conduct experiments for RAST MMLU and GPQA datasets to further demon- strate the generalization on various test settings. Similarly to previous sections,
Table 7: Experiment results of RAST general reasoning tasks with the Qwen- 2.5-14B and Qwen-2.5-32B models as Mbase and A Simple-RI.
primary models from the Qwen-2.5 family across various sales with the correspond- ing RL-trained variants from SimpleRL-Zoo [76]. The decoding configurations still follow from Appendix B.
Models
MMLU
GPQA
1w5n-225-14B
62,4
24.8
+A
75.1
46.7
SimpleRL-144B
77.8
50.3
The results are shown in Table 7. We can see that the performance brought by RAST is consistent across both datasets, even if the expert RL model is still trained on mathematical reasoning datasets, which further strength- ens the effectiveness and generalization of our method.
Qwen-2.5-328
61.7
38.1
+AR
74.8
44.7
+AR
79.6
49.0
SimpleRI-332B
81.4
48.3
C.3 Performance of Majority@k
We first introduce an additional metric for this additional evaluation.
Definition of Majority@k: This metric evaluates accuracy based on the majority prediction among multiple inference per problem. Formally,
Majority @k N
(majority (ūi, 01.2..... xr,a3 y:)
where the majority function returns the prediction most frequently appearing among the k inference runs for the i-th problem. Compared with Pass @k that represents diversity, the metric of majority @ reflects the robustness and consistency of the model performance.
We visualize the results in Figure 8, showing the progression of majority @ as increases across six mathematical reasoning benchmarks. As expected, majority @ k improves monotonically with larger k reflecting the benefit of aggregating more diverse sampled trajectories. In most cases, our method (gray bars) bridges a significant portion of the performance gap between the base model Mbase (blue line) and the RL-trained model MRL (red line), particularly on datasets like MATH500 and GSM8K, where approach nearly saturates the performance ceiling. This suggests strong consistency among sampled outputs and robust transfer of reasoning behaviors.
In contrast, more challenging datasets such as AIME 2024 and AMC 2023, wider gap remains between RAST and MRr, indicating that these tasks induce more divergent reasoning paths, where achieving high consensus remains difficult. Nevertheless, even in these harder settings, RAST offers
26

---

80
89-A
80
90
40
40
13.3
74.6
72.5
70
0,
84
34.5
35.2
35.5
70
83.1
83.5
23.4
60
50
50.5
62.
60
50
40
72
70.7
75.8
78.4
72
66
78
28
32
31.2
26.4
32.7
0
29.4
32
28
32
30
30
62:1
20
16
16
8
(a) AMC 2023
(b) MATH500
(c) Minerva Math
50
100
100
60
30.7
30.3
34
36.7
40
96.3
96.5
96.3
52
49.3
48.9
30
30
96
96
44
46.2
44
20
10.7
25.4
0
20
96
95.9
94
36
39
36
10
31.3
9.7
0
20
(d) AIME 2024
(e) GSMBK
Olympiad Bench
M RL
bore
Our method
The illustration of majority @k for different values l on 6 mathematical reasoning datasets, where Mbase is Qwen-2.5-32B, RAST uses A ARIB, and M m is the Rl-trained version of Mbase.
Figure 8:
substantial improvements over the base model, demonstrating that reasoning diversity and consensus can be meaningfully enhanced without full RL training.
C.4 Response Length
To better understand the behavioral effects of RAST model generation, examine the average response length per problem across six mathematical reasoning benchmarks, as shown in Table 8. We report the number of output tokens generated by RAST under each setting.
Firstly, we observe that the RL-trained model generally increases the output length compared to the base models, suggesting that it encourages more verbose or exploratory reasoning paths. This behavior aligns with the goal of RL to promote step-by-step reasoning and self- self-verification, which naturally results in longer outputs as the model articulates intermediate steps and justifications. We find that RAST inherits this trait: applying logit deltas from RL-trained experts to base models generally leads to increased response lengths, particularly when using deltas from much smaller expert models (e.g.. A R1.8)) some cases, the response length under RAST even exceeds that of the corresponding RL-trained model. For example, on AMC and GSM8K, Qwen-2.5-P 2.5-7B augmented with A R1.8 produces significantly longer outputs than both its base and RLZero counterparts, reaching average of 1362.8 and 354.9 tokens, respectively.
Additionally, we found that the performance each setting is generally negatively correlated with the length of the generated outputs. Specifically, the length of generated outputs is always the shortest with RAST when we apply AR1ι than ARL5B with 32B base models, More interestingly, we found that the best performance achieved using RAST usually comes with the shortest length, indicating that RAST can achieve both efficiency and effectivvnness if configured properly. This also brings up another door for the recent topic that try compress the reasoning trajectories of RL-tuned models [68, 77, 3, 67, 43].
Overall, these results demonstrate that RAST not only activates reasoning behaviors but also affects the generation style. This controllability opens promising directions for future work in tuning the verbosity and structure of generation by manipulating transfer signals, and supports the broader narrative that RAST serves not just as reasoning enhancer also tool for decoding-time style modulation.
27

---

Branching out
alternatively, another, try, suppose, consider, different, assume, also, option
Backtracking
however, but, mistake, contradiction, wrong, revisit, actually, again, flawed
Self-verification
check, verify, confirm, satisfy, plug, back, substitute, ensure, validate, test
Figure 9: The set of manually curated reasoning tokens that corresponds to three key reasoning behaviours.
Table 8: The response length of output generated per problem by RAST on six mathematical reasoning benchmarks.
Models
Math500
AIME24
AMC
Minerva
Olympiad
GSM8K
Qwen-2.5-32B
566.9
1446.5
782.9
757.8
1103.4
275.6
+ARo..
685.1
1417.5
1024.2
799.5
978.3
333.1
614.5
1033.2
895.8
635.4
870.1
307.8
+ARRua
591.9
1164.6
863.9
605.7
886.9
297.6
32B RLZero
659.2
1183.5
940.4
655.8
992.4
307.2
Qwen-2.5-14B
803.8
1138.9
820.2
921.5
1237.0
227.5
706.6
2092.0
1131.9
1070.6
1438.1
337.6
RAz
606.9
1092.4
919.1
636.3
889.4
309.8
682.7
1477.8
1034.9
667.1
1068.4
318.7
Owen-2.5-78
651.3
1765.9
756.0
657.2
1033.2
263.9
R1.sB
862.7
2127.4
1362.8
944.6
1123.5
354.9
7B-RLZero
682.6
1410.5
1074.6
731.7
1033.7
330.3
C.5 Case Study Token-Level Behavior Shift
As preliminary study mentioned in Section 2.1, we reveal that given the decoding path from RL model MRI., the base model Mhase actually will largely recover the path, with PRC 95. 25. (for 32B model). In this section, delve deeper into this study. Firstly, M tuned by RAST by feeding the decoding path of MRL to M. The PRC in this experiment is O = 96.3 37%, which is even higher. This phenomenon indicates that RAST provides efficient guidance for the base model Mbsse in the search space, echoing the conclusion in [74].
repeat the study using
Apart from the overall quantitative view, we also present more intuitive interpretation in Figure 10. We can see that the generated output from Musse follows only one thinking and contains many erroncous steps without self-verification. However, the output from M demonstrates markedly different reasoning behavior. first proposes and tests candidate solution, then explicitly ver- ifies its correctness, and finally reasons about the function behavior to rule out other possibili- ties. To make it more rigorous, we compute the KL. divergence [26] online during inference as LLD (li, ci), M(1,, where det the current prefix. Notably, we found that the KLDD for tokens such as "check" reaches 837.9, which is far larger than normal tokens that usually stay below 1.0. These behavioral differences underscore the effectiveness RAST in activating reasoning behaviors.
28

---

Math Problem:
for 19
Generated output from A how
a step-by-stop process to solve 1. First, rewrtte 3 (3'F equals 2. Now, rowrtte equation as
9* + 10". 3. Rearrange the equation to isolate 10" side: 9" 19. 4. Divide both sidess of the equation by 9":
or ge 5. Simplify the ieft-hand side using the property of exponents:10" = (9 y = 9 ( 6. Divide both
sides of the equation by 9% ( gr. Z. Let ( Then the equation becomes (5) y. Rearrange the equation
to get quadiatic equation: y 19
Solve the quadratic equation using the quadratic formulay -h= V Anm In
case,
e
Therefore,
Generated output from MR.
To solve equation, we will proceed step by step. Isolate the exponential terms: 10 2. Analyze equation for possible solutions: We
find such that both sides can find solution. 3. Test 3 19 10
equation are equal. Let's test simple values of
This
= 2 solution. Verily there other solutions:
To Bfeck there are any other solutions, we can analyze the behavior of the functions 3 and 10 - 19. The function
3 = (3-y= = 9 an exponential function with base 9. The function 10" 19 is also an exponential function with base 10, For 9 less
shifted down by Both functions are increasing, but grows
19, 9" greater than 10 19. Therefore, 2 the only solution where sides equation equal.
Figure 10: A case study comparing generated outputs for the same math problem sampled from MATH500 from Masse and M obtained by RAST. The red denotes erroneous thinking steps from Mbssc while the green texts indicate remarkably large KLD, with deeper color denoting larger KLD.
D Memory Estimation Details
We estimate the memory considering tensor parallel and CPU offloading (since these are common tricks during training) based on the following dimensions: (i) model memory footprint (FP16), (ii) optimizer states, and (iii) activations & buffers.
Model Memory Footprint To estimate the memory consumed by model parameters, consider GRPO training with three model instances: policy, critic, and reference. For each, the parameter size is calculated (model size / tensor parallelism factor) x 2 bytes, assuming FP16 precision. For example, a 14B model with tensor parallelism would require approximately (14B / 4) 2B x 21GB per GPU. This accounts for only the static model weights without any optimizer states or intermediate activations.
Optimizer States with CPU Offloading. We assume DeepSpeed ZeRO Stage 3 is used to offload optimizer states entirely to CPU memory, which a common practice in current RL training [79]. Under the Adam optimizer, each parameter typically requires FP32 states, leading to total memory footprint of approximately 2x model size x 4 bytes x number of models. These optimizer states are excluded from GPU memory but included in CPU memory estimates. For example, in a 14B GRPO setup, this results in approximately 156 GB of CPU RAM usage for the optimizer alone. In our paper, we mainly focus on the GPU memory overheads; therefore, if using CPU offloading, the memory requirement for optimizer states could almost be neglected.
Activations and Buffer Overhead. Activation memory is estimated based common usage pat- terns for transformer- based LLMs under long context lengths (e.g., 2048 tokens) and moderate batch sizes. We assume gradient checkpointing is enabled reduce activation memory, typically resulting in 12-25 GB usage per GPU depending on model size and training configuration. Additionally, we account for 5-8 GB per GPU for auxiliary memory needs such as gradients, residual connections, attention caches, and NCCL communication buffers. These components are summed to estimate total GPU memory usage across devices.
E Future Directions
In this section, we briefly discuss the potential future directions following RAST.
Ensemble Methods. One natural extension of RAST involves ensemble strategies [9] across multiple small-scale expert models. Given that each expert model may encode slightly different
29

---

reasoning strategies depending on its training trajectory or initialization, aggregating their logit-level deltas could lead to more robust reasoning activation. This ensemble mechanism can be used to reduce variance, enhance generalization across tasks, and potentially adapt to unseen domains with improved resilience. We did very initial exploration in this direction, by ensembling A R14B, A ARB and A R1 to the 32B base model of Qwen2.5 MATH500. We found that the results is actually around 75.6, which does not outperform RAST with a single A ALAB. We suspect the primary reason is due to the similarity reasoning behaviour encoded in AR across scales in the same Qwen model family. Therefore, the simple ensembling approach will hurt the performance, and also bring much memory overheads for inference. However, argue that this is still interesting direction for exploration if we could identify unique and diverse reasoning behaviors of different models before ensembling.
LoRA from Small Models. Another intriguing direction is to bridge the output-space corrections from RAST back into the parameter space through low-rank adaptation (LoRA) [21]. Instead of applying logit deltas directly in the decoding phase, we could naturally distil a lightweight LoRA module [80] for each RL-tuned model to imitate the adjustments suggested by smaller RL-tuned models. We may even build reasoning-centrie LoRA hub [22] that enables gross-reasoning-bbhavii controllable reasoning setting.
generalization and flexible reasoning pattern combination for
Beyond Reasoning. While RAST focuses on reasoning activation, the core idea of activating the reasoning behaviors from smaller RL-tuned models via output-space alignment may generalize to other capabilities, such as code generation, instruction following, or factual grounding. Future work may explore how RAST-style activation compares with or complements other alignment techniques, including preference-based fine-tuning or reward modeling.
F
Limitations
While RAST demonstrates strong empirical performance and introduces practical paradigm for decoding-time reasoning enhancement, it also comes with several limitations that suggest directions for future research.
Lack of Theoretical Understanding. Our method is largely motivated by empirical observations and intuition about how reasoning behaviors are reflected in output distributions. However, the theoretical foundations for why logit-level adjustments from small RL-tuned models can be activated effectively across model scales remain underexplored. deeper understanding when and why such activation works and its potential failure modes would help establish more rigorous guarantees and improve method design.
Limited Gains on More Difficult Benchmarks. Although RAST consistently improves perfor- mance across a range of reasoning datasets, the improvements on more challenging tasks, such as AIME, are relatively modest. These datasets often involve abstract reasoning, multi-step derivations, or comminn-ppecffi heuristics that may not be sufficiently captured by smaller expert models. This limits the degree of reasoning behavior activation and suggests that RAST may benefit from combining with more targeted adaptation techniques.
Computational Trade-offs and Expert Quality Dependency. Despite avoiding full RL fine-tuning on large models, RAST still requires access to a reasonably strong small expert model trained with RL, which can be computationally expensive obtain. Furthermore, the quality and generalization ability of RAST are inherently tied to the effectiveness of this small expert. the expert model is poorly aligned or fails to exhibit robust reasoning behaviors, the transferred deltas may provide little benefit.
30


Reviews:
[{"review_id": "FSro5u1pVP", "reviewer": "Reviewer_pRmh", "summary": "The authors propose RAST, which replaces RL post-training on large LLMs by RL post-training on smaller LLMs and a simple inference-time steering adjustment. The authors show empirically that RAST often allows large LLMs to perform on reasoning tasks as if they were RL post-trained (or even better), while only incurring the costs of RL post-training on smaller LLMs and a negligible steering adjustment. The authors also show that RAST increases trajectory diversity in the large LLM.", "questions": "* It would be useful to have an ablation where the small model is RL trained for the right answer format, and see if RAST still works. Moreover, experiments around the Instruct models mentioned above would be important to understand the method’s validity. Would you be able to kindly provide this?\n\n* It would also be important to have ablations around individual reasoning behaviors to determine which ones are actually important.\n\n* Could the authors explain how RAST outperforms RL on the larger model in Figure 3?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n* The method is clear, simple, and novel. \n* The authors' core insight - reasoning behaviors learnt during RL can be used to elicit better performance from a larger model - is plausible.\n* The authors demonstrate empirically that using the single additive logit difference term allows larger models to perform much better on reasoning benchmarks, without additional fine-tuning or weight edits. This works and scales consistently across model sizes and families.\n\n# Weaknesses\n\n* One of the core issues is that the paper never swaps out or ablates the specific behaviors it claims to transfer (e.g. branching, backtracking, self-verification). Experiments that study whether a small model trained to produce the right formatting of answers, or an s1 (https://arxiv.org/abs/2501.19393)-like approach where the small model can force the larger one to answer, or just testing alternative behaviors would be useful. Without such ablations, it is difficult to say whether RAST is truly improving reasoning skills.\n\n* The SimpleRL-Zoo method is used for the small RL teacher and uses MATH500 in its reward data. This also forms the test set, which makes empirical results on the same invalid. It is possible that the improvements could stem from just re-activating memorized solutions rather than transferring actual reasoning skills.\n\n* The experimental setup has additional issues: All comparisons use the base models rather than instruction tuned versions, and show improvements on these base models. Instruction tuning already elicits reasoning capabilities and performance similar to what RAST seems to provide. It would be important to either compare directly against Instruct tuned models or to use RAST on an Instruct model and see if it still improves performance (along with the ablations mentioned above).\n\n* The experiment measuring path-coverage does so on only 50 teacher forced traces from MATH500 on models of the same family. Qwen’s smaller versions contain distillation data from larger versions and have essentially the same training data, so it is unsurprising that these models have similar distributions on benchmark questions. Broad generalisation about “reasoning alignment” seems overstated.\n\n(The reviewer is considering increasing their score if the weaknesses are adequately addressed in the rebuttal)", "was_revised": true, "final_justification": "Reading the rebuttals, my technical concerns have been largely mitigated. What holds this paper back is the relative lack of conceptual contribution.", "rebuttal": "Thanks for the constructive comments. We sincerely appreciate your recognition of RAST's clarity, novelty, and core insights, as well as the empirical validation of its effectiveness and scalability across models without additional tuning. Below, we provide detailed responses to each of your comments and hope to address any further considerations you may have:\n\n> **W1 & Q2: The lack of ablations isolating the specific reasoning behaviors being transferred, and the importance of determining which reasoning behaviors are important.**\n\nThanks for the thoughtful comment. We agree that demonstrating specific reasoning behaviors being transferred is an important aspect of our claim. Therefore, we conducted such an analysis in **Appendix C.3**. In this analysis, we first manually curate a set of tokens that reflects certain reasoning behaviors as shown in Figure 9. We then statistically test the frequencies of those key tokens as an approximation of appearances for reasoning behaviors. The results are shown in Figure 10. We can see that the reasoning behaviors of RAST are much closer to the corresponding large RL model. We believe that this keyword-based quantitative analysis provides concrete evidence beyond accuracy/performance, indicating that RAST is really steering the base model towards improving real reasoning skills.\n\nWe agree with Q2 on the importance of determining important reasoning behaviors. However, this question lies somewhat beyond the scope of our current work, which aims at developing a method for reasoning activation to reduce the costly RL training effort. Determining different reasoning behaviors remains a valuable future direction for follow-up works, as discussed in lines 750-755. As an initial trial, **our analyses in lines 694-700** reveal the correlation between certain reasoning behaviors with specific datasets. For example, AIME 2024 exhibits a pronounced amount of \"branching out\". This indicates that the AIME dataset needs careful traversing and more sophisticated inductive reasoning capabilities.\n\n> **W2: The use of MATH500 in both training and evaluation raises concerns about whether improvements reflect true reasoning transfer.**\n\nThanks for the comment. We respectfully think *there is a misunderstanding*. We would like to clarify that **MATH 500 is not incorporated in the reward data** for the small RL model training in SimpleRL. In fact, according to Section 2.1 of the Training dataset in [1], the SimpleRL authors used MATH as the training data, which differs from MATH500. That being said, the training and test datasets do not overlap. So the problem of “re-activating memorized solutions” does not apply in this case.\n\nDespite this, our evaluation is not limited to MATH. Specifically, we evaluate RAST across 5 other diverse mathematical reasoning benchmarks—including GSM8K, AIME 24, AMC23, Minerva, and Olympiad—ensuring that our conclusions are based on a comprehensive and diverse set of tasks, rather than bias or coincidence arising from any single dataset.\n\n> **W3 & Q1: Comparison and integration of RAST with instruction-tuned models.**\n\nThanks for your constructive comment. We only incorporate base models in our setup because:\n-  It is common practice in the literature of RLVR [2] to evaluate methods on base models (explained as zero-RL in line 67), and \n- The small RL models we used are trained from a base model directly. We intentionally avoid introducing additional instruction-tuned priors to ensure a clean experiment setting for examination of RL-trained models only.\n\nTo address your concerns, we additionally conduct experiments on direct inference with instruction-tuned models as a comparison. The results are shown in the following table (we use the instruction-tuned version of Qwen2.5 series):\n\n|       | MATH500 | AIME | AMC  | Minerva | Olympiad | GSM8K |\n|-------|---------|------|------|---------|-----------|--------|\n| 7B    | 70.3    | 8.9  | 49.9 | 23.2    | 33.9      | 89.5   |\n| 14B   | 75.5    | 12.8 | 55.3 | 28.1    | 37.1      | 92.9   |\n| 32B   | 80.3    | 16.6 | 63.7 | 28.1    | 42.5      | 95.4   |\n\n\nTogether with Table 1 in the paper, we can see that direct inference with instruction-tuned models does not match the performance of RAST, especially for difficult datasets such as AIME and Minerva. This experiment further strengthens the effectiveness of RAST.\n\nWe also conducted statistical analysis on reasoning token behaviors and compared the normalized token frequencies between the instruction-tuned model and RAST following Appendix C.3:\n\n|                     | MATH500 | AIME | AMC  | Minerva | Olympiad | GSM8K |\n|---------------------|---------|------|------|---------|-----------|--------|\n| **32B-instruct**    |         |      |      |         |           |        |\n| Branch out          | 0.39    | 1.10 | 0.70 | 0.25    | 1.03      | 0.06   |\n| Backtrack           | 0.18    | 0.66 | 0.34 | 0.15    | 0.42      | 0.03   |\n| Self-verify         | 1.01    | 1.85 | 1.33 | 0.83    | 1.98      | 0.20   |\n|                     |         |      |      |         |           |        |\n| **32B + $\\Delta$ 14B** |         |      |      |         |           |        |\n| Branch out          | 0.46    | 1.41 | 0.75 | 0.30    | 1.35      | 0.07   |\n| Backtrack           | 0.23    | 0.79 | 0.43 | 0.21    | 0.60      | 0.03   |\n| Self-verify         | 1.10    | 2.30 | 1.47 | 0.86    | 2.25      | 0.21   |\n\n\nWe found that RAST is better than an instruction-tuned model in terms of reasoning behavior activation.\n\n> **W4: Path Coverage Rate is measured on limited, closely related models.**\n\nThanks for your insightful comment. We conducted experiments entailing more datasets, including math and general reasoning (AIME, GPQA, MMLU), and an additional model family (Llama 3.1-8B) for PCR calculation. The results are shown in the following table:\n\n|            | MATH500 | AIME   | GPQA   | MMLU   |\n|------------|---------|--------|--------|--------|\n| Qwen-7B    | 96.03   | 96.67  | 93.73  | 94.54  |\n| Qwen-14B   | 95.60   | 96.28  | 93.67  | 93.91  |\n| Qwen-32B   | 95.22   | 96.58  | 92.93  | 93.32  |\n| Llama-8B   | 91.15   | 93.16  | 93.23  | 90.44  |\n\n \nWe can see that our conclusion still holds, with over 90 PCRs across all the different model families and different benchmark data points. The results prove the generalization of RAST. We will add these results to Section 2.1 in the newer version of our paper.\n\n> **Q3: Figure 3 interpretation.**\n\nThe gray bar in Figure 3 represents the performance by RAST, where the red line indicates the performance of larger RL models. We can see that in all benchmarks except AIME 24, RAST (gray bar) is always above the RL models (red line). On AIME 24, when k grows to 32, RAST chases the performance of larger models.\n\n[1] Zeng, Weihao, et al. \"Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.\" arXiv preprint arXiv:2503.18892 (2025).\n\n[2] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025)."}, {"review_id": "sdVF2inoHd", "reviewer": "Reviewer_zUV7", "summary": "In this paper, the authors proposed a method called RAST,  trying to enhance the reasoning capabilities of large language models (LLMs) by transferring reasoning behaviors from smaller models trained with reinforcement learning (RL). The authors assume that the changes in output probabilities induced by RL are largely model-size invariant, allowing for efficient reasoning activation in larger models without the need for full-scale RL training, however, this assumption does not have any theoretic support.  RAST involves injecting logit adjustments from a small RL-trained model into larger models, the experimental results show performance improvements on various mathematical reasoning benchmarks.", "questions": "I have listed all the weak points of the paper, which are the questions authors need to address.", "limitations": "See the weak points listed above.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1.The core idea of transferring knowledge from smaller models to larger ones is not entirely new. Similar concepts have been explored in other domains, such as knowledge distillation and model compression. The paper does not provide a clear distinction or significant advancement over these existing methods. \n2.The experiments are limited to mathematical reasoning tasks and specific datasets. While the results are promising, the paper does not demonstrate the generalizability of RAST to other types of reasoning tasks or domains. The effectiveness of RAST on more complex or diverse reasoning scenarios remains unexplored.\n3.The paper lacks a formal theoretical analysis of why the logit adjustments from smaller RL-trained models can effectively activate reasoning behaviors in larger models.\n4.The performance of RAST is highly dependent on the quality of the small RL-trained models. If the small models are not well-trained or fail to capture robust reasoning behaviors, the transferred adjustments may not provide significant benefits. This dependency limits the applicability and reliability of RAST.\n5.While RAST avoids full RL fine-tuning on large models, it still requires access to a reasonably strong small expert model trained with RL, which can be computationally expensive to obtain. The paper does not provide a detailed analysis of the trade-offs between the computational cost of training the small expert models and the benefits gained from transferring their reasoning behaviors.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper. Below, we provide detailed responses to each of your comments and respectfully hope you can consider adjusting the current rating:\n\n> **W1: The idea of knowledge transfer from small to larger models is not new.**\n\nWe respectfully believe that this concern may reflect *a misunderstanding of our contribution*.\n\nFirst, we are not claiming to build a brand-new method for simply transferring knowledge from small to large models. Instead, we first reveal an important insight by introducing and verifying our key hypothesis in line 40. Built upon our insight, we propose a simple and intuitive method to activate reasoning capabilities in large models through a smaller RL-trained model. This way, we can bypass the costly training resource by training large RL models.\n\nTechnically, our methodology belongs to the research line of decoding-time steering of model behaviors, and **we have already discussed this research line in detail in Section 5.2**. It differs fundamentally from knowledge distillation or model compression that you mentioned.\n\n> **W2: RAST is only evaluated on math reasoning tasks and specific tasks.**\n\n(i) In the community of RL for reasoning, mathematical reasoning problems are adopted as common practices for evaluation traditions [1,2], so we follow the tradition and mainly use mathematical reasoning to test RAST in a controlled setting. \n\n(ii) Our goal is not to claim a universal solution for all general reasoning problems. We believe that it is unrealistic to expect a single paper to address the full space of general reasoning tasks. Instead, we aim to provide a focused and well-supported demonstration that reasoning activation via test-time RL-guided decoding is effective and efficient—starting with math reasoning as a representative domain.\n\n(iii) We have conducted experiments on domains other than math. In section 3.4, we provide experiments on code reasoning and results in Table 3 demonstrate RAST’s generalizability. During the rebuttal period, we also test on other datasets like MMLU and GPQA. Following the experiment setting in Section 2.1, we first calculate the PCR for the Qwen-2.5 series in the following:\n\n|      | MMLU  | GPQA  |\n|------|-------|--------|\n| 7B   | 93.73 | 94.54  |\n| 14B  | 93.67 | 93.91  |\n| 32B  | 92.93 | 93.32  |\n\n\nWe can see that the PCR for other general reasoning tasks (MMLU, GPQA) is also pretty high, with over 90%. This indicates that our current hypothesis holds for reasoning generalization. \n\nWe then apply RAST to MMLU and GPQA, and the results are shown in the following:\n\n|                 | MMLU  | GPQA  |\n|-----------------|-------|--------|\n| 14B base        | 62.4  | 24.8   |\n| 14B + $\\Delta$ 7B  | 75.1  | 46.7   |\n| 14B RL          | 77.8  | 50.3   |\n| 32B base        | 61.7  | 38.1   |\n| 32B + $\\Delta$ 7B  | 74.8  | 44.7   |\n| 32B + $\\Delta$ 14B | 79.6  | 49.0   |\n| 32B RL          | 81.4  | 48.3   |\n\nWe can see that the performance brought by RAST is consistent across both datasets, which further strengthens the effectiveness and generalizability of our method.\n\n[1] Wang, Yiping, et al. \"Reinforcement learning for reasoning in large language models with one training example.\" arXiv preprint arXiv:2504.20571 (2025).\n\n[2] Havrilla, Alex, et al. \"Teaching large language models to reason with reinforcement learning.\" arXiv preprint arXiv:2403.04642 (2024).\n\n> **W3: RAST lacks a formal theoretical analysis.**\n\nOur work is primarily application-driven, aiming to demonstrate a practical and effective method for transferring reasoning behaviors across model scales. That being said, while a formal theoretical analysis would certainly enrich the understanding of the underlying mechanisms, we believe it is not a prerequisite for demonstrating the utility and novelty of our approach in this setting.\n\nAdditionally, we actually already provided a bunch of analyses in the paper, trying to offer insights on “why the logit adjustments from smaller RL-trained models can effectively activate reasoning”. The analyses include PCR (Section 2.1), transferability signal (Section 4.1), and token-level behavior shift (Section 4.2). \n\n> **W4: The performance of RAST is highly dependent on the quality of the small RL-trained models.**\n\nWe believe this is not a weakness or limitation, but rather an expected and inherent feature — similar to any machine learning system, the quality of the model’s output definitely is affected by the input to the system. \n\nRAST is designed to be modular and model-agnostic: it can leverage a reasonably good small model that encodes helpful reasoning behaviors, without requiring changes to the large model or costly joint training. This flexibility actually enhances RAST’s applicability, as users can plug in different small experts tailored to their domain or task without retraining the large model.\n\nIn our experiments, we show that small RL-trained models across different scales can provide meaningful guidance to much larger base models, resulting in consistent performance improvements across diverse reasoning benchmarks. This highlights the robustness of the transfer/activation mechanism.\n\n> **W5: Lack a detailed analysis of the trade-offs between the computational cost of training the small expert models and the benefits gained.**\n\n**We actually already have such an analysis in Section 4.4**. Results in Table 4 show that with only 50% training resources, RAST can recover more than 80% performance of the trained models. In fact, the efficiency advantage is exactly one of our major claims made in the paper.\n\n**We hope the above responses address your concerns. We are also happy to engage in discussions with you if the reviews could be more constructive and detailed.**"}, {"review_id": "BTgvXwBV0x", "reviewer": "Reviewer_Tjf7", "summary": "This paper introduces a new framework Reasoning Activation in LLMs via Small-model Transfer (RAST), an efficient alignment technique that transfers reasoning ability from a small reasoning model to larger reasoning model. The method is started from clear motivation and observation that the majority of token probabilities remain even after the base model has been trained with reinforcement learning. It just selectively adjusts the probabilities of certain tokens which influences the model’s deeper reasoning such as branch out, self-verification, and backtracking. Based on the observations, the authors propose RAST to transfer the reasoning capability from smaller reasoning models to larger base models by adding probability differences between rl trained and base model in small model to larger base model. Experiments conducted across various math and coding domains, using models of different sizes and architecture shows that RAST consistently improves the performance without heavily training the large model. Additionally, the paper shows some interesting analysis about the reasoning behavior of LLMs.", "questions": "[Q1] Can the method be combined with speculative decoding to accelerate the inference speed? Discussing this could make the method more efficient in terms of both latency and training costs.\n\n[Q2] Is there a way to apply RAST between models that use different tokenizers? For example, from Deepseek-R1 to Llama.\n\n[Q3] Please discuss about the concept of RAST-aware training, which could even further enhance model performance training small models", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "[S1] The paper is well-written and easy to read with clear visualizations.\n\n[S2] The motivation and the result of observations are intuitive and provide a meaningful way to understand the nature of the reasoning of LLMs.\n\n[S3] The method is well-motivated and demonstrates great performance.\n\n[S4] RAST has the potential to significantly enhance the model efficiency by reducing the training cost.\n\n[W1] Currently, RAST can only be used between models that have the same tokenizer which limits flexibility of the method.\n\n[W2] The method requires 3 models simultaneously during inference time, which necessitates heavy gpu memory and may increase latency. \n\n[W3] Some typos. L150: meaningless hyperlink on text,  L224: (MBase, MRL), L743: (MAHT500)", "was_revised": true, "final_justification": "After reading the authors' rebuttal and further discussions with the authors, I have decided to maintain my score of 5. My main justification is that most of my initial concerns have been addressed, although some issues remain.\n\n**Resolved**\n\n1. Compute budgets of RAST: The authors have demonstrated the computation budget comparison, which can prove the significance of reducing the training computation while increasing the test-time budget.\n\n2. Some discussion about future works: The authors gave a great discussion about interesting future works such as combining with speculative decoding and RAST-aware training.\n\n**Unresolved**\n\n1. Difficulties in using models with different tokenizers: While the authors referenced prior work and suggested potential solutions, these approaches remain suboptimal and are difficult to generalize in practice.", "rebuttal": "Thank you for the encouraging review and constructive comments! We are happy that you think RAST is intuitive and well-motivated, with great performance and significant potential for efficiency. Below, we provide detailed responses to each of your comments and hope to address any further considerations you may have:\n\n> **W1 & Q2: Applying RAST between models with different tokenizers.**\n\nThanks for the insightful comment. While we assume the same tokenizers across our experiments, we think that RAST could be applied between models with different tokenizers. The core technique here is tokenizer alignment/mapping across different LLMs, which is another significant research area in the community. So we do not incorporate it in our paper. RAST actually complements those methods, for example, Contextual Dynamic Mapping [1] could be first applied and then we can calculate the distribution for decoding accordingly.\n\n[1] Chen, Yijie, et al. \"Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping.\" ACL Findings 2025.\n\n> **W2: RAST requires 3 models simultaneously for inference.**\n\nThanks for your comment. We acknowledge that our current setup involves 3 models at inference time, which introduces additional memory and compute overhead during inference. However, the inference usage is in exchange for the computationally costly training. We estimate the training and inference FLOPs based on [1] and training hardware/time in [2]. Specifically, for 14B models, they used 16 H100-80G GPUs for 15 hours; and for 32B models, 64 H100-80G GPUs are used for 36 hours. We assume peak FLOPs for H100 as 1979 TFLOPS with 50% FLOPs utilization during training. The results are presented in the following table:\n\n|                      | Training     | FLOPs         | Inference                            | FLOPs/token     |\n|----------------------|--------------|---------------|--------------------------------------|-----------------|\n| 32B RL               | 32B RL       | 8.21*10^21    | 32B RL                               | 6.4*10^10       |\n| 32B base + $\\Delta$ 14B | 14B RL       | 8.55*10^20    | 32B base, 14B RL, 14B base           | 1.76*10^11      |\n\n\n\nWe can see that the reduced training FLOPs are all worth the increased inference FLOPs. Additionally, the inference stage could be combined with other techniques that can further improve the efficiency, just as you mentioned in Q1, which we will elaborate on later.\n\n[1] Kaplan, Jared, et al. \"Scaling laws for neural language models.\" arXiv preprint arXiv:2001.08361 (2020).\n\n[2] Zeng, Weihao, et al. \"Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.\" arXiv preprint arXiv:2503.18892 (2025).\n\n> **W3 & Presentation: Typos.**\n\nThanks for pointing them out! We have fixed them accordingly.　\n\n> **Q1: Potential of combining RAST and speculative decoding to improve inference speed.**\n\nThanks for the interesting question. We believe there is great potential for RAST to be integrated with speculative decoding for inference acceleration. We provide some initial thoughts here. \n\nOne direct way is to use a draft base model to propose tokens (e.g., 7B base model), and then verify them using the full RAST-composed logits. This may reduce the number of full composite forward passes and decoding steps, especially when the draft is accurate.\n\n> **Q3: Discussion on RAST-aware training.**\n\nThanks for the insightful question. We believe RAST-aware training holds great potential in boosting model performance. We envision several directions in the following:\n\n(i) **RAST-regularized RL training:** One promising approach is to convert the logit deltas and the corresponding branching/backtracking signals into explicit regularization terms during training [1]. These signals can penalize or encourage certain decision patterns, enabling small models to internalize the reasoning behaviors by enforcing trajectory exploration/exploitation during the RL training process.\n\n(ii) **Adapter-based distillation:** RAST outputs/deltas can be used as supervision targets to train task-specific adapters or lightweight finetuning modules (e.g., via LoRA or LoRA Hub-style infrastructure). For the task-specific adapters, speculative decoding [2] would be a good direction to consider. We could also decompose different reasoning behaviors from RAST via LoRA, and achieve controllable and composable reasoning in the style of LoRA-Hub.\n\n(iii) **RAST-enhanced data augmentation:** RAST can be used to generate multiple high-quality and diverse trajectories. For example, we can use test-time scaling to generate new trajectories by sampling on the position of key reasoning tokens. These trajectories—enriched with exploratory behavior and reasoning patterns—serve as additional supervision signals, effectively augmenting the training dataset for training smaller language models to be generalizable and effective. \n\nOverall, we think RAST provides a new paradigm for training efficiency and provides huge potential for follow-up work. We had some initial discussions on Appendix E, and we will incorporate these new insights into the next version of our paper.\n\n[1] Wang, Shenzhi, et al. \"Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning.\" arXiv preprint arXiv:2506.01939 (2025).\n\n[2] Fu, Tianyu, et al. \"R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing.\" arXiv preprint arXiv:2505.21600 (2025)."}, {"review_id": "RR7mqrooqr", "reviewer": "Reviewer_YtSM", "summary": "The authors propose RAST, a decoding-time method that performs a reasoning knowledge transfer from a smaller to a larger model. The subject of the transfer is what the authors hypothesize to be small, selective adjustments of LM head probabilities. The authors claim that RAST enables large models to match/exceed performance they would otherwise gain by RL on math and coding tasks.\nThe experiments of the paper are designed to demonstrate that RAST improves decoding diversity and pass@k accuracy, overcoming what the authors claim to be known limitations of RL-based approaches. RAST is positioned as a \"new avenue for decoding-time reasoning enhancement\".", "questions": "See weaknesses.", "limitations": "No on the limitations and no on the potential negative societal impact, though the latter is not particularly important given the nature of the work.\n\nOn the topic of limitations, the paper does include some discussion of the trade-offs in transfer effectiveness, there is no dedicated part of the paper making conceptual outline of the work's/experimentation's limits.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. Clear formulation of the cornerstone hypothesis of the paper.\n2. Comprehensive, extensive evaluation (Table 2, Table 3, Figure 3), supplemented by clear interpretation of results (Sections 3.2 and 3.3).\n3. Contemporary relevance of the work, especially on the front of resource efficiency (1-2x memory and 2x compute reduction with 60-80% average performance recovery; Section 4.4).\n\n**Weaknesses**\n1. The authors put the review of the related work at the end of the paper but do not use this to their advantage to make concrete comparisons and links to related work. On this front, I find the current state of Section 5 lacking. \n2. I question the relevance of the inclusion of Figure 5 (referenced vaguely from Section 4.2). Perhaps the authors could elaborate on the connection between the Figure 5 and lines 237-245. The current formulation is largely anecdotal and borderline irrelevant, in stark comparison to some other parts of the paper.", "was_revised": true, "final_justification": "In the light of the author response I increase my score.", "rebuttal": "Thank you for the positive review and constructive comments! We are grateful for your recognition of our paper’s clear formulation, cornerstone hypothesis with comprehensive experiments and clear analyses. Below, we provide detailed responses to each of your comments and hope to address any further considerations you may have:\n\n> **W1: The related work section lacks concrete comparisons and integration with the main discussion.**\n\nThank you for the feedback. The current related work section is organized based on the unique position of our paper. Our goal is to establish a new paradigm for leveraging small RL-trained models to guide larger models at test time, thus circumventing the high cost of RL training for large models. To the best of our knowledge, there is no existing baseline that directly fits this setup. \n\nTherefore, we choose two subsections of topic-wise — RL training for reasoning models, and technique-wise — decoding-time strategy for behavior steering.\nWe will further refine Section 5 to make these contributions more explicit and will contrast our approach with the literature (e.g., training-free alignment) following your advice.\n\n> **W2: The relevance and clarity of Figure 5 are unclear.**\n\nThanks for the constructive suggestion. We appreciate the opportunity to clarify the purpose of Figure 5 and its connection to Section 4.2 (Lines 237–245). \n\nFigure 5 serves as a concrete, intuitive illustration of the token-level behavior shift we discuss in Section 4.2. While our primary analysis is quantitative (e.g., high PRC scores, KLD spikes on specific tokens), Figure 5 complements this by showing how these shifts manifest in actual model outputs. Specifically, it demonstrates how the RAST-guided model $\\tilde{M}$ introduces reasoning-critical steps—such as proposing, verifying, and ruling out possibilities—that are missing from the base model $M_{base}$. \n\nLines 237–245 describe this exact difference in reasoning patterns: while $M_{base}$​ proceeds deterministically and makes early errors, $\\tilde{M}$ engages in more structured reasoning, including self-verification (\"check\") and hypothesis testing (\"possible\"). These moments of divergence correspond precisely to tokens with high KLD, as quantified and highlighted in green in Figure 5. This visual alignment between semantic generation and measured divergence provides strong and interpretable support for the behavior shift hypothesis. Therefore, Figure 5 is meant to enhance interpretability by showing exactly what kinds of reasoning behaviors are being introduced by RAST and where. We will revise the text to make this connection more explicit.\n\n> **Limitation: Lack of a dedicated discussion outlining the conceptual and experimental limitations of the work.**\n\nThanks for your attention to this point. Due to space limitations, we moved the limitation discussion of RAST to the Appendix. Please refer to **Appendix F** for a conceptual outline of limitations both method-wise and experiment-wise."}, {"review_id": "1nelrE7zux", "reviewer": "Reviewer_o8y4", "summary": "The paper proposes RAST, a method to transfer reasoning capability from a small RL-trained model to a larger base LLM without retraining the large model. The authors aim to mitigate the cost of RL training, which only adjusts output distributions on a small subset of reasoning-critical tokens. The authors show that adjustments can be extracted as logit differences between a small model and its RL-enhanced version. At inference time, injecting this difference into the logits of a larger base model can improve reasoning performance.", "questions": "Same as weaknesses. The additional questions are:\n\n1. The method proposed in the paper is interesting. An related quesiton is that how the relative size of LLMs influences the performance. For example, whether the difference in a 100M LLM can be used to improve the performance of a 70B model. If not, what is the failure relative size.\n\n2. For robustness of temperature and $\\lambda$, it is better to change them jointly. Changing them separately may hide some sensitivity.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. RAST elegantly bypasses costly RL on big models by reusing small-model RL adjustments.\n\n2. The paper verifies an interesting insight: RL only perturbs output probabilities of reasoning-critical tokens.\n\nWeaknesses:\n\n1. The authors mainly focus on the mathematical reasoning problems. However, the **Hypothesis** in the paper does not constrain itself to the math problems. For example, whether the reasoning for general tasks also shares this property is still unknown. Thus, for a logical argument, either experiments in other domains or constraining the argument to math is necessary.\n\n2. Another concern is whether such an argument is influenced by the RL algorithm. For example, whether GRPO/PPO/REINFORCE/RAFT  share the same property is still unknown. The **Hypothesis** should be refined from this perspective.\n\n3. Section 2 is overall too coarse. The presented experimental results cannot support the hypothesis. For example, the different tokens of the base and RL models have not been statistically studied. Figure 10 is hard to understand and interpret, since the normalization procedure is not presented.", "was_revised": true, "final_justification": "The authors' rebuttal largely resolves my concerns. However, since many experimental results are provided in the rebuttal period, the detailed experimental settings are not provided to further justify the quality of these results. Thus, I am toward the borderline of acceptance.", "rebuttal": "Thanks for your effort in reviewing our paper and the insightful comments. We appreciate your recognition of our paper’s elegance and interesting insights. Below, we provide detailed responses to each of your comments and hope to address any further considerations you may have:\n\n> **W1: Validation of the Hypothesis on reasoning for general tasks.**\n\nThanks for the thoughtful review. In the community of RL for reasoning, mathematical reasoning problems are adopted as common practices for evaluation traditions [1,2], so we follow the setting and mainly use mathematical reasoning to test our hypothesis. \n\nTo address your concern, we conducted an additional experiment on general reasoning benchmarks, including MMLU and GPQA. Following the experiment setting in Section 2.1, we first calculate the PCR for the Qwen-2.5 series in the following:\n\n|      | MMLU  | GPQA  |\n|------|-------|--------|\n| 7B   | 93.73 | 94.54  |\n| 14B  | 93.67 | 93.91  |\n| 32B  | 92.93 | 93.32  |\n\n\nWe can see that the PCR for other general reasoning tasks (MMLU, GPQA) is also pretty high, with over 90%. This indicates that our current hypothesis holds for reasoning generalization. \n\nWe then apply RAST to MMLU and GPQA, and the results are shown in the following:\n\n|                 | MMLU  | GPQA  |\n|-----------------|-------|--------|\n| 14B base        | 62.4  | 24.8   |\n| 14B + $\\Delta$ 7B  | 75.1  | 46.7   |\n| 14B RL          | 77.8  | 50.3   |\n| 32B base        | 61.7  | 38.1   |\n| 32B + $\\Delta$ 7B  | 74.8  | 44.7   |\n| 32B + $\\Delta$ 14B | 79.6  | 49.0   |\n| 32B RL          | 81.4  | 48.3   |\n\n\nWe can see that the performance brought by RAST is consistent across both datasets, which further strengthens the effectiveness and generalizability of our method.\n\n[1] Wang, Yiping, et al. \"Reinforcement learning for reasoning in large language models with one training example.\" arXiv preprint arXiv:2504.20571 (2025).\n\n[2] Havrilla, Alex, et al. \"Teaching large language models to reason with reinforcement learning.\" arXiv preprint arXiv:2403.04642 (2024).\n\n> **W2: Refine hypothesis w.r.t. different RL algorithms.**\n\nThanks for raising this point. From a literature perspective, prior work [1] has shown that RL fine-tuning—regardless of the specific algorithm used—tends to result in models that share similar weights with only a small portion of the parameters modified.\n\nFrom an experimental standpoint, we have also conducted additional analysis using models trained with different RL algorithms from [2] to examine whether similar behavior transfer effects hold in the following table. Note that all the base models here are Qwen-2.5-Math-7B series following [2], and we report the results on MATH500.\n\n|                     | GRPO   | PPO    | (Weighted-)REINFORCE |\n|---------------------|--------|--------|-----------------------|\n| PCR on MATH500      | 97.04  | 96.78  | 97.55                 |\n\n\nPCRs remain pretty high and consistent for RL models trained with a diverse range of algorithms (GRPO/PPO/REINFORCE). Therefore, both conceptually and experimentally, we have sufficient evidence to believe that our hypothesis is generalizable to different RL algorithms.\n\n[1]  Mukherjee, Sagnik, et al. \"Reinforcement Learning Finetunes Small Subnetworks in Large Language Models.\" arXiv preprint arXiv:2505.11711 (2025).\n\n[2] Zhu, Xinyu, et al. \"The surprising effectiveness of negative reinforcement in LLM reasoning.\" arXiv preprint arXiv:2506.01347 (2025).\n\n> **W3: Statistical analysis on different tokens for base and RL models to better support the Hypothesis.**\n\nThanks for your comment and for reading our Appendix. **Figure 10 in Appendix C.3** is exactly such a statistical analysis for demonstrating tokens related to reasoning behaviors in base, RL, and RAST models. We first manually curate a token list corresponding to three different reasoning behaviors. Then we use Spacy to statistically calculate the token frequencies in model-generated responses. Specifically, we compute the token appearances for each category (branch-out, backtrack, self-verify) in each math problem’s 32 generated trajectories, with the averaged number of appearances per trajectory as normalized frequency. For example, in Figure 10 (a), 1.101 means that tokens belonging to “self-verify” appear 1.101 times on average per problem. As stated in lines 694-700, RAST steers the model behavior from base models by actually changing the corresponding reasoning behaviors. \n\nTokens are important signals demonstrating reasoning behaviors from models, and are a widely adopted approach for analysis in the community [1,2]. Therefore, we believe our hypothesis and the main conclusions are well justified through the experiments mentioned above.\n\n[1] Gandhi, Kanishk, et al. \"Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.\" arXiv preprint arXiv:2503.01307 (2025).\n\n[2] Yeo, Edward, et al. \"Demystifying long chain-of-thought reasoning in llms.\" arXiv preprint arXiv:2502.03373 (2025).\n\n> **Q1: How does the relative size between the small and large models affect RAST’s effectiveness?**\n\nThank you for the interesting question. In our current setup, we use a 1.5B RL-trained model to guide larger base models ranging from 7B up to 32B, and we consistently observe performance improvements across tasks. This suggests that RAST can be effective even when the small model is an order of magnitude smaller than the target model.\n\nIn practice, it is commonly observed that RL training becomes more effective at larger model scales [1], while very small models (e.g., ~100M) often struggle to learn meaningful reasoning behaviors via reinforcement learning. In many such cases, even SFT can outperform RL. As a result, directly using a 100M RL-trained model to guide a 70B model is unlikely to yield strong benefits. In this case, those extremely small models can be trained, e.g., via data distillation from a larger RL expert, offering another lightweight and performant alternative.\n\nThat being said, our primary motivation is to avoid the high cost of applying RL to large models, not necessarily to minimize the size of the small model. Training a small-sized model, say a 1.5B RL model, is already much cheaper and more accessible, yet still sufficient to capture reasoning behaviors.\n\nAdditionally, in a conceptual view, the $\\Delta$ captured is used as the key driver of transfer effectiveness. A larger $\\Delta$ intuitively tends to indicate more useful reasoning signals, which in turn improves the impact of the guidance on the target model.\n\n[1] Cao, Yuji, et al. \"Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods.\" IEEE Transactions on Neural Networks and Learning Systems (2024).\n\n\n> **Q2: Jointly change for sensitivity analysis.**\n\nThanks for your constructive comment. We have conducted the experiments jointly, and the results are listed in the following (here we only list a subportion due to character limit in rebuttal; row indicates different $\\lambda$ while column indicates different temperatures):\n\n\n|       | 0.8  | 0.9  | 1.0  | 1.1  | 1.2  |\n|-------|------|------|------|------|------|\n| 0.3   | 80.5 | 80.4 | 80.6 | 80.2 | 80.1 |\n| 0.4   | 80.4 | 80.4 | 80.0 | 80.2 | 80.0 |\n| 0.5   | 80.9 | 81.2 | 81.3 | 80.2 | 80.3 |\n| 0.6   | 81.4 | 81.4 | 80.9 | 80.4 | 80.2 |\n| 0.7   | 81.0 | 81.0 | 80.6 | 80.2 | 80.0 |\n\n\nWe can see that our conclusion still holds: when the choice is within a certain range for decoding temperature and $\\lambda$, RAST demonstrates a robust performance and is insensitive to parameter choices."}]

Author rebuttal:

Meta-review: Brief overview of the paper.    
RAST is a decoding-time technique that steers a large base model using a small RL-trained model’s logit residuals relative to its base counterpart. The paper motivates this with high path-coverage overlap between base and RL models and reports gains on math and some coding benchmarks, together with a memory-focused efficiency argument.  

 Strengths  
- Clear, simple mechanism with coherent motivation.  
- Consistent empirical gains across model sizes and families on multiple benchmarks.  
- Useful analyses (e.g., path coverage; token-level behavior signals) and a practical efficiency narrative focused on avoiding large-model RL training.  

 Weaknesses / what is missing  
- No direct comparisons against established decoding-time contrastive methods, RAST’s update is mathematically a contrastive logit adjustment, yet there is no direct comparison to DExperts (expert/anti-expert product-of-experts), DoLa (contrasting layers), or Contrastive Decoding on the same reasoning setups, leaving attribution of gains uncertain.  
- Potential evaluation confounds (possible train–test proximity on math datasets) and lack of uncertainty quantification; variance across seeds/temperatures/prompts is not reported in the main results.  
- Incomplete efficiency picture, memory estimates are provided, but end-to-end latency/throughput under the three-model decoding setup are not benchmarked in-paper.  

 Principal reasons for recommendation  
While the idea is timely and empirically promising, the absence of head-to-head decoding-time baselines, unresolved evaluation confounds, and missing latency/throughput evidence weaken the strength of the central claims.

 Discussion and rebuttal summary.  
Reviewers highlighted missing behavior ablations, possible data overlap, lack of instruction-tuned baselines, related-work integration, tokenizer constraints, and inference overhead. The authors added token-frequency behavior analyses, clarified training data distinctions, provided instruction-tuned comparisons, and later shared preliminary tokens/s numbers indicating modest latency. Some concerns were alleviated, but several outstanding items (direct baselines vs contrastive decoding, formal contamination audits, standardized efficiency measurements) remain only partially addressed.

Decision  
Borderline. Slightly above the acceptance threshold, provided that the authors address the missing comparison with contrastive decoding methods: DExperts (expert–anti-expert product-of-experts), DoLa (contrasting layers), or Contrastive Decoding in the final revision.
Decision: Accept (poster)