Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Authors: Baohao Liao
Yuhui Xu
Hanze Dong
Junnan Li
Christof Monz
Silvio Savarese
Doyen Sahoo
Caiming Xiong
Venue: ICML 2025 poster (2025)
Keywords: Large language models
Process reward model
Speculative decoding
Primary area: deep_learning->large_language_models
TL;DR: Reward-Guided Speculative Decoding for faster inference and better accuracy.

Abstract: We introduce Reward-Guided Speculative Decoding (RSD), a novel framework aimed at improving the efficiency of inference in large language models (LLMs). RSD synergistically combines a lightweight draft model with a more powerful target model, incorporating a controlled bias to prioritize high-reward outputs, in contrast to existing speculative decoding methods that enforce strict unbiasedness.
RSD employs a process reward model to evaluate intermediate decoding steps and dynamically decide whether to invoke the target model, optimizing the trade-off between computational cost and output quality. We theoretically demonstrate that a threshold-based mixture strategy achieves an optimal balance between resource utilization and performance. Extensive evaluations on challenging reasoning benchmarks, including Olympiad-level tasks, show that RSD delivers significant efficiency gains against decoding with the target model only (up to 4.4X fewer FLOPs), while achieving significant better accuracy than parallel decoding method on average (up to +3.5). These results highlight RSD as a robust and cost-effective approach for deploying LLMs in resource-intensive scenarios.

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Baohao Liao
Yuhui Xu 2 Hanze Dong
Junnan Li Christof Monz'
Silvio Savarese
Doyen Sahoo? Caiming Xiong?
Abstract
capabilities. When data and model size are scaled traordinary levels, performance can reach unprecedented heights. Large models demonstrate remarkable capabilities across diverse tasks, showcasing robust generalization and advanced reasoning skills (Brown et al., 2020; Hurst et al., 2024; Anthropic, 2024; Team et al., 2024).
We introduce Reward-Guided Speculative Decod- ing (RSD), a novel framework aimed at improv- ing the efficiency of inference in large language models (L..Ms). RSD synergistically combines lightweight draft model with more powerful target model, incorporating controlled bias to prioritize high-reward outputs, in contrast to ex- isting speculative decoding methods that enforce strict unbiasedness. RSD employs a process re- ward model to evaluate intermediate decoding steps and dynamically decide whether to invoke the target model, optimizing the trade-off between computational cost and output quality. We theo- retically demonstrate that fhreshold-based mix- ture strategy achieves an optimal balance between resource utilization and performance. Extensive evaluations on challenging reasoning benchmarks, including Olympiad-lewe tasks, show that RSD delivers significant efficiency gains against decod- ing with the target model only (up to 4.4x fewer FLOPs), while achieving significant better accu- racy than parallel decoding method on average (up to +3.5). These results highlight RSD as a robust and cost-effective approach for deploying LLMs in resource-intensive scenarios. The code is avail- able at https:/github..com waann//BohaooLaolooo iao/RRDD
These advancements result in high computational and eco- nomic costs. While training is resource-intensive, inference at scale is even costlier, requiring vast computational infras- tructure and energy to serve billions of queries (Patterson et al., 2021). The exponential growth in inference costs makes it a key challenge for large model deployment, high- lighting the need for efficient techniques (Frantar al., 2022; Lin al., 2024; Xu al., 2023; Sun et al., 2023; Zhang et al., 2023b; Li et al., 2024a; Xu et al., 2024; Liao & Monz, 2024) to reduce energy use and ensure scalability.
Specifically, sequential token generation in large LLMs in- curs significantly higher computational costs compared to smaller models. This increased latency can hinder their deployment in real-time or high-throughput applications. To address this issue, parallel decoding techniques, such speculative decoding, have emerged effective solu- tions (Leviathan et al., 2023). Speculative decoding operates by leveraging a smaller, lightweight model to generate can- didate outputs, which are then validated and refined by the larger model. This approach significantly reduces the num- ber of decoding tokens required by the larger model, thereby accelerating the overall process. The smaller model serves guide, proposing sequences that the larger model can confirm or adjust, leading to faster inference without com- promising quality. Furthermore, speculative decoding en- sures efficiency by maintaining high-quality outputs through careful calibration of the smaller model. By aligning the smaller model's 's predictions with the larger model's capa- bilities, this method minimizes discrepancies and enhances reliability during inference.
1. Introduction
Scaling laws are widely recognized by the machine learning community as a foundational principle for the development of large language models (Hestness al., 2017; Kaplan et al., 2020; Hoffmann et al., 2022). They emphasize that in- creasing both model size and dataset scale leads to improved loss reduction and, consequently, enhanced generalization
Despite advancements in parallel decoding, speculative de- coding remains underutilized for complex reasoning tasks, particularly multi-step generation. A key limitation the strict unbiasedness requirement, which ensures the final token distribution matches the large model's but restricts flexibility in exploring diverse completions (Holtzman et al.,
Equal contribution 'Language Technology Lab, University of Amsterdam "Salesforce Research. Correspondence to: Yuhui Xu <yuhui.uu@ www.auuaesaattt.ecommm....
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Stap 2: Riaauune
Step 2: 25
Kiviboate por
Skep 3: Evailaae pet
Final Resulr: /
Draft
e0066
y=0,7
m-)
Step I: Elvibaase par
Step Z:Enuauure 1 y=0.7
Skep 3: Evalaate per رlln
Final Result:
Target
p-0,6
-
Avoiwaae 2 24p28_ppp
Unmatched Draft Tokens
Unmatched Draft Tokenss
Wrong Draft Tokens
Matched Tokens
Eivabaat t (22
Step Evaaluate ? Aoit
Skep Evoluate po) fll
Final Result: i
SD
Step 1: Evatunte f
Siep 2: Eivibate pat 1 mid r=0.7
Evoluate per
Final Romlt: i
p-m r-2.6
g
Drafl Model
Target Model Relisbio]
RSD
[Small Fast]
Step Evalaate f!?
Reward
RSD Step
Accepted
fuat y
Rejocted
Arcepued
Figure 1. Reward-GGuide Speculative Decoding (RSD). This diagram illustrates how RSD improves upon standard speculative decoding (SD) by incorporating reward-gnn sorddd selection. SD strictly enforees exact token matching between the draft and target model, leading unnecessary computations when mismatched tokens are discarded. In contrast, RSD evaluates draft outputs based on reward signals r and selectively refines them, reducing reliance on exact matching and improving efficiency. The process starts with a small and fast draft model generating preliminary results, followed by larger and more reliable target model verifying and refining predictions. Darker hackground regions indicate higher computational costs, showing how SD wastes resources on rejected tokens, whereas RSD reduces unnecessary steps by accepting useful draft outputs even when they do not exactly match, balancing efficiency and accuracy.
2020). While unbiasedness maintains theoretical fidelity, it often reduces efficiency, especially when the draft model di- verges from the large model. High-quality tokens (e.g., those favored by a process reward) may still be rejected if their probabilities under the large model are too low, leading to wasted computation and negating potential speedups. This dependence inflates overhead and limits speculative decod- ing's benefits, particularly in long-trajectory reasoning tasks like math and coding. Allowing controlled bias, where the final distribution deviates slightly from the large model, can improve performance. If a draft token is correct but does not match the large model's distribution exactly, strict rejection is counterproductivv. Reward-guider acceptance retains valuable partial solutions, reduces unnecessary queries, and can even surpass the large model's performance. Thus, more adaptive approaches are needed balance efficiency and robustness, ensuring broader real-world applicability.
is robust against the distribution shifts issue between the draft and target models while optimizing resource allocation. By dynamically adjusting when to invoke the larger model, RSD significantly reduces unnecessary computations while maintaining or even surpassing the quality of traditional inference approaches. This approach is particularly well- suited for long-horizon reasoning tasks, where balancing computational cost and accuracy is critical.
Contributions. We propose Reward-Guidee Speculative Decoding, a novel approach to accelerate LLM inference, particularly for reasoning tasks. It introduces an adaptive decoding framework that dynamically mixes outputs from draft and target model, guided by a reward function that evaluates output quality at each step. This enables efficient, high-quality reasoning by constructing flexible mixture dis- tribution, PRSD. balancing efficiency and accuracy through reward-based weighting. RSD employs rejection sampling to selectively refine draft outputs, ensuring scalability. Theo- retically, we derive optimal weighting strategies under com- putational constraints, maximizing efficiency without perfor- mance drop. Extensive experiments on GSMSK, MATH500, Olympiad Bench, GPQA, MMLU STEM, and GaoKao- 2023-En show that RSD not only improves the reasoning accuracy up to 3.5 than SD on average, but also significantly reduces the inference computation with up to 4.4x fewer FLOPs, compared to using the target model alone.
In this work, we introduce Neward-Guided Speculative De- coding (RSD), a novel framework that balances efficiency and accuracy by integrating computationally lightweight "draft" evaluations with reward-driven refinements from a more capable "target" model. Unlike traditional specula- tive decoding, which strictly enforces unbiasedness, RSD leverages reward signals to adaptively select high-value draft outputs rather than discarding mismatched tokens out- right. The process begins with the draft model generating candidate steps, which are then evaluated using a reward function. Steps with sufficiently high reward scores are accepted to continue the reasoning trajectory, while lower- scoring steps trigger speculative corrections using the target model. As illustrated in Fig. 1, this adaptive mechanism
2. Reward-Guided Speculative Decoding
Notations. Let all tokens be embedded in Euclidean space. The prompt is represented as x e Ixdd and the response as
2

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
y E IRLxd, The response y can be further decomposed into sequence of steps ",,""""",, which we denote yl.x-
Probability Density
P m P w
For language models, we consider the iterative process of generating a sequence of steps Vi.n given an input r and a model m (small/draft model) or M (large/target model). At each step i, the context 2; is constructed by combining the initial input r with the sequence of previously generated outputs y1-i-1. such that 2, [a, 11:
wP
PRSD
T
Figure 2. Illustration of Reward-Guided Speculative Decoding
When r(yi is large, indicating a highly preferred or high-quality output, w( (24) approaches In this case, (0(yl22,) pm dominates the mixture, and PRSD(UlEe pri- marily reflects the predictions of P This shows high confidence in the smaller model's ability to produce reliable results. Conversely, when [[vl]z]) is small, (rr(s)) 2:)) ap- proaches O. The mixture weight then shifts toward M- which allows the larger model PM to dominate. Since M is assumed to be sufficiently robust and reliable, it compensates for the smaller model by effectively handling low-quality outputs, ensuring overall performance stability.
Using this context, the next output y: is sampled from con- ditional distribution: yi (ye (x) or Vi ~ M (3ilzs), where Pm corresponds to small draft model m,; PM cor- responds to a large target model M.
Our goal is to optimize the expected reward at each step with computation constraints. For each step i, we define reward function r(u]|=) (2) r(ys lz. V1:1-1), which evalu- ates the quality of the generated step y within the sequence Vi:i given prompt x. A higher reward r(uelz;) indicates a greater likelihood that the model output aligns with the desired response given the x and y1:-11-
Fig. 2 depicts the mixture distribution PRSD. which is in- fluenced by the quality of the output yı|z;, The blue curve represents the distribution P Pm. and the green dashed curve represents P M. The orange dotted line shows the weighting function w(y (2)), which adjusts the mixture between these two distributions. In regions where y yi|z; corresponds to high-quality outputs, the weighting function is elevated, placing more weight on P., to prioritize efficiency. For low low-quality outputs, w decreases, shifting the emphasis to- ward PM .M and thereby penalizing low-quality samples. The red curve represents the resulting mixture distribution PRSD. which adapts based on the quality of the output. demon- strating more efficient sampling process for higher-quality outputs and a more penalized approach for low-quality ones. Regarding the efficiency, large proportion of samples are drawn from P.,. m. with a small part sourced from PMM Ad- ditionally, we maximize Ey.-Prso "(yi a., 91:1-1) to ensure that the model performs effectively in reasoning tasks.
We consider the case that the expected reward achieved by the large model M each step satisfies:
Sy [r(us] [2)]] 2 Eyr-P... [r(wa
(1)
i.e., the target model M should outperform or at least match the draft model m in terms of the expected reward at every step. Our analysis is based on the fact that the large model's predictions yield higher quality outputs, leveraging its ca- pacity for complex reasoning and contextual understanding.
We define the distribution PRSD as a dynamic mixture of P.. and PM, where the combination depends on the quality of the conditional output yı|2g Specifically, we have:
(ysD(y:lz:) (2 P (uilz)) (2) + v(nilz) (M(la)))
where w(-) and v(-) are weighting functions that dynami- cally adjust based on the quality of the output wi|z, Unlike Pm. we assume Par is sufficiently robust and reliable (also costs more): therefore, constant. This ensures that PM M always contributes to the mixture and is not rejected outright, as it acts as stable fallback for handling fow-quality outputs. In our approach, w()(2))) is determined by a reward function r, such that:
2.1. RSD Algorithm
set 0(ys[2;) v, where is
The Reward-Guidee Speculative Decoding (RSD) algorithm operates by dynamically mixing the draft model m and target model M at each generation step, with the objective of balancing efficiency and quality. The algorithm leverages the reward function to guide this dynamic mixing, ensuring that necessary higher-quality steps are more likely to be generated by the target model M, while the draft model m is used for cost-effective generation when possible. Below, we describe the key components of the algorithm. At each decoding step i, the algorithm follows these steps:
w(y]2z)) w. (v)lz)) = (((()1z)))
where r(wl2z) measures the quality or preference for the conditional output vi|zz. The function w-)) maps r(y]a,) to value in [0, 11, reflecting the confidence in Pm. For example, =:)il=:) could depend on factors like the accu- racy or relevance of y 124 and (((())lz:) controls how much weight is assigned to P.,. relative to PM. Because (2( P (yile (2:) is an unnormalized distribution, the con- stant v ensures proper normalization of the mixture.
The draft model generates a candidate 9J given the prompt and previous outputs.
Generate Draft Step:
Compute Reward: The reward function r(y: 2;),
3

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Algorithm 1 RSD: Reward-Guided Speculative Decoding
Algorithm 2 Acceptance Criterion A_
Input: Prompt 2, draft model m. target model M. process reward model T; acceptance criterion Au: EOS token 8, max
Input: value r E R. weighting function w : R [0.
Compute weighting function w(r)
length N
if w() = 0 or 1 then
Assign V1:0 4-
Ar w(r)
for i= to N do
else
Generate draft step d. +- no(a, M1:i-1) Compute reward r; <- r (y:: (2, 11---1 if A. (r:) then
Sample u ~ 24 Am = 1(w(r) 2 u)
end if
Output: Ar)
Accept the draft step y, D1 else
Generate a target step yi + (x, v1+-1) end a E y/ then break end end for Output: Response y1:s
Table 1. Variants of Weighting Function (r) and their definitions
Weighting Function
Definition
w(r =p w(r) 1(r 2 w(r) min(1, m0)
Constant p E (0, 1)
i 2 6, else 0
Clipping r within [0.1]
w(r) max ltr)
Sigmoidal transformation
w()) Ite 1te-a(r-1)
Logistic function
where 2, = [x, 91=1-1), evaluates the step's s quality,
y/2. Assume that wr (ylz) = ((((y)2)), the
3. Apply Acceptance Criterion: The reward r; as- sessed using A. (r.). If accepted, ÿs is used; otherwise, the target model M generates new step.
PusD(ylz)= = w, (ul=)) (ule)) (2)
4. Sample from Mixture Distribution: Accepted steps come from P m. rejected ones from Pa dynamically balancing efficiency and accuracy.
where wi (r(y))) z) is the weighting function that adjusts the relative contributions of the draft model P m. and is the normalizing constant V == 1- Wp-
5. Repeat Until Termination: Steps are generated until the BOS token appears or sequence reaches length N.
2.2. Acceptance Criterion A. and Weighting Function
Proposition 2.2. Given the following assumptions:
Computational Efficiency. The primary advantage of the RSD algorithm lies in its ability to combine the strengths of both the draft and target models while minimizing the com- putational cost. By using the draft model m for most of the generation process, the algorithm reduces the computational burden compared to a strategy that always uses the target model M. The dynamic weighting function w(rr ensures that the draft model is used when the generated steps are of sufficient quality, thereby maximizing efficiency.
w(r is non-decreasing in T; [r EP,,, [r(l=)))
it follows that the expected value of rz]l2) under the RSD in- duced distribution satisfies: Ep |r(y))] 2 Ep... z)).
The weighting function w(-) plays crucial role in adjusting the mixture distribution. Several variants of the weighting function are considered. Table 1 provides summary of different choices. Each of these variants provides new trade- off between draft model and target model. Intuitively, a binary function can maximize expected reward under a strict sampling budget constraint, a smooth weighting might in practice handle noisy reward model outputs more gracefully.
Moreover, by employing rejection sampling, the algorithm only resorts to the more expensive target model M when necessary, ensuring that high-reward steps are generated while keeping the overall cost low. This balance between cost and quality is particularly important in large-scale appli- cations, where both efficiency and performance are critical.
Formal Description of the Algorithm. The RSD algorithm is formally described in Algorithm 1, and the acceptance criterion is outlined in Algorithm 2. In the following, we present the theoretical basis for the mixture distribution used in the algorithm. We also provide the final distribution of the proposed algorithm in Proposition 2. That means the expected reward from P PrSD lies within the reward bounds defined by P.P. m (3)2) and PM(2)22
2.3. Optimal Weighting
In the following Proposition, we demonstrate that the opti- mal weighting function for maximizing reward under a con- strained sampling budget is binary step function, which assigns weight of 1 only high-reward outputs.
Proposition 2.3. Given a constrained sampling budget v/ Egg-p... (r(1 S 7, 7 E (0, 1), the optimal sampling
Proposition 2.1. Given the Algorithm I and 2. for each step

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Draft Model
Target Model
43.4%
56.6%
Reward Score
0.3
0.3
All
0.2
0.2
37.3%
62.7%
10
40 0 10
30
40
Correct
Step
Step
Figure 3. Left: A comparison of the reward scores for all questions generated by the draft model and the targel model within the RSD framework. Middle: focused comparison the reward scores for correctly answered questions generated by the draft model and the target model in the RSD framework. Right: The winning rate (in terms of reward) comparison between the draft model and the target model, highlighting the proportion cases where each model outperforms the other. RSD is configured with Qwen2.5-Me -atthhhtttthh-- 1.5B-- the draft model, Aiven.5-MMtth- Maath-BBIInstrue -nstuutttt the target model, and Skywork-ol -ell---p--PPP PRM-7B the PRM.
strategy that maximize the reward is
serving as the alternative distribution. In standard spec- ulative decoding, often uses PM Pm propose alternative generations, which allows unbiased final distri- bution with = 1. Intuitively, if token (or reasoning step) is significantly less likely under the large model than under the draft model (i.e., if Pac is small), it may indi- P... cate a suspicious scenario that the draft model unable to handle effectively. More generally, hybrid approaches can combine both the reward function r and the ratio Psr Pm
r 2 (2), ff(())=) (=),;,
w+(u)2) w--wll)=
where (z(2) is the largest possible threshold that makes the function satisfy the constraint.
Note that when a binary function is used, Algorithm al- most surely degenerates into a deterministic procedure, pro- ducing a definite acceptance rejection outcome. In real implementation, 5, can be a hyper-parameter 5.
for example: w(yl=) = min 1 r(yl=) Par(yl=) P.. Fr by
normalizing these two quantities in differentiable man- ner. These ratio-based hybrid weighting functions may help address situations where (i) the reward model is noisily correlated with true correctness or (ii) the small and large model distributions diverge substantially.
2.4. Discussion
Process Reward for Each Model. As stated in Eq. (1), we expect the target model to have a higher reward. Fig. 3 confirms this on MATH500, showing that for correctly an- swered questions within RSD, PM consistently outperforms Pm in reward (Middle figure).
Hence, the weighting functioon in Reward-Guidee Specula- tive Decoding can be flexibly designed to incorporate pro- cess rewards, likelihood ratios, any combination thereof, as long it meets the non-decreasing requirement in (when appropriate) and remains bounded within [0, 1]. This flexibility allows RSD induced algorithm to be adapted to different practical constraints (e.g., distribution mismatch, reward model availabiliy/aaccraay)) while still reaping the efficiency gains of speculative decoding.
Comparison with the reward of Pa- Since PRSD a mixture of P,PPm and PM. its reward is weighted sum of their expected rewards. Thus, PrSD can exceed P in expected reward when we use aggressive w that only assigns to high-reward regions. In cases where Pm has a higher expected reward than MM. PRSD also achieves a higher reward. We also notice this phenomenon empirically.
3. Empirical Results
General Weighting Function and SD. Notably, reward- based weighting functions w, not the only option. An alternative approach explicitly defines the weighting func- tion in terms of the likelihood ratio:
Models. To assess the performance of RSD, we employ both general-purpose and math-focused LLMs as our target and draft models, specifically 2w5 (Yang et al., 2024a), I.ama-3 (Dubey et al., 2024), and Qwen-2.5-Math (Yang et al., 2024b). Our system utilizes Skywork-ol-Open PRM (ol Team, 2024) the process reward model (PRM), as it was the most advanced open-source PRM available during our experiments (Zheng et al., 2024). The reward score ranges from to 1, the higher the better.
PM P P
w(y 2) min [..a
where a > 0 is a hyperparameter controlling how quickly the method transitions from the draft model to the target model. This formulation is algorithmically similar to spec- ulative decoding, with a as tunable parameter and P
5

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Table Accuracy on reasoning henchmarks. In general, = 0.7 offers good trade-off between accuracy and efficiency for all tasks. 6" is the optimized threshold for each task, as different tasks have different complexity of reasoning. Refer to $B.1 for detailed results.
Target Model
Draft Model
GaoKao
Olympiad Bench
GPQA Diamond
MMLU STEM
Method
PRM
Setting
MATH500
GSM8K
2023 En
Avg.
Math Model, Torger and Druft: Qwen2 www.n.s.MMaar
PRM Skywork al O1-Open- PRM
Single Model
83.2
65.3
Majority Noting
1.5B
maj 16
79.0
88.9
69,9
45.5
27.3
65.9
62.3 (-3.0)
Best-of-N
1.5B
7B
N
82.2
93.3
69.4
44.9
27.3
71.4
64.8 (.0.5)
SD
7B
1.5B
83.4
95.6
67.3
40.6
28.8
72.0
64.6 (-0.7)
RSD
1.58
1.58
0.7
82.6
94.5
68.8
39.6
38.4
71.4
65.9 (+0.6)
RSD
7B
1.58
7B
8 0.7
84.6
95.5
68.3
42.1
33.8
72.3
66.1 (+0.8)
82.6
(+1.0)
RSD
7B
1.5B
7B
84.6
95.8
68.3
43.6
34.1
72.6
66.5 (+1.2)
Single Model
72B
85.6
95.8
73.0
48.4
42.4
85.8
71.8
Majority Voting
1.5B
maj 1
80.2
89.8
71.2
64.0 (-7.8)
Best-of-N
1,5B
7B
N 64
82.4
94.5
68.6
44.3
27.8
72.4
65.0 (-6.8)
Majority Voting
maj
96.5
73.8
69.8 (:2.0)
Best-of-N
7B
7B
N 64
86.2
97.2
71.4
44.4
36.4
75.4
68.5 (-3.3)
72B
84,8
95.8
71.7
47.6
41.4
85.3
71.1 (-0.7)
RSD
72B
1.5R
0.7
86.4
96.4
73.0
49.8
42.9
84.7
72.2 (+0..)
RSD
72B
7B
7B
0.7
88.0
96.7
74.0
49.9
42,9
84.4
72.7 (+0.9)
6"
72.6
RSD
72B
7B
7B
88.0
96.9
74.0
49.9
42.9
85.6
72,9 (+1.1)
General Model, Target and Draft: Qwen2.5 -nsttruct, PRM: Skywork- o1-Open-PRM
Single Model
77.4
92.0
64.9
38.8
28.8
57.4
59.9
Majority Voting
maj@
66.4
82.1
54.8 (-5.1)
Best-of-N
1.5B
TB
N
10
73.4
89.7
60.5
32.7
23.7
69.4
58.2 (-1.7)
7B
1.5B
77.8
91.8
63.1
39.1
26.3
56.2
59.1 (-0.)
RSD
7B
1.5B
1.5B
0.7
73.6
90.8
64.2
39.0
31.3
71.6
(+1.9))))
RSD
7B
1.5B
7B
0.7
75.0
93.3
66.2
39.9
20.2
61.8
59.4 (-0.5)
RSD
1.58
1.58
8"
74.8
92.3
65.2
40.0
31.3
71.7
62.6 (+2.7)
RSD
7B
1.5B
7B
75.8
93.3
66.2
40.9
29.8
66.3
62.1 (+2.2)
General Model, Torget and Draft: Llama-3. 1 I-Instruct, PRM: Skyworks ol -1-Open-PRM
Single Model
8B
49.4
83.9
41.3
14.5
20.2
39.1
41.4
Majority Voting
maj N
38.0
30.8 (-10.6)
of
IB
7B
16
52.6
74.8
45.7
14.4
14.1
31.0
38.8 (-2.6)
47.0
83.4
41.1 (-0.3)
RSD
8B
IB
1,5B
8
0.7
50.0
83,9
41.8
15.7
20.2
37.2
41.5 (+0.1)
RSD
8B
1B
7B
0.7
50.4
85.4
41.8
18.1
19.7
36.2
41.9 (+0.5)
RSD
1,5
50.0
84.1
43.4
18.1
20.2
38.8
42.4
RSD
8B
1B
7B
50.6
85,5
42.6
18.1
19.7
38.7
42,5 (+1.1)
Datasets. We evaluate our method on a diverse set of reasoning tasks, including GSM8K (Cobbe et al., 2021b), MATH500 (Hendrycks et al., 2021), MMLU STEM (Hendrycks al., 2020), OlympiadBench (He al., 2024), BaoKao-2023-HB (Liao et al., 2024), GPQA (Rein et al., 2023), and Minerva Math (Lewkowycz et al., 2022).
est reward. For majotity voting and Best-of-N, we prefer a large number of samplings (more cost than the target model only) to show their converged performance. (3) Speculative decoding (SD): We also include speculative decoding with number of speculative tokens accelerate inference (Leviathan et al., 2023).
method designed to
Baselines. We consider three categories of baselines: (1) Target model only: This baseline uses the target model independently, requiring more cost than RSD. (2) Draft model with or without PRM: This category includes pop- ular test-time scaling methods that achieve the best possi- ble performance using the draft model. Specifically, we consider majority voting, Best-of-N (BoN) (Brown et al., 2024; Cobbe et al., 2021a) that selects the highest-scoring response (last step) among N candidates based on a PRM, beam search (Chen et al., 2024a) that leverages a PRM to choose the optimal decoding path, process Best-of-N that samples N candidate steps and selects the one with the high-
Default Setting, All experiments were conducted on NVIDIA A100 GPUs, using vLLMM (Kwon et al., 2023) as the backend. use temperature
C. and top-p 0.8 for majority voting, (process) Best-of-N and beam search, while setting temperature for the remaining methods. For process Best-of-N, beam search and RSD, we define a generation ended with \n in)nn as a reasoning step, and then apply PRM to rate this step. We employ the binary step function (the second option in Table 1) as the weighting function and set = 0.7. For brevity, RSD (7B/72B/7B) denotes RSD with 7B draft model, 72B target model and a 7B PRM. SD (7B/72B)
and top-p
6

---

Reward-GGided Speculative Decoding for Efficient LLM Reasoning
Table Comparison with search-based methods. Beam Search
88
RSD (78/728/78)
and Process Best-of-N use a 1.5B base model and a 1.5B PRM.
Accuracy on MATH500
87
(7N/220/2.8))
BoN (78/78, N-64)
Method
Setting
MATH
GSM8K
Minerva Math
86-
Target (728)
500
Single Model (I.5B)
73.8
85.0
29.0
85
(1.58/70/78)
SD (78728)
Process Best-of-N
N
75.8
87.8
32.7
Process Best-of-N
16
76.0
87.9
31.2
84
Beam Search
beam size
78.2
88.9
33.5
Target
Beam Search
beam size
78.2
88.4
32.4
83
-S (1.58/78)
RSD (1.58/7B/1.5B)
6=0.7
82.6
94.5
34.6
BoN (3. 50V 20W.8.N.10000 No
(1.58/78/2.50)
82
10
10?
FLOPs / Question (log scale)
dcotes SD with 7B draft model and 72B target model. BoN (1.5B/15B) denotes BoN with a 1.5B base model and a 1.5B PRM. Without explicitly mentioning, models chosen from the Swwen-255-Matte hatth-Instruc family are used.
Figure 4. Flops accuracy MATH500.
outperforms both search-based methods over all three hench- marks. These results highlight a critical insight: for certain complex or "hard" reasoning steps, search- methods struggle find optimal solutions due the combinatorial explosion of potential candidates, leading to suboptimal performance. On the other hand, our approach leverages a larger model's capacity to generate plausible solutions more directly, bypassing the need for exhaustive search. By utiliz- ing the PRM as a feedback mechanism, RSD benefits from step-wise guidance, which helps mitigate the challenges of reasoning in high-complexity tasks. This suggests that, rather than relying on a purely search-based strategy, incor- porating larger models and targeted feedback mechanisms can lead to more efficient and effective reasoning, especially in cases where the search space is vast or the reasoning steps are particularly intricate.
3.1. Reasoning Benchmarks
We evaluate RSD across a diverse set of reasoning bench- marks, as summarized in Table 2, and observe: (1) Test-time scaling methods like majority voting and Best-of-N, which rely on extensive sampling with a draft model, consistently underperform single target model on average. This finding highlights the importance of larger model for reasoning tasks, as its performance cannot be easily matched by a smaller model with increased computation. (2) While SD is theoretically unbiased, guaranteeing accuracy equal to the target model, often underperforms in practice. This discrepancy, as also noted by Chen al. (2023a), arises due to floating-poin errors. Moreover, in cases where draft model outperforms the target model (e.g.. Table B.1 and domain-specializzed draft models), SD's strict unbiasedness leads to worse performance compared to the draft model. Thus, the decision to use SD must account for such scenar- ios. In contrast, RSD mitigates this concern by leveraging a PRM, which evaluates the quality of reasoning steps from the draft model. (3) Among all evaluated methods, RSD consistently outperforms the single target model on average when using an optimized 6. Even with a fixed 8 = 0.7, RSD achieves better results in 7 out of 8 settings. Notably, on the challenging GPQA benchmark, RSD (1 (1.5B/BB/15BB sig- nificantly surpasses the single target model (38.4 vs. 32.8), demonstrating the effectiveness of this efficient approach.
3.3. Computation Analysis
To evaluate the computational efficiency of our method, we compare RSD with speculative decoding and Best-of-N on MATH500. Following (Kang et al., 2024; Sardana et al., 2023). we adopt the standard approximation of FLOPs for transformers with N parameters, i.e. 2N per inference to- ken. Note that the inference cost for PRMs is also included in the calculations. As shown in Fig. 4, RSD (1.5B/BB/7B) outperforms both SD (1.5B/7B) and Target (7B), achieving an accuracy improvement of 1.2 and 1.4, respectively, while using fewer FLOPSs. Moreover, RSD (7B/22B/7B) achieves a notable accuracy of 88.0 on MATH500, compared to 85.6 for Target (72B), with nearly 4.4x fewer FLOPs. When compared to BoN (7B/7B.N=64), RSD (7B/72B/7B) de- livers 1.8 points higher accuracy at a significantly lower computational cost. These results clearly demonstrate both efficiency and effectiveness of RSD.
Additionally, a larger PRM (7B) slightly enhances perfor- mance compared to smaller PRM (1.5B), especially complex datasets like GPQA and MATH500, where the increased reasoning capacity of a larger PRM proves benefi- cial. Results with general models, such as Qwen2.5-Instruet and Llama-3. 1..ama-3.1-Imm Instrrct,, are even improved more, validating RSD's robustness and generalizability.
3.4. Ablation Studies
Threshold d. Fig. 5 illustrates the relationship between the threshold 8, accuracy, and the proportion of questions solved solely by the draft model within RSD. As d increases, accuracy improves, peaking at 5 0.7, before experiencing
3.2. Comparison with Search-Based Methods
We also compare our method with beam search (Chen et al., 2024a) and process Best-of-N in Table RSD significantly
7

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
84
100
86
84.6
Target Model only
Accuracy on MATH500
84
Draft Modill only
Accuracy on MATH500
82-
Questions (%)
82
81.8
81.4
80-
80
79.2
78
77.6
70-
40
76-
76-
20
74
Accuracy
Questions Solved by Draft Model only
0.0
0.2
0.4
0.6
0.8
1.0
Constant
Binary 6-0.7
Clipping Sigmoldal Logiittcc 1.6-0.6
Figure 5. The impact of threshold 8 with RSD (1.5B/7B/7B).
p- 0.6
Figure 6. Accuracy of weighting functions from Table I with RSD (1.5B/7B/BB. All settings share similar inference cost.
slight decline. Notably, the accuracy remains consistently higher than that of using only the target model (6 = 1.0) once surpasses 0.6. A higher 5 corresponds to a stricter rejection of reasoning steps generated by the draft model. However, even at 5 = 0.7, the draft model alone can still solve 48% of questions, as the rewards for its reasoning steps on these questions are sufficiently high. This eliminates the need for the target model to engage with these 48% of questions, distinguishing RSD from SD, which always involves both the draft and target models for every question.
Table 4. Accuracy model merge. denotes that we merge these two models. Refer to $B.3 for detailed setting and number.
Method
Target
Draft
PRM
Setting
Arg. Accuracy
Single Model
7B
65.3
SD
7B
1.58
64.6 (-0.7)
RSD
7B
1.58
1.58
0.7
68.9 (+0.5)
7B
(+0.5)
RSD
7B
1.58"
1.58
0.7
63. (-0.3)
RSD
78"
1.58
7B
= 0.7
66.7 (+1.4)
(see Table 2). Therefore, adding an additional PRM for RSD incurs minimal overhead compared to SD.
This adaptability makes RSD an effective method for au- tomatic compute allocation, allocating less compute (draft model only) to simpler questions and more compute (both draft and target models) more challenging ones. For more detailed discussion of the automatic compute alloca- tion for questions in different levels, refer to $B.2. Addition- ally, 5 serves as a critical role in balancing the computational efficiency of the draft model with the precision of the target model, ultimately optimizing overall performance.
Here, we further investigate the possibility of merging mod- els enhance the usability of RSD by reducing the number of served models. As shown in Table 4, merging models does not necessarily degrade performance and remains supe- rior to SD. Interestingly, merging larger models even results in performance improvements, consistent with observations reported by Yadav et al. (2024).
Weighting Function. In Table we present several can- didate weighting functions. Here evaluate their perfor- mance under comparable inference costs, as shown in Fig. 6. First, all candidates outperform the draft model alone, un- derscoring the critical role of incorporating a larger model within the reasoning loop. Second, the constant weighting function, which does not utilize rewards, performs the worst, emphasizing the significance of PRM feedback. Lastly, the binary step function achieves the best performance, even sur- passing the single target model. Additionally, it introduces 5 as hyperparameter, allowing for flexible control over inference costs to accommodate varying budget constraints. Thus, we use the binary step function in this paper, and leave the exploration of other rejection schemes to future.
Robustness to PRMs. To investigate RSD's robustness to the choice of PRM, include two more strong PRMs, i.e. Owen2.5-Math - PRM-7B and 72B (Zhang et al., 2025) in Table 5. RSD equipped with different PRMs consistently outperforms SD, with more gains from larger PRM.
Gemeral-Domain Task. An important component of RSD is PRM. To the best of our knowledge, there is not yet a PRM for domain generation. However, there many outcome reward models (ORMs) for open-ended generation. Could we use ORM instead of PRM in RSD?
Here, we utilize Llama-3.22 3.2- 1-- (Grattafiori et al., 2024) as the draft model, Llama-3. 1-8.3..1-8---n-tuee as the tar- get model, and Skywork-Reward-LLinn Reward-Llama-3. 1-8--00.2 (Liu et al., 2024) as the ORM. The 805 prompts from AlpacaE- val (Dubois et al., 2023) are used for the generation. And the model outputs
4. Discussion
evaluated with AlpacaEval2.0 (Li et al., 2023) against the outputs from gpt4 turbo. Similar to the setting for PRM, we define a generation ended with \n), as a reasoning step, and apply the ORM to score this step. The score of Seywork-Rewann NeeaaddLlamm--3 Llama-3.1 3.1.-BBBBBBBBB. 18.1-8B-v. v0.2 ranges from oo to co. We didn't 't extensively tune the reward threshold,
PRM Overheads and Model Merge. In MATH500, the average number of reasoning steps per question is 18, sug- gesting that the PRM is invoked 18 times per question, akin to generating 18 tokens. Furthermore, even a tiny PRM (1.5B) outperforms the single target model in RSD accuracy
8

---

Reward-GGided Speculative Decoding for Efficient LLM Reasoning
Table Accuracy on reasoning benchmarks with different PRMs. RSD is robust to choice PRM. 6 0.7 works well universally.
Method
PRM
Setting
MATH500
GSM8K
GaoKao
Olympiad Bench
GPQA Diamond
MMLU STEM
Avg.
2023
Draft: Queen25 Mwth---aaat----imee----- Target:
5 5-Mahh Instruct-BB
Single Targot Model
83.2
95.7
66.8
41.2
32.8
71.8
65.3
SD>
83.4
95.6
67.3
40.6
28.8
72.0
64.6 (-0.7)
RSD
Skywork-a al Cal--Ope.PRMM. PRM-1..B
0.7
82.6
94.5
68.8
71.4
65.9 (40.6)
RSD
Skywork-ol--O -ol----p--PR---7-
8 0.7
84.6
95.5
68.3
42.1
33.8
72.3
66.1 (+0.8)
RSD
Swee2.5-Math-PRM---1
0.7
83.2
95.5
67.3
41.3
35.3
72.5
65.9 (+0.6)
RSD
Owen2.5-Math-PPR. -277B
0.7
84.0
95.7
68.1
42.0
33.3
75.6
66.5 (+1.2)
ing (Elhoushi et al., 2024; Zhang et al., 2023a) leverages parts of the large language model (LLM) parameters as the draft model while using the original base model as the verifier. Parallel decoding (Stern et al., 2018; Cai et al., 2024) further enhances efficiency by introducing draft mod- els to streamline the process. Unlike previous speculative decoding methods, our approach utilizes process rewards to perform stepwise speculative reasoning.
Table Results general-domain benchmark. AlpacaEval. The PRM used here is an outcome reward model, Skywork-Rewer -lama-3.1-8B-v.22.
Method
Win Rate (%)
Draft Model only (Llama-3.2-IB B.ttaa...2----mmme I-Instuuctt Target Model only (Llama-3.1-88) -88-ttit)) RSD
7.09
24.47
18.85
Reward Models on Reasoning. Reward models play a crucial role in selecting correct reasoning trajectories dur- ing training (Chen et al., 2024a; Wang et al., 2024; Zhou et al., 2025) and inference (Brown et al., 2024). Outcome Reward Models (ORMs) (Yu al., 2023; Dong et al., 2024) are trained exclusively on the model's final output, whereas process reward models (PRMs) (Lightman et al., 2023) rely step-level annotations, providing dense and granular reward signals at each reasoning step. Scaling test-time compute (Snell et al., 2024; OpenAI, 2024) has gained sig- nificant traction with the advancement of reward models. Techniques like Best-of-N (Brown et al., 2024; Cobbe et al., 2021a; Dong et al., 2023) leverage ORMs to select the sam- ple with the highest reward from N candidates. Building on this, tree search methods have been introduced, enabling per-step predictions rather than relying solely on the final answer (Chen et al., 2024a; Qi et al., 2024; Yao et al., 2024). These methods are enhanced by process feedback, such as process reward models (PRMs). We propose a novel appli- cation of PRMs to accelerate reasoning during inference.
and empirically chose 8 = 0.
As shown in the In Table 6, even with an ORM instead of a PRM, RSD achieves a significantly better win rate than the draft model, showing RSD's robustness across different tasks. Among all generated tokens, 65% tokens are gener- ated by the draft model only without any intervention of the target model. We believe that general-domain PRM and dedicated tuning of 5 could further boost the performance.
Combine RSD with SD. RSD is not inherently opposed to SD; in fact, they can be seamlessly combined to en- hance efficiency. For instance, during a rejected step, SD (draft+target) can be utilized regenerate the step. This ap- proach allows for further optimization of RSD's efficiency without incurring additional costs.
Specialized PRM. In our experiments, we rely on an open- source general PRM. However, training or fine-tuning a specialized PRM that is closely aligned with the draft model could further enhance performance. Such a PRM would bet- ter recognize high-quality reasoning steps generated by the draft model, thereby increasing the acceptance rate. Future work could explore specialized PRM training or fine-tuning.
6. Conclusion.
We propose Reward-Guiiee Speculative Decoding (RSD), a novel framework that enhances L.MM inference efficiency, particularly for reasoning-intensive tasks. RSD dynamically combines lightweight draft model with more capable tar- get model, using reward function guide output selection at each step. This approach balances computational cost and quality by selectively refining outputs based on process rewards. RSD achieves significant efficiency gains over SD and BoN while maintaining accuracy benchmarks. Ex- tensive evaluations across reasoning tasks highlight RSD's robustness, adaptability, and effectiveness, making a prac- tical solution for LLM deployment.
5. Related Work
Speculative Decoding. Speculative decoding (Stern et al.,
2018; Leviathan et al., 2023; Xia et al., 2024; Chen al., 2023a; Zhang et al., 2023a: Sun et al., 2024a; Chen al., 2023b; Li et al., 2024b) achieves lossless acceleration by employing a draft model to predict subsequent tokens and verify them in parallel. Tree-based speculation (Miao et al., 2024; Fu et al., 2024; Sun et al., 2024b; Chen et al., 2024b) extends this approach by generating multiple candidates to increase the acceptance rate. Self-speculative decod-

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Acknowledgements
Chen, G., Liao, M., Li, C., and Fan, K. Alphamath almost zero: process supervision without process. arXiv preprint arXiv:2405.03553 2024a.
This research was partly supported by the Netherlands Or- ganization for Scientific Research (NWO) under project number VI.C.192.080.
Chen, Z., Yang, X., Lin, J., Sun, C., Chang, K. C.-C,, and Huang, J. Cascade speculative drafting for even faster Ilm inference. arXiv preprint MXXx:2312.11400 2023b.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning by proposing Neward-Guider Specula- tive Decoding (RSD), a framework aimed at improving the efficiency and scalability of large language model inference. The potential societal implications of this work are primar- and
Chen, Z., May, A., Svirschevski, R., Huang. Y., Ryabinin, M., Jia, Z., and Chen, B. Sequoia: Scalable, robust, and hardware-aware speculative decoding. arXiv preprint arXi::2402.12377 2024b.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint vxXiv:2100.4100 2021a.
ily positive, RSD facilitates more energy-efficinn cost-effective use of computational resources, contributing to the sustainability of deploying large-scale Al systems,
However, as with any advancement in machine learning, there are ethical considerations to be acknowledged. The increased efficiency of language models could lead to wider accessibility and adoption, which, while beneficial in many respects, may also exacerbate risks such misuse for gen- erating misinformation or biases in outputs. We strongly encourage researchers and practitioners to apply RSD re- sponsibly and consider incorporating safeguards to mitigate these risks.
Cobbe, K.,, Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 16168, 2021b.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang. T. Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023.
Overall, we believe the contributions of this work align with the broader goal of advancing AI technologies in ethical and sustainable manner, with no immediate negative societal consequences requiring further discussion.
Dong, H., Xiong, W., Pang. B., Wang. H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863. 2024.
References
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint ar arXiv:2407.2178. 2024.
Anthropic, A. Claude 3.5 sonnet model card addendum.
Claude-3.5 Model Card, 3, 2024.
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21788 2024.
Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I,, Ba, J., Guestrin, C., Liang, P., and Hashimoto, B. Alpaca- farm: A simulation framework for methods that learn from human feedback, 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners.
Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L.. Mahmoud, A., Acun, B., Agarwal, S., Roman, A., et al. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16711 2024.
Advances in neural information processing systems, 33: 1877-1901, 2020.
Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint mrXiv:2401.107// 2024.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXi::2210.17323 2022.
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013//8 2023a.
Fu, Y., Bailis, P., Stoica, I., and Zhang, H. Break the se- quential dependency of 1lm inference using lookahead decoding. arXiv preprint arXiv:2402.0203: 2024.
10

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Goddard, C., Siriwardhana, S., Ehghaghi, M., Meyers, L.,, Karpukhin, V., Benedict, B., McQuade, M., and Solawetz, J. Arcee's mergekit: A toolkit for merging large language models. In Proceedings of the 2024 Conference on Empir- ical Methods in Natural Language Processing: EMNLP 2024 Industry Track, Miami, Florida, USA, November 12-16, 2024,00000 pp. 477-485, Association for Computational Linguistics, 2024.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H.,, and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 61 611-626, 2023.
Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In Inter- national Conference Machine Learning, pp. 19274- 19286. PMLR, 2023.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dalle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint ar arXivv2407.21178, 2024.
Lewkowyzz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V. V., Slone, A., Anil, C., Schlag, I., Gutman-Sole T., Wu, Y., Neyshabur, B., Gur- Ari, G., and Misra, V. Solving quantitative reasoning problems with language models. In Koyejo, S., Mo- hamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans,
He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L.. Shen, J., Hu, J., Han, X., Huang. Y., Zhang, Y., et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad- level bilingual multimodal scientific problems. arXiv preprint ar Ar0v:22402. 14008, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M.. Song, D., and Steinhardt, J. Mcasuring mas- sive multitask language understanding. arXiv preprint wxXvv2000.033300 2020.
LA, USA, November 28- December 9, 2022, 2022.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacac- val: An automatic evaluator of instruction-followinng models. https: / /ggthub...... www.tatsuttttataaaaeaaaa a al 1apca_eval,5 20033.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv 2021.
preprint arXiv:2103.0338
Hestness, J., Narang. S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint 171::01712.00000 2017.
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: Lim knows what you are looking for before generation. arXiv preprint arXiv:2404.144: 14469, 2024a.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E.,, Cai, T., Rutherford, E., Casas, D. d. L.,, Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2003115556 2022.
Li, Y. Wei, F. Zhang, C., and Zhang, H. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077, 2024b.
Liao, B. and Monz, C. 3-in-1: 2d rotary adaptation for efficient finetuning, efficient batching and composability. CoRR, abs/2409.00119, 19, 2024. doi: 10.48550/ARXIV. 2409.00119. Liao, M., Luo, W., Li, C., Wu, J., and Fan, K. Mario: Math reasoning with code interpreter output-a reproducible pipeline. arXiv preprint arXiv:2401.00 00//99000 2024.
Holtzman, A.. Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference on Learning Representations (ICLR), 2020.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint AXXi:2335.200550 2023.
Kang, J., Li, X. Z., Chen, X., Kazemi, A., Sun, Q., Chen, B., Li, D., He, X., He, Q. Wen, F,, et al. Mindstar: Enhancing math reasoning in pre-trained Ilms inference time. arXiv preprint arXiv:2405. 16265, 2024.
Lin, J., Tang, J., Tang, H., Yang. S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-awan weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint wxXiv:200100866 2020.

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Liu, C. Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y. Skywork-reward: Bag of tricks for reward modeling in Ilms. arXiv preprint arXiv:2410.1844 2024.
Sun, Z., Suresh, A. T., Ro, J. H., Beirami, A., Jain, H., and Yu, F. Spectr: transport. Advances in Neural Information Processing Systems, 36, 2024b.
Fast speculative decoding via optimal
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., et al. Specinfer: Accelerating large language model serv- ing with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. pp. 932 022-949,220 2024.
Tang, Z., Zhang, X., Wang, B., and Wei, F. Mathscale: Scaling instruction tuning for mathematical reasoning, In Forty-first International Conference on Machine Learn- ing, 2024, Vienna, Austria, July 21-27, 2024. Open- Review.net, 2024.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z. Wang. S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXh::4433005330 2024.
ol Team, S. Skywork-ol open series. https:/ // huggi wgface co/ Skywork, November 2024. URL https: https/////ggggggggeeegeoo co/ wwt../oorkkkk.....
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural In- formation Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 9998-6008, 2017.
OpenAL. URL
Learning reason with llms, 2024.
https://opends com/ index/ learning-to-rease mmmm
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.- M., Rothchild, D., So, D., Texier, M., and Dean, Car- bon emissions and large neural network training. arXiv preprint ar 100i 10350, 2021.
Wang. P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y.. Chen, D.,
Wu, Y., and Sui, Z. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426 9439, 2024.
Qi, Z., Ma, M., Xu, J., Zhang, L. L., Yang, F,, and Yang. M. Mutual reasoning makes smaller lims stronger problem- solvers. arXiv preprint arXiv:2408.08 06195, 2024.
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022 2023.
Xia, H., Yang, Z., Dong, Q. Wang. P., Li, Y., Ge, T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.0785.. 2024.
Sardana, N., Portes, J., Doubov, S., and Frankle, J. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint orXiv:2401.00444 2023.
Xu, Y., Xie, I.., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., and Tian, Q. Qa-lora: Quantization- aware low-rank adaptation of large language models. arXiv preprint arXiv:2309. 14717, 2023.
Snell, C., Lee, Xu, K., and Kumar, A. Scaling I1m test- time compute optimally can be more effective than seal- ing model parameters. arXiv preprint arXiv:2408.03314 2024.
Xu, Y., Jie, Z., Dong, H., Wang, L., Lu, X., Zhou, A., Saha, A., Xiong, C., and Sahoo, D. Think: Thinner key cache by query-driven pruning. arXiv preprint arXiv:2407.21018. 2024.
Stern, M., Shazeer, N., and Uszkoreit, J. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018.
Yadav, P., Vu, T., Lai, J., Chronopoulou, A., Faruqui, M.,
Bansal, M., and Munkhdalai, T. What matters for model merging at scale? CORR, abs/2410.03611 2024.
Sun, H., Chen, Z., Yang, X., Tian, Y., and Chen, B. Tri- force: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404.1 1912, 2024a.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li,
C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint 1511v:241..15115, 2024a.
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math techni- cal report: Toward mathematical expert model via self- improvement. arXiv preprint arXiv:2409.122 1212,, 2024b.
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint orXiv:2306.1 /695, 2023.
12

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Treee of thoughts: Deliherate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yu, F,, Gao, A., and Wang, B. Outcome-supervised verifiers for planning in mathematical reasoning. arXiv preprint vrXiv:2311.09724, 2023.
Zhang. J., Wang, J., Li, H., Shou, I.., Chen, K., Chen, G., and Mehrotra, Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint ar arXiv:2309.081 0866,, 2023a.
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L.,, Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710, 2023b.
Zhang, Z., Zheng, C.,, Wu, Y., Zhang. B., Lin, R., Yu, B., Liu, D., Zhou, and Lin, The lessons developing process reward models in mathematical reasoning. arXiv preprint ar wrxiv:2501.0730) 2025.
Zheng, C., Zhang, Z., Zhang, B., Lin, R., Lu, K., Yu, B., Liu, D., Zhou, J., and Lin, J. Processbench: Identify- ing process errors in mathematical reasoning. CoRR, abs/2412.06559,, 2024.
doi: 10448550ARXIV.2412.
06559,
Zhou, P., Wang, K., Chang, J., Gao, Z., Kallus, N., Wein- berger, K. Q., Brantley, K., and Sun, W. Q#t: Provably op- timal distributional rl for IIm post-training. arXiv preprint arXiv:2502.20548 2025.
13

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
A. Proof
A.1. Proof Proposition 2.1
Proof. For simplicity, we assume that y = 2/22.
The final distribution PRSD (?) combines contributions from two sampling paths:
Accepted samples from m.-
Fallback samples from PM after rejection.
According to Law of Total Probability,
PRSD (y) P(accept y from Pm.) + P(eject Pm and draw ÿ from PM).
Term
Term 2
For accepted samples from P the acceptance probability for ÿ Pm is ((r(r))):
Term = P (v) w(r()))
For samples from P.. the rejection probability from Pm v. where:
Ea (((())))
After rejection, is drawn from Past
Term 2 = v. P.
By combining two terms
PRSD(ū) (r) + PP
where v + Eg-Pi, (w(r(()) =
A.2. Proof of Proposition 2.2
Proof. We aim to prove that under the conditions:
1. (r) is non-decreasing in
2. Ep [r(v[=)] 2 Ep... (r(y]=)) =)],
the expectation of r(y]=) under PrSD satisfies:
EPaSD [r(v]=)] 2 Ep [r(]=)]]
By definition:
E [r(v==]] = f (u(r( + +vP r()l))) r(vl=)dy
where // = Ep... [rr()[=))]] Substituting
Epash [x([2)] = Ep. (r((( (1- Ep [w((())))) [r(v]=)]
Term
Term
The condition Ep Epggo [r[)[=)] Ep... [r(y]=)] 2 can be rewritten as
Ep. [w(rr) + Ep... (w)r) Ep. 2
14

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Note that:
Ep (r)rr) Covp,, Ep [w(r)] Ep,,. [r]-
Substitute this into the inequality:
Covp (w(r), + Ep,,. (prr: Ep,,
Ep... [r] 2 0.
Simplify the terms involving Ep ... [r]:
Co (w(r), + (1,Ep,,,,<rr) (Ep,, Ep,.. [r]) 2 0.
1. Covariance Term (Covp,, (w(r), r)): Since (r) non-decreasing in r, higher values of correspond to higher values of (r). This implies Covp (w(r),r) 0.
2. Expectation Difference Term (11 Ep.. (r)) (Ep (r) Ep... [r])):
- Ep,,,, [w]]] = v 2 (since v a normalizing constant for a valid probability distribution). By the second condition, Ep., [r] Ep... [r] 2 0.
Thus, this term is non-negative
Both terms in the inequality:
Covp,, (w(r), 0 and (1 Ep,., (w(r)) (Ep [r] Ep,.. [r)) 2 0
are non-negative under the given conditions. Therefore:
EPASD [r(y]=)] 2 Ep Ep.,.(yyla) [r(y]=)].
The conditions w() is non-decreasing in and Ep,, [r[v]=)] 2 Ep... [r[y]2)] are sufficient guarantee Epasn [r(y|=)] Ep.. [v]]=)]
A.3. Proof of Proposition 2.3
Proof. For simplicity, assume that ÿ y/2. Our optimization problem is
((w,) Pg. (ū)r(ū) + Es vr(ū)
subject to the inequality constraint:
V == Ei (w) (f)) S 7
0 S w()) I 1.
Equivalently,
((w,)= Ep.,. (+) (L Ep (w)l Ep [r()]
The Lagrangian's is given by
L(wir, A) A)= = / (mm (PM - (()l) dē A (1 7) / (1mm))/pmmm0)ussll (o))s
The Lagrangian derivative yields:
dc Ow,(ū)
P (r(()) - (A+ R)),
where R = Eg-p,,, rū)).
Setting this to zero gives the threshold rule: the optimal W,"" (ū) is:
if r (A + R)
(
0, if (A+ R) < 0.
KKT Conditions.
'eere, integral can also be defined for a counting measure space, allowing to he applicd to the discrete case.
15

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Primal feasibility: Eg-P... (w)
Dual feasibility:: > 0.
Stationarity: The first-order condition holds.
Complementary slackness: A ((1 -7) - P (ū)(() 0.
Conclusion.
If Eg-P (2 = 7, then À 2 0 and the solution is tight, as in the equality case.
If Ej-P... [ (7) 7, then = and the constraint satisfied with slack.
The solution is threshold-baaee function on r(ÿ). where (Ø) for larger r(ū) and (.) for smaller r(ū).
adjusted ensure that the constraint Eg-P... (w)
7 is satisfied.
Thus, The optimal sampling strategy involves "using Pm " (i.e., wi (i)) 1) for the higher values rlū) and "using Pas" (i.e., w, (ÿ) 0) for the lower values of r(t). The threshold on rlğ) is selected such that the constraint Eg.p... [w, 2 ? met.
Threshold-basee based strategy: (T) (r(0)st)> where / is chosen to satisfy the constraint Eg (+)(1) 2 -
B. Additional Empirical Results
Table I. Accuracy on CN Middle School 24 (Yang al.. 2024b) and College Math (Tang et al., 2024). Due to SD's strict unbiasedness, achieves worse accuracy draft model if draft model outperforms the target model, while RSD doesn't have this issue.
Method
Target
Draft
PRM
Setting
CN Middle School 24
College Math
Single Model
1.5B
75.2
48.0
Single Model
7B
72.3
46.8
SD
1.5B
73.3
46.9
RSD
7B
1.5B
7B
8 8= 0.7
78.2
48.2
B.1. Tuning of
RSD employs threshold, to decide whether accept a reasoning step generated by the draft model. the reward score of reasoning step exceeds 6, it is accepted. However, reasoning tasks vary in complexity, leading diverse reward distributions. Using fixed 5 may not yield optimal accuracy across different tasks.
The results for varying 8 values are presented in Table B.2. Overall, 8 = 0.7 emerges as reliable choice across various settings. Slight adjustments within the range [0.6, 0.7, 0.8, 0.9] can further improve performance.
B.2. Different Reasoning Complexity
Thanks to the human annotated complexity levels in MATH500 (5 levels, the higher the harder), here we investigate how RSD works for questions different complexity. As shown Fig. B.1, the involvement of the target model (6 & 0) consistently improves the accuracy compared with the draft model only (5 = 0). The improvement varies for different levels, most +4.7 for level 1, +5.6 for level 2, +6.7 for level 3, +16.4 for level 4 and +15.7 for level 5, showing the importance of the target model for harder questions.
For the same 6, one also observe that the proportion of questions solved by the draft model alone decreases with an increasing level. For example at 5 = 0.7, draft model alone solves 84% questions in level 1, 67% questions in level 2, 58% questions in level 3, 44% questions in level 4 and 19% questions in level 5. It shows that harder questions need more involvement of the target model. In this way, RSD can be considered as method for automatic compute allocation, less compute for easy questions and more compute for hard questions, which is different from SD that always needs both target and draft models for every question.
16

---

Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Table B.2. Accuracy with different Overall, 0.7 works well different models and tasks. However, since complexity different tasks varies, slight tuning of ó offers better accuracy.
Target Model
Draft Model
GaoKao 2023 En
Olympiad Bench
GPQA Diamond
MMLU STEM
Method
PRM
Setting
MATH500
GSMBK
Avg.
Math Model, Target and Druft:
M-een2.--att--aae PRM: Skywork Open-PRM
RSD
1.58
L.B
0.6
80.4
67.8
40.6
32.3
69.1
63.9
RSD
7B
1.58
1.58
5 0.7
82.6
94.5
68.8
39.6
38.4
71.4
65.9
RSD
7B
1.5B
1.5B
8 0.9
81.2
95.5
68.3
39.4
32.3
71.6
64.7
RSD
7B
1.5B
7B
8 0.6
83.6
95.4
68.3
43.6
30.3
71.6
65.5
RSD
7B
1.5B
7B
0.7
84.6
95.5
68.3
42.1
33.8
72.3
66.1
RSD
7B
1.58
0.8
83.6
95.8
67.8
40.9
34.3
72.6
65.8
RSD
7B
1.5B
7B
0.9
83.4
95.7
68.1
40.1
32.8
72.5
65.4
RSD
72B
L.5B
L.5B
8
0.6
83.0
93.5
71.9
48.4
36.9
78.1
68.6
RSD
72B
1.5B
1.5
0.7
94.7
72.7
47.7
40.4
82.5
70.3
RSD
72B
1.5B
1.51
0.8
86.6
95.6
71.4
48.0
40.9
85.1
71.3
RSD
72BB
1.5B
1.5B
0.9
85.4
95.6
72.2
47.7
44.4
83.5
71.8
RSD
72B
1.5B
7B
0.6
85.8
96.0
72.5
49.0
41.9
82.5
71.3
RSD
72B
1.5B
7B
0.8
87.4
96.3
73.0
48.3
40.9
85.4
71.9
RSD
72B
0.9
86.2
96.0
72.7
48.9
41.4
85.6
71.8
RSD
72BB
7B
L.5B
0.6
85.0
97.0
72.2
48.4
43.9
82.2
71.5
RSD
728
L55
73.0
49.8
42.9
84.7
72.2
RSD
72B
7B
1.58
8 =
0.8
86.6
96.6
73.2
48.7
43.9
85.2
72.4
RSD
72B
7B
7B
0.6
88.0
96.6
72.5
49.6
40.4
82.5
71.6
RSD
72B
7B
7B
0.7
88.0
96.7
74.0
49.9
42.9
84.4
72.7
RSD
72B
7B
7B
0.8
87.4
96.9
73.5
48.3
41.9
85.6
72.3
RSD
72B
7B
0.9
73.5
48.3
42.4
85.4
72.0
Model, Target Droft: Qumm2.5-Instrruc
Skywork 1-Oppn.PPRM
RSD
7B
1.5B
1.5B
0.6
72.8
89.7
63.6
38.5
22.2
71.7
59.8
RSD
7B
1.5B
1.5B
0.?
73.6
90.8
64.2
39.0
31.3
71.6
61.8
RSD
7B
1.5B
1.5B
8 == 0.8
74.8
91.6
64.4
38.8
23.7
67.1
60.1
RSD
40.0
28.3
59.3
0.6
37.3
29.8
66.3
RSD
7B
1.5B
7B
8 0.7
75.0
93.3
66.2
39.9
20.2
61.8
59.4
RSD
7B
1.5B
7B
0.8
74.2
92.4
62.9
40.6
21.7
$8.2
58.3
RSD
7B
1.5B
7B
0.9
75.8
92.1
65.2
40.9
26.8
56.1
59.5
Generul Model, Torget Drafi:
Skywork Open- PRM
RSD
8B
1.5
0.6
49.0
40.8
18.1
19.2
34.5
40.7
RSD
1.5B
0.?
50.0
83.9
41.8
15.7
20.2
37.2
41.5
RSD
8B
1B
L.5B
8 == 0.8
48.6
84.1
41.8
16,3
18.7
38,8
41.4
15.3
41.5
16.9
RSD
8B
IB
7B
8 0.7
50.4
85.4
41.8
18.1
19.7
36.2
41.9
RSD
8B
1B
7B
0.8
84.2
41.3
16.4
18.2
37.9
41.4
RSD
8B
7B
0.9
50.0
83.5
42.3
16.1
18.2
38.7
41.5
Table B.3. Accuracy of model merge. denotes that we merge these two models.
Method
Target Model
PRM
Setting
MATH500
GSM8K
GaoKao 2023 En
Olympiad Bench
GPQA Diamond
MMLAU STEM
Avg.
Model
Math Model, Target Druft: Qwen2.5 Math Instruct, PRM: Skywork Open PRM
Single Model
66.8
71.8
SD
7B
1.5B
83.4
95.6
67.3
40.6
28.8
72.0
64.6
RSD
7B
1.5B
1,5B
8
0.7
82.6
94.5
68.8
39.6
38.4
71.4
65.9 (-0.6)
RSD
7B
1.5B
7B
0.7
84.6
95.5
68.1
42.1
33.8
72.3
66.1 (+0.8)
RSD
7B
1.5R*
1.5
83.4
95.1
67.3
39.3
33.3
71.6
(-.0
RSD
7B"
1.5B
78"
0.7
84.0
95.6
69.4
43.0
35.4
72.6
66.7 (+1.4)
B.3. Model Merge
To reduce the number of models required for facilitating RSD's usage, we consider merging either the target model with the PRM or the draft model with the PRM. Here, we focus on the simplest merging strategy-lines merging-ust
17

---

Reward-Guidel Speculative Decoding for Efficient LLM Reasoning
level-1
level-2
level-3
95
96
80
80
80
94
94
94-
60
60
33
93
Accuracy on MATH500
92
40
92-
20
20
20
81
90
0.0
0.2
0.4
0.6
0.8
1.0
level=4
level= 5
85
100
100
80
60
60
Accuracy Questions by Model only
75
56
20 54
70
50
0
5
Figure B.I. The bchaviour of RSD (1.5B/7B/7B) for different 8s and questions in different complexity levels (the higher the level, the harder question.).
and denotes all questions are solved by the draft model alone and the target model only, respectively, Overall, the involvement of the target model improves the accuracy. The improvement is more obvious for harder question, +16 for level 4 addition, with an increasing level, questions solved by draft model only decrease questions need more involvement of the target model.
same 6, demonstrating harder
MergeKit (Goddard et al., 2024), leaving the exploration of more advanced merging methods for future work.
The PRM and the policy model have different architectures. Specifically, the PRM includes projection layer atop the final transformer layer (Vaswani et al., 2017), which projects the hidden dimension to scalar output, whereas the policy model employs an 1m.head. We merge only the shared layers, retaining the PRM's projection layer and the policy model's 1m.head. For interpolation weights in the linear merging process, we tested only [0.6, 0.4] and [0.5, 0.5], with the target or draft model receiving 0.6 and the PRM 0.4. The [0.6, 0.4] configuration performed slightly better.
As shown in Table B.3, the results indicate the following: (1) Overall, the merged model outperforms SD; (2) Merging improves performance more substantially in larger models (+1.4 +0.8). This observation aligns with the findings of Yadav et al. (2024).
Draft Model
Target Model
42.4%
57.6%
Reward Score
0.6
0.6
0.4
0.4
39.8%
60.2%
30
20 Step
Correct
Step
Figure B.2. Left: A comparison of the reward scores for all questions generated by the draft model and the target model within the RSD framework. Middle: focused comparison the reward scores for correctly answered questions generated by the draft model and the target model in the RSD framework. Right: The winning rate comparison between the draft model and the target model, highlighting the proportion cases where each model outperforms the other the RSD framework. RSD is configured with Qwen2.5-Man 5-Math- 1,5B Instruct the draft model, Qwen2.5 Math-72B Instruct as the target model, and Skywork-ol-Oppe PRM-7B the PRM.
18


Reviews:
[{"review_id": "74MA6jaclI", "reviewer": "Reviewer_p3vx", "summary": "This paper introduces a method to guide speculative decoding using a reward model. Unlike standard speculative decoding, where a larger model verifies the outputs of a smaller model, the proposed approach determines acceptance or rejection based on reward signals. Specifically, the authors design a series of weighting functions to normalize the reward as a probability that stochastically decides whether to accept or reject the generated responses. Experimental results demonstrate that this method maintains efficiency while improving reasoning performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.** PRM vs large models’ likelihood\n\n> The core issue is the misalignment between the large model (LM) and PRM, leading SD to reject high-quality tokens due to low LM likelihood. While the reviewer suggests this implies high FP rates from LM, these are often style-/format-related, not correctness errors. LMs generalize better but are NOT trained for correctness. In contrast, PRMs are explicitly trained to identify correctness, making them more suitable for detecting errors, while LM likelihood would point out both style inconsistency and errors. Thus, our algorithm uses LM guidance only after PRM rejection, leveraging both PRM's specialized correctness identification and LM’s strength in generalizing better completions. Using both for rejection is also promising; we leave it for future work as noted in Sec 2.4.\n\n$~$\n\n**Q2.**  Calculation of FLOPs.\n\n> As described in Sec 3.3, we follow the standard FLOPs approximation for transformer models with N parameters, i.e., approximately 2N FLOPs per inference token, as adopted in prior works [1,2]. We provide a detailed example below:\n> The FLOPs calculation for RSD (7B/72B/7B) is presented in the table below.\n> \n||**Target**|**Draft**|**PRM**|\n|--------------|--------------|--------------|--------------|\n|Model Size| 72B | 7B|7B|\n|Tokens/Question| 67 | 396|18|\n|TFLOPs|9.648|5.544|0.252|\n>\n> [1] Scaling laws for neural language models.\n>\n> [2] Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.\n\n$~$\n\n**Q3.**  More efficiency metrics (throughput).\n\n> We measure the througput and provide the following table. We use batch size 256 and MATH500. \n> The observation is similar to Figure 4: RSD is faster than SD.\n\n | Method | Throughput |\n | --- | --- |\n | Single Draft (1.5B) | 1.00 $\\times$ (4697 tokens/s) |\n | Single Target (7B) | 0.70 $\\times$ | \n | SD(1.5B/7B) | 0.83 $\\times$ |\n | RSD(1.5B/7B/1.5B) | 0.91 $\\times$ |\n | RSD(1.5B/7B/7B) | 0.85 $\\times$ |\n\n$~$\n\n**Q4.** Search-based baselines\n\n> We compare RSD to the search-based baselines (Process Best-of-N and Beam Search) in Table 3, where RSD significantly outperforms the search-based baselines that only utilize a draft model and PRM. It shows the importance of involving a larger model in the reasoning path for correction. \n\n$~$\n\n**Q5.** Why RSD outperforms the large model?\n\n>**Proposition (Improved Expected Reward via RSD):**\nAssume that for each decoding step the reward function $r(y|z)\\in[0,1]$ and consider $\\omega(r) = \\mathbf{1}(r \\geq \\delta)$ for some threshold $\\delta \\in [0,1]$. Define the RSD mixture distribution as $ \\mathbf{P} _ {\\text{RSD}}(y| z) = \\omega(r(y| z)) \\mathbf{P} _ m(y | z) + \\nu\\mathbf{P} _ M(y| z), $\nwhere $\\nu = 1 - \\mathbf{P} _ {y\\sim \\mathbf{P} _ m}\\{r(y| z) \\geq \\delta\\}$.\nThe expected reward of the RSD distribution is \n$$\\mathbf{E} _ {\\mathbf{P} _ {\\text{RSD}}}[r(y|z)] = \\alpha\\,\\mathbb{E} _ {\\mathbf{P}_m}[r(y| z)| r(y| z) \\geq \\delta] + (1-\\alpha)\\,\\mathbb{E} _ {\\mathbf{P}_M}[r(y| z)],$$\nwith $\\alpha = \\mathbf{P} _ {y\\sim \\mathbf{P}_m}\\{r(y| z) \\geq \\delta\\}.$\n\n>The RSD distribution outperforms the target model $\\mathbf{P} _ M$ (i.e., $\\mathbf{E} _ {\\mathbf{P} _ {\\text{RSD}}}[r(y| z)] > \\mathbf{E} _ {\\mathbf{P} _ M}[r(y| z)]$) if and only if\n$$\\mathbf{E} _ {\\mathbf{P}_m}[ r(y| z) | r(y| z) \\geq \\delta] > \\mathbf{E} _ {P_M}[r(y| z)].$$\n\n>Thus, RSD yields a higher expected reward than $\\mathbf{P}_M$ provided that the subset of tokens generated by the draft model $\\mathbf{P}_m$ (i.e. those with $r(y| z) \\geq \\delta$) has a higher reward than $\\mathbf{P}_M$. If one can choose $\\delta$ so that only high-quality tokens from the draft model are accepted, then the mixture of these with the fallback tokens from $\\mathbf{P}_M$ lifts the overall expected reward of RSD above that of the large model alone.\n\n>This result offers a theoretical explanation for why RSD can outperform the large model: it leverages the fact that, when the draft model’s top-performing outputs (as measured by $r$) are even better on average than the target model’s outputs, selectively accepting these outputs increases the overall quality.\n\n>In summary, for RSD to be advantageous over $\\mathbf{P}_M$, one must choose a threshold $\\delta$ such that:\n$$\\mathbf{E} _ {\\mathbf{P}_m}[r(y| z)| r(y| z) \\geq \\delta] > \\mathbf{E} _ {\\mathbf{P}_M}[r(y| z)].$$\n\n>Under this condition, RSD leads to an improved expected reward compared to $\\mathbf{P}_M$.\n\n$~$\n\n**Q6.** Estimation of δ\n\n>**A6.**  As noted in Prop. 2.3, $\\delta$ corresponds to a quantile of the reward distribution under a given compute budget. While theoretical guidance informs its range, estimating the exact quantile requires access to empirical reward statistics -- hence, practical tuning (e.g., grid search) is necessary and suggested by our analysis. Moreover, Table 2 demonstrates that performance is not highly sensitive to $\\delta$, suggesting our method is robust across a reasonable range and not reliant on fine-tuning.", "claims_and_evidence": "Most of the claims in this paper are clear and well-supported. However, the statement that “High-quality tokens (e.g., those favored by a process reward) may still be rejected if their probabilities under the large model are too low” raises some concerns. This claim suggests a potential misalignment between the large model and the reward model. However, the proposed algorithm still trusts the large model’s output to correct responses rejected by the reward model (as shown in Eq. 1). This implies that while the large model has high precision, it may also exhibit a high false positive rate. Further justification is needed on this to explain why the reward model serves as a more reliable verifier than the large model’s likelihood or a combination of both.", "methods_and_evaluation_criteria": "The proposed methods are evaluated on 6 math reasoning benchmarks, which align with the major claim and the choice of reward models. The experiments also include a comparison on FLOPs.", "theoretical_claims": "The theoretical results are partially checked (Props 2.1,2.2,2.3). It remains unclear what the “largest possible threshold that makes the function satisfy the constraint” refers to until I reach the appendix.", "experimental_designs_or_analyses": "The experimental designs are overall sound. More details on how FLOPs are recommended to be included. It is also suggested to include more efficiency metrics, such as throughput (tokens per secs).", "relation_to_broader_scientific_literature": "The proposed method has relevance to recent topics scaling inference-time compute to facilitate LLM reasoning, in which an LLM samples the solution according to a reward to do reasoning.", "essential_references_not_discussed": "No critical references known to me are missing.", "other_strengths_and_weaknesses": "Strength:\n+ The overall writing quality is good, with clear mathematical formulations that effectively convey intuition to the reader. I found the results and discussions in Appendix C.2 particularly insightful, as they illustrate how RSD dynamically allocates computational resources based on task complexity.\n+ The proposed approach is simple yet effective, offering a novel way to leverage reward models not only for enhancing reasoning accuracy but also for improving efficiency.\n+ The experimental results are compelling, demonstrating that reward-guided decoding can even surpass the performance of the strongest large model.\n\n\nWeakness:\n- The theoretical results do not fully explain why RSD outperforms the large model.\n- The motivation for why the reward model serves as a better acceptance criterion than the larger model is not entirely convincing. Since the algorithm ultimately relies on the larger model for corrections, it raises the question of why the reward model improves the verification process over large models’ likelihood. See “Claims And Evidence” for more details.\n- Additionally, while speculative decoding in RSD is guided by process rewards, the baselines rely on outcome rewards. To ensure a fair comparison, it would be beneficial to include additional search-based baselines that also incorporate process rewards.", "other_comments_or_suggestions": "Ln 30-31, “significant better” -> “significantly better”.\nLn 325-326, “majotity” -> “majority”", "questions_for_authors": "The ablation results indicate that the threshold $\\delta$ plays a crucial role in reasoning accuracy but is highly sensitive across different tasks. Given its definition in Proposition 2.3, can $\\delta$ be estimated beforehand based on theoretical results, rather than relying solely on empirical tuning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dF5Xn4kPI8", "reviewer": "Reviewer_GXeJ", "summary": "This paper introduces Reward-Guided Speculative Decoding (RSD), a novel framework designed to improve the efficiency of inference in large language models (LLMs) by combining a lightweight draft model with a more powerful target model. Extensive evaluations on challenging reasoning benchmarks demonstrate that RSD significantly reduces computational costs (up to 4.4× fewer FLOPs) while improving accuracy compared to using the target model alone or parallel decoding methods. The results highlight RSD as a robust and cost-effective approach for deploying LLMs in resource-intensive scenarios, particularly for complex reasoning tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.** The reliance on a process reward model, which may introduce additional overhead.\n\n> This is indeed a very practical concern, since RSD utilizes one more model, i.e. the process reward model (PRM), than speculative decoding (SD). However, accoding to our experiments, the overhead is minor from three perspectives:\n> * **The PRM is small:** According to Table 2, a 1.5B PRM could already offers consistent better performance than all baselines with different draft and target models.\n> * **The inference cost of PRM is minimal:** RSD applies PRM to score the step instead of tokens from the draft model. The average number of steps per question in MATH500 is 18. The inference cost of PRM is similar to generate 18 tokens per question.\n> * **The PRM can be merged with the target or draft model:** In Table 4, we explore the possibility of merging the PRM with the draft or target model. Surprisingly, model merging doesn't obviously degrade the performance, and even results in better performance when merging the larger models. In this way, RSD shares the same number of models as SD. A further investigation of model merging between PRM and proxy model is left to future work.\n\n$~$\n\n**Q2.** How does RSD perform on non-reasoning tasks like text generation or summarization?\n\n> This is a very interesting suggestion. An important component of RSD is PRM. As far as we know, there is not yet a PRM for general-domain generation. But there are plenty of outcome reward models (ORMs) for open-ended generation. **Could we use ORM instead of PRM in RSD?**\n\n> **Experimental setup:** \nWe utilize Llama-3.2-1B-Instruct as the draft model, Llama-3.1-8B-Instruct as the target model, and Skywork-Reward-Llama-3.1-8B-v0.2 as the ORM. The 805 prompts from AlpacaEval [1] are used for the generation, and the model outputs are evaluated with AlpacaEval2.0 against the outputs from gpt4_turbo. Similar to the setting in the paper, we define a generation ended with \"\\n\\n\" as a reasoning step, and apply the ORM to score this step. The score of Skywork-Reward-Llama-3.1-8B-v0.2 ranges from -$\\infty$ to $\\infty$. We didn't extensively tune the reward threshold $\\delta$, and empirically chose $\\delta=0$.\n\n> **Results:**\nAs shown in the following table, even with an ORM instead of a PRM, RSD achieves a significantly better win rate than the draft model, showing RSD's robustness across different tasks. Among all generated tokens, 65% tokens are generated by the draft model only without any intervention of the target model. We believe that a general-domain PRM and dedicated tuning of $\\delta$ could further boost the performance.\n\n| Method | Win Rate (%) against gpt4_turbo |\n| --- | --- |\n| Single Draft Model | 7.09 |\n| Single Target Model | 24.47 |\n| RSD | 18.85 |\n\n$~$\n\n**Q3.** Could the framework be extended to handle multimodal inputs?\n \n > Yes, we believe our RSD could be seamlessly extended to the multomodal reasoning tasks, because:\n > * The key component of RSD is PRM. As shown in Q2, even an ORM could act as a PRM. We believe that the existing multimodal ORM can be used in RSD for multimodal reasoning tasks.\n > * We noticed that a new multimodal PRM [2] is released recently. It can be used perfectly in our framework.\n>\n> However, the exploration of multimodal reasoning tasks is out of the scope of this work, and there isn't any multimodal PRM when we conducted this research, we leave this investigation to interested readers.\n \n$~$\n\n > [1] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. Yann Dubois, Balázs Galambosi, Percy Liang, Tatsunori B. Hashimoto\n >\n >[2] VisualPRM: An Effective Process Reward Model for Multimodal Reasoning. Weiyun Wang, Zhangwei Gao, et. al., Wenhai Wang\n\n$~$\n\n---\n\nThank you very much for your thoughtful suggestions, making our work more solid. We have incorporated the new results in the updated version by our side.\n\nIf these revisions address your concerns, we kindly request a reconsideration of the scores. Should you have any further questions, we are happy to assist.", "claims_and_evidence": "The paper claims that Reward-Guided Speculative Decoding (RSD) improves LLM inference efficiency by combining a draft model with a target model, guided by a process reward model. Evidence includes up to 4.4× fewer FLOPs and improved accuracy on reasoning benchmarks like MATH500 and GSM8K. The results show RSD outperforms traditional methods and models.", "methods_and_evaluation_criteria": "RSD uses a draft model to generate candidate outputs, which are evaluated by a reward model to decide whether to invoke the target model. Evaluation is based on computational efficiency (FLOPs) and accuracy across reasoning tasks. Benchmarks include GSM8K, MATH500, and Olympiad-level tasks.", "theoretical_claims": "The paper claims that a threshold-based mixture strategy optimally balances computational cost and output quality. Theoretical analysis shows that RSD ensures higher expected rewards compared to using the draft model alone.", "experimental_designs_or_analyses": "Experiments compare RSD to baseline methods like speculative decoding and target-only models across multiple reasoning benchmarks. Metrics include FLOPs, accuracy, and the proportion of questions solved by the draft model alone. Ablation studies explore the impact of threshold values and weighting functions.", "relation_to_broader_scientific_literature": "RSD builds on speculative decoding and process reward models, addressing limitations in traditional methods. It aligns with research on efficient LLM inference and reasoning tasks.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "A weakness is the reliance on a process reward model, which may introduce additional overhead.", "other_comments_or_suggestions": "How does RSD perform on non-reasoning tasks like text generation or summarization?", "questions_for_authors": "Could the framework be extended to handle multimodal inputs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lldqzLt789", "reviewer": "Reviewer_XUtc", "summary": "The paper introduces Reward-Guided Speculative Decoding (RSD), an improved version of SD which replies on a process reward model to determine the quality of a step instead of exact match. RSD combines a lightweight draft model with a more capable target model, integrating a reward-based mechanism to optimize computational cost and output quality. RSD allows controlled bias by accepting high-reward outputs from the draft model, even when they do not perfectly match the target model’s predictions. The authors provide theoretical justification for a threshold-based mixture strategy, ensuring an optimal balance between computational efficiency and performance. Empirical results on various reasoning benchmarks, including GSM8K, MATH500, OlympiadBench, GPQA, MMLU STEM, and GaoKao-2023-En, demonstrate that RSD achieves up to 4.4× fewer FLOPs while improving accuracy by up to +3.5 points compared to standard speculative decoding.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$~$\n\n**Supplementary Material:** https://anonymous.4open.science/r/Rebuttal-supp-6595-C4A2/appendix_rebuttal_ICML.pdf\n\n$~$\n\n---\n\n**Q1.** Why RSD outperforms the large model?\n\n> Please refer to our rebuttal to **Reviewer p3vx Q5**.\n\n$~$\n\n**Q2.** Alternative reward models or potential biases\n\n> As we mentioned, one might consider alternative formulations:\n>- **Likelihood-Ratio-Based:** One could incorporate a term like $\\frac{\\mathbf{P}_M(y|z)}{\\mathbf{P}_m(y|z)}$ to define the weighting function (e.g., $w(y|z)=\\min(1, \\alpha\\,\\frac{\\mathbf{P}_M(y|z)}{\\mathbf{P}_m(y|z)})$). Such a formulation may capture discrepancies between the two models’ distributions, but it may also introduce bias if the likelihood estimates are themselves noisy or miscalibrated.\n>- **Hybrid:** One might combine the process reward and likelihood ratio—e.g., $w(y|z)=\\min(1, \\beta\\,r(y|z)\\,\\frac{\\mathbf{P}_M(y|z)}{\\mathbf{P}_m(y|z)})$. While this hybrid approach can potentially leverage complementary strengths, its sensitivity to misestimations in either component may induce systematic biases. For example, if the reward function overestimates quality in regions where the draft model already performs well, the weighting may overfavor the draft model, reducing the benefits of the target model’s corrections.\n\n>In both cases, under our assumption, the underlying distribution will be better than small model induced one. Whereas, the choice of reward function can significantly affect the acceptance probability, and thus the final mixture distribution. When we choose more weights towards likelihood, it will biased to exact-match of large model distribution, so that the cost would be higher (similar to SD).\n\n>You may also consider imperfect reward (Q5).\n\n$~$\n\n**Q3.** Performance by question difficulty.\n\n> We show it in Tab C.4 (in link). One can observe:\n> * For simpler questions (level=1), RSD and SD performs the same;\n> * For harder questions (level>2), RSD consistently outperforms SD.\n> With the help of PRM, RSD selectively includes the target model for some reasoning steps, correcting the wrong reasoning step and leading to better performance.  \n\n$~$\n\n**Q4.** Different reward models & variations in reasoning complexity.\n\n> In Tab C.5 (in link), we includes more PRMs, Qwen2.5-Math-PRM-7B and Qwen2.5-Math-PRM-72B, released during the reviewing cycle. \n> * RSD is robust to PRM. Although Skywork-o1-Open-PRM (regression model) and Qwen2.5-Math-PRM (classification model with num_labels=2) are trained differently, they all perform quite well across different tasks.\n> * $\\delta=0.7$ consistently yields strong results across all four PRMs, appearing to be a sweet spot for balancing efficiency and performance when the reward score lies in [0, 1].\n\n> We also draw the accuracy and efficiency wrt $\\delta$ in Fig C.3 (in link). These two PRMs behave similarly:\n> * Involving the target model ($\\delta \\neq 0$) consistently improves accuracy over using the draft model alone ($\\delta = 0$), with greater gains on harder questions.\n> * For the same $\\delta$, the proportion of questions solved by the draft model alone decreases with an increasing level, showing that harder questions need more involvement of the target model.\n\n$~$\n\n**Q5.** Imperfect reward models\n\n> In practice, $\\hat{r}(y|z)$ is an approximation of the oracle reward $r_{\\text{oracle}}(y|z)$. We can derive non-asymptotic convergence bounds under assumptions on the reward model’s error distribution.\n\n> *Assumption (Sub-Gaussian Error):* Assume that for each reasoning step, the estimation error\n$$\n\\epsilon(y|z) = \\hat{r}(y|z) - r_{\\text{oracle}}(y|z)\n$$\nis sub-Gaussian with parameter $\\sigma^2$; that is, for any \\(t > 0\\),\n$$\n\\Pr\\left(|\\epsilon(y|z)| \\geq t\\right) \\leq 2\\exp\\left(-\\frac{t^2}{2\\sigma^2}\\right).\n$$\n> Under this assumption, when using $n$ independent observations, standard concentration inequalities imply that, with probability at least $1-\\delta$,\n$$\n\\left|\\frac{1}{n}\\sum_{i=1}^n \\hat{r}(y_i|z) - \\mathbf{E}[r_{\\text{oracle}}(y|z)]\\right| \\leq \\sigma \\sqrt{\\frac{2\\log(2/\\delta)}{n}}.\n$$\n\n> Even if the reward model is imperfect, the empirical average reward converges to the oracle at $O\\Bigl(\\sqrt{\\frac{\\log(1/\\delta)}{n}}\\Bigr)$. \nThe bias introduced in the acceptance decision and in  $\\mathbf{P}_{\\text{RSD}}$ can be controlled in a non-asymptotic manner. Even with imperfect reward estimates, the impact on the RSD  diminishes as more data become available.\n\n$~$\n\n**Q6.** Open-ended generation tasks.\n\n> Refer to Q2 of Reviewer GXeJ.\n\n$~$\n\n**Q7.** Choice of $\\delta$ based on the difficulty.\n\n> From Fig C.1, $\\delta=0.7$ performs consistently well across difficulty levels, achieving top accuracy in 4/5 levels. Q4 further confirms $\\delta=0.7$ works well for different PRMs. \n>\n> That said, dynamically selecting $\\delta$ based on question difficulty (eg using a LLM to predict difficulty and adjust $\\delta$ accordingly) could improve performance and efficiency. We leave this for future work.", "claims_and_evidence": "Most claims in the paper are well-supported with both theoretical analysis and empirical evidence:\n1. The claim that RSD reduces computational cost while maintaining or improving accuracy is supported through detailed FLOPs analysis and comparisons across multiple benchmarks.\n2. The claim that RSD outperforms traditional speculative decoding (SD) on reasoning tasks is substantiated with controlled experiments using various LLM sizes and evaluation datasets.\n\nSome aspects could benefit from additional justification:\n1. The assertion that RSD can “even surpass the large model’s performance” requires more scrutiny. While some results suggest that RSD achieves higher accuracy than the single target model, this may depend on careful hyperparameter tuning (e.g., choice of threshold δ), and the exact conditions under which this occurs need further clarification.\n2. The theoretical analysis of reward-guided acceptance could be extended by considering alternative reward models or potential biases introduced by specific reward function choices.\n\nOverall, the claims are convincing, but further ablation studies on reward function variations and distribution mismatch effects would strengthen the argument.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are well-aligned with the problem of efficient inference for LLM reasoning:\n\n1. Evaluation on a broad set of reasoning benchmarks (GSM8K, MATH500, OlympiadBench, etc.) is appropriate because speculative decoding has traditionally been underutilized for multi-step reasoning tasks.\n2. Comparisons with baseline methods (single target model, speculative decoding, Best-of-N, beam search) ensure a fair assessment of RSD's performance.\n3. Computational efficiency is rigorously analyzed using FLOPs per question, which is a relevant metric for assessing inference cost.\n\nHowever, a more detailed breakdown of performance by question difficulty (e.g., comparing RSD with SD for simple vs. complex problems) could provide deeper insights into when RSD is most beneficial.\n\nThe chosen evaluation methods are sound, but the paper could further explore RSD’s robustness to different reward models and variations in reasoning complexity.", "theoretical_claims": "The proofs in the paper are logically consistent, they assume an idealized reward function without considering potential noise in reward estimation. A discussion on the sensitivity of these theoretical results to imperfect reward models would be useful.", "experimental_designs_or_analyses": "I reviewed the experimental setup, and it appears to be methodologically sound:\n\n1. Controlled comparisons with speculative decoding and single-model baselines ensure a fair evaluation.\n2. Hyperparameter tuning for δ is systematically explored, showing that a threshold around δ = 0.7 balances efficiency and accuracy.\n3. Computational cost is measured in FLOPs, providing an objective comparison.\n\nLimits:\n1. One possible limitation is that the experiments primarily focus on math and reasoning tasks. It would be interesting to see how RSD performs on open-ended generation tasks (e.g., summarization, dialogue).\n2. The choice of process reward model (PRM) is not extensively analyzed—would a different PRM significantly change RSD’s effectiveness?\n\nOverall, the experimental setup is robust, but future work could explore how different PRMs influence performance and whether RSD generalizes beyond reasoning tasks.", "relation_to_broader_scientific_literature": "The paper builds on prior work in speculative decoding and efficient inference for LLMs, making several novel contributions:\n\n1. Improves upon speculative decoding (SD): Prior SD methods (Leviathan et al., 2023) enforce strict unbiasedness, whereas RSD introduces reward-guided acceptance to enhance efficiency.\n2. Incorporates a process reward model (PRM): Similar to reinforcement learning approaches (Dong et al., 2023), but applied to speculative decoding for stepwise validation of reasoning.\n3. Demonstrates efficiency gains over standard decoding strategies: The results align with recent efforts to reduce inference cost via speculative sampling (Chen et al., 2023) and model acceleration (Frantar et al., 2022).", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Weakness:\nThe performance of the PRM is quite important for the process, yet this is can also be the limit of performance. Although the result shows it can go beyond the target model, but still limited.", "other_comments_or_suggestions": "The authors discussed about the choice of \\delta and some experiments to explore the pick of it. I am wondering if it is possible to make the selection of the parameter as dynamic based on the difficulty of the question.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces process reward models (PRMs) into the speculative decoding framework. Instead of requiring the large model to strictly verify the small model’s output, a reward model is used to judge whether the small model’s output is acceptable. This allows token-level differences, reducing the intervention frequency from the large model. As a result, inference becomes more efficient without sacrificing reasoning quality. The method achieves up to 4.4× speedup compared to using the large model alone, and improves accuracy by up to 3.5 points over parallel decoding with the small model on reasoning tasks. 

All reviewers are in favor of acceptance. The paper presents well-designed experiments and introduces insightful ideas. I recommend acceptance.

One minor suggestion, raised by reviewers p3vx and XUtc, concerns the criterion for selecting the reward threshold $\delta$. The paper would benefit from a clearer explanation of how $\delta$ is chosen, along with an analysis of its impact across broader domains to support practical applications.
Decision: Accept (poster)