MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE
Authors: Zongle Huang
Lei Zhu
ZongYuan Zhan
Ting Hu
Weikai Mao
Xianzhi Yu
Yongpan Liu
Tianyu Zhang
Venue: NeurIPS 2025 spotlight ()
Keywords: Speculative decoding
MoE
Inference
Acceleration
Primary area: deep_learning

Abstract: Large Language Models (LLMs) have achieved remarkable success across many applications, with Mixture of Experts (MoE) models demonstrating great potential. Compared to traditional dense models, MoEs achieve better performance with less computation. Speculative decoding (SD) is a widely used technique to accelerate LLM inference without accuracy loss, but it has been considered efficient only for dense models. In this work, we first demonstrate that, under medium batch sizes, MoE surprisingly benefits more from SD than dense models. Furthermore, as MoE becomes sparser -- the prevailing trend in MoE designs -- the batch size range where SD acceleration is expected to be effective becomes broader. To quantitatively understand tradeoffs involved in SD, we develop a reliable modeling based on theoretical analyses. While current SD research primarily focuses on improving acceptance rates of algorithms, changes in workload and model architecture can still lead to degraded SD acceleration even with high acceptance rates. To address this limitation, we introduce a new metric 'target efficiency' that characterizes these effects, thus helping researchers identify system bottlenecks and understand SD acceleration more comprehensively. For scenarios like private serving, this work unveils a new perspective to speed up MoE inference, where existing solutions struggle. Experiments on different GPUs show up to 2.29x speedup for Qwen2-57B-A14B at medium batch sizes and validate our theoretical predictions.

MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE
Zongle Huang 1.3
Lei Zhu?
Zongyuan Zhan
Ting Hu²
Weikai Mao?
Xianzhi Yu?
Yongpan Liu 133
Tianyu Zhang?*
Tsinghua University Huawei Noah's Ark Lab 3BNRist
(huangz123)0maill tsinghua. edu. en (ypliu)@tsinghua. edu. en
{zhulei168, zhanzongyuan, huting35, maoweikai, moweekaai,uuiaanzal zhangtianyus9) com
Abstract
Large Language Models (LLMs) have achieved remarkable success across many applications, with Mixture of Experts (MoE) models demonstrating great potential. Compared to traditional dense models, MoEs achieve better performance with less computation. Speculative decoding (SD) is widely used technique to accelerate LLM inference without accuracy loss, it has been considered efficient only for dense models. In this work, first demonstrate that, under medium batch sizes, MoE surprisingly benefits more from SD than dense models. Furthermore, as MoE becomes sparser - the prevailing trend in MoE designs - the batch size range where SD acceleration is expected to be effective becomes broader. To quantitatively understand tradeoffs involved in SD, we develop reliable modeling based theoretical analyses. While current SD research primarily focuses on improving acceptance rates of algorithms, changes in workload and model architecture can still lead to degraded SD acceleration even with high acceptance rates. To address this limitation, introduce new metric larget efficiency that characterizes these effects, thus helping researchers identify system bottlenecks and understand SD acceleration more comprehensively. For scenarios like private serving, this work unveils a new perspective to speed MoE inference, where existing solutions struggle. Experiments on different GPUs show up to 2.29x speedup for Qwen2- 57B-A14B A14B at medium batch sizes and validate our theoretical predictions.
Introduction
Recent years have witnessed remarkable success in Large Language Models (LLMs), with Mixture of Experts (MoE) architectures showing tremendous potential. Unlike dense models use single feed-forward network (FFN) to process all inputs, MoE models replace the FFN with multiple specialized "expert" networks plus router that selectively activates only a few experts for each input token. Such sparsity in structure enables MoEs with more parameters to achieve higher computational efficiency, and multiple state-of-theeart LLMs, such Deepseek V3 [[] and Qwen2.5-Max [2), all MoEs. MoE model architectures are evolving toward larger scales with increased sparsity S
and more balanced workload distribution among experts [5] 6..
Speculative decoding (SD) is a lossless technique to accelerate LLM inference, but conventional wisdom suggests that its efficacy diminishes when applied to MoEs. In SD, smaller draft model is introduced to rapidly generate multiple candidate tokens, while the larger target model verifies these predictions in parallel, preserving only correctly speculated tokens. For dense models' inference, the time taken to generate a single token and verify multiple ones is roughly the same, as both tasks require the full set of parameters to be loaded once. Therefore, SD gains acceleration through fewer forward
Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

rounds of the target model and shorter decoding time of the draft model. However, this acceleration has been demonstrated to diminish in MoEs B as the multiple draft tokens verification activate more experts than single token, leading to larger memory access and significantly longer verification time compared to standard decoding step.
In this work, we challenge the conventional belief' and demonstrate that, under moderate batch size, SD can be more effective for MoEs than for dense models. Our key insight is that when the batch size is moderate such that all experts are already activated in a single decoding step, verifying multiple draft tokens will not incur additional expert parameter loading costs. Furthermore, as the MoE becomes sparser, each expert processes fewer tokens per parameter loading, leading to lower utilization of arithmetic units and thereby creating greater acceleration opportunities for SD.
The insight above is supported by comprehensive theoretical analyses, through which we identify a new metric target efficiency to quantify how systemic factors (such workload and target model architecture) affect SD speedup. In contrast to existing SD works that use acceptance rate [ 10 an algorithmic metric to evaluate how accurately the draft model speculates the target model, our proposed target efficiency isolates extrinsic factors like algorithm selection and focuses on intrinsic system bottlenecks caused by the target model's computational and memory access requirements. As demonstrated in the following sections, even with similar acceptance rates, systemic factors can greatly impact SD effectiveness, making our metric target efficiency necessary for comprehensive understanding of SD acceleration.
As further step, we build quantitative modeling of SID speedup for MoE based these theoretical analyses. The consistent matching between our modeling and experiment results confirms the reliability of our analyses. Additionally, the modeling itself provides approach for analyzing the execution time of different components, making the end-to-end SD acceleration results more transparent and explainable.
Our work offers new perspective for lossless MoE acceleration, particularly well-suited for private [3 [4]. Private serving has gained popularity among enterprises seeking to safeguard data and model security, with typical applications such as in-house chatbots. These environments typically process moderate batches containing tens of requests. Additionally, findings can be applied to latency-critical scenarios where large batch sizes are infeasible, or memory- restricted environments where MoEs exceed GPU capacity.
serving scenarios [12
In Summary, the main contributions of our work are:
We refine the conventional belief that speculative decoding cannot effectively accelerate MoEs, demonstrating that under moderate batch sizes, SD is actually more effective in wider range of batch sizes for sparser MoEs than dense models.
Based on theoretical analysis, we developed a reliable modeling for SD speedup, thus making the acceleration process transparent and explainable. Existing metrics only assess algorithmic optimization efficiency and cannot fully explain SD speedup, so we introduce new systemic metric target efficiency that reveals speedup opportunities inherent in the target model.
Our findings can be applied to accelerate scenarios like private serving. Experiments on various model demonstrate that SD achieves the highest
GPUs with the Qwen2-57B. speedup at the moderate batch size, reaching 2.29x. These experiments also validate our theoretical prediction that SD is more favorable for sparser MoËs.
2 Related Work
MoE acceleration. MoE has emerged as a promising LLM architecture, and many techniques optimize its inference. Model compression methods, including pruning [6], quantization [8], distillation [19 20). and decomposition [21 22), have been applied to MoEs and achieved great acceleration. They sacrifice model quality for speedup, as in dense models. When MoEs are too large to fit in GPU memories and offloading becomes necessity, several system-level approaches have emerged to optimize inference latency through improved scheduling techniques. Expert prefetch- ing [23 24 predicts and pre-loads experts for upcoming layers based on previous activation patterns, thus overlapping expert loading with current layer computation. Expert caching (25] 26 caches most frequently activated experts in GPU memory, leveraging expert locality to reduce expensive

---

offloading. Compared to them, our work unveils a new perspective for MoE acceleration that is lossless and doesn't 't depend on expert imbalance.
Speculative Decoding. Speculative decoding (SD), initially proposed by [0] and (9]. has emerged as widely adopted technique for accelerating LLM inference without sacrificing generation quality, Basic SD employs a smaller model to rapidly generate draft tokens, which are then verified in parallel by the target model that needs to be accelerated. Afterwards, more algorithms are developed to lift the
acceptance rate of draft tokens. [27 T1| 28 [29 7 30 31 adopt tree-structured generation patterns rather than chains explore a broader range of potential completions. [ 7 30 31] propose to replace draft models with specifically trained speculative heads integrated in the target model.
Despite advances in SD algorithms, it has long been considered ineffective for large batches 32 33
27 or MoF 7 8 since the verification time these cases significantly increases. Until recently, MagicDec 34 first challenged that in long-sequence regimes, SD can effectively accelerate large batches, primarily due to the significantly increased KV cache altering the computation-to-men memoory access ratio of the model. However, SD research for MoE remains unexplored. In response, our work fills this gap, unveiling that under certain conditions, SD can effectively accelerate MoE models.
3 Theoretical Analysis
In this section, present the theoretical analyses supporting our conclusion that SD can be more effective for MoE than dense models at moderate batch sizes. We begin by formalizing general SD speedup and introducing our new metric target efficiency (Sec. 3.1 Then, focus MoEs, analyzing how workload and MoE sparsity collectively affect the number of activated experts and SD) speedup (Sec. 3.2 Based on these analyses, we develop performance model that aligns with GPU
results (Sec. 3.3 We further discuss the practical value of our theoretical findings (Sec. 3.4).
Preliminaries. LLM inference time is collaboratively determined by computation and memory access. When an operator is processed on GPU, memory access and computation operations are pipelined and overlapped, causing the more time-consuming operation to become the bottleneck and determine the overall processing time, as depicted by the roofline model [35 36). The roofline model ridge point (RP) of hardware and the arithmetic intensity (AI) of software are defined as Eq. []] When AI RP, the system is memory-bound and adding more computation will not significantly iecrease processing time. When Al > RP, the system is compute- compute-bound, and increases in computation will directly reflect in processing time. In this paper, when describe system as "more memory-boundl is smaller.
we mean RP
peak computation power (unit: Flops)
computation operation (wnit: times)
RP
AI
peak memory bandwidth (umit: bytes/second)
memory access volume (unit: bytes)
3.1 Formulation of Speculative Decoding Speedup and Target Efficiency
We first formalize the processing time of speculative decoding, denoted as TSD. To generate a sequence of length S, SD goes through Rt rounds, each containing three stages: ① the draft model proposes y tokens as specified by the speculation strategy; @ the target model verifies these tokens; @ rejection sampling [9] discards incorrectly predicted tokens based on logits from target and draft models. We use Ty s) and (b s) to represent the time for once forwarding of the target and draft model, respectively, where b and are the formal arguments for batch size and the number of tokens to processs[Thrreee Therefore. the time for processing a batch containing B requests is given by:
TSD = R x (Tpropose + Tuerify + Traject) = R x (7- (n_,B( (T(,,)) + Treject
(2)
Then the speedup of SD to normal auto-regression decoding TAR is given by:
Since we work with typical sequence lengths and moderate batch sizes, the impact of KV-cache on performance is limited, allowing us to omit the already generated sequence length from our analysis. For cases where KV-cache becomes the dominant factor, see (34]
3

---

TAR
S-Tr(B.1)
Speedup
(3)
TsD
R. (7 (T-TD(B, 1) + (T(BBB 7) Treject
S
1
(4)
R
7 (B.II)
(b(1.1)
+
Tr(B.r) TH (T.11
Traiect
g represents the average length of accepted tokens per SD round, which can be further expressed R
as a x (7 + 1). Here, a is the ratio of actually generated tokens to the theoretical maximum draft tokens in previous works [10 77. which represents the probability of the target model accepting new draft token given the prefix. can be computed from shown in Eq. [ The numerator follows from [10], and the denominator accounts for all y draft tokens accepted, plus a bonus token generated during the forward verification pass.
accepted. We note that differs from the acceptance rate commonly referenced
1-a
expected generated tokens maximal possible accepted tokens
1-a
Y +
The denominator of Eq. consists of three terms.
To(B.1) is the ratio of draft-model forward time [r,,
target-model forward time, reflecting the relative volume of draft and target models. It is also
kept small (usually less than 1/10 27 [0]) ensure the speculation is efficient. Treisct even
Tr(B,1)
smaller, since Treject only involves sampling rather than model inference. Tr(B.r) which is the ratio Tr((,I)*
of multi-token forward time over single-token forward time, has the biggest value among these three items and significantly affects the final speedup. As indicated by Eq. 4 its increase causes speedup reduction. Two different factors drive its increase, explaining SD's ineffectiveness under (1) large batches for both dense models and MoE, and (2) MoE with small batches, respectively:
(1) The compute-boundness. The model's Tr(B.r)
(z(,II) approaches 1 when more memory-boume
(smaller batch size B) but increases to 7 when more compute-bound (larger batch size B).
(2) The extra memory loads. For small Bs, (B,))) is notably greater than Ty [B, 1) as more experts activated and need to be loaded. Since the system is still memory-boo -ouud at small Bs, memory load profoundly determines the processing time.
(B which helps understand the systemic causes of SD acceleration degradation listed above. Our experiments shown in Fig. demonstrate that target efficiency consistently reflects the trend of SD speedup variations. Despite the importance of this value, previous works have rarely noticed it, primarily due to differences in research focus. Previous SD research mainly addresses the question by lifting the acceptance rate:
Therefore, we define target efficiency (B.1.))
Given the target model, which draft model or algorithm achieve better speedups?
In contrast, our work focuses on the following question by examining target efficiency:
Under the same level of ulgorithmic optimization, which types of target models or workloads are more favorable for SD?
We believe target efficiency help researchers understand SD more comprehensively. Even when target-draft pairs have the same acceptance rate a, changes in the target model's architecture and the workload can significantly affect overall speedup. By introducing target efficiency, we can decouple algorithmic optimization from systemic optimization, thus helping to identify the systemic acceleration bottlenecks and assess potential speedup.
3.2 Moderate Batch Size Enables Speculative Decoding Speedup for MoE
Although SD is ineffective for MoE under small batches, we demonstrate in this subsection that at moderate batch sizes-an overlooked regime previous studies-SSt SD speedup increases and benefits more from MoE with higher sparsity. Essentially, when the batch size falls within ranges where

---

addyvattdd Experts (N09))
Nommattee tokeex pw excent Teghh01
60
1.0
80
50-
130
0.8
80-
Number Actovaodd
30-
20
0.4
20-
Thersticall Huma-Eval
20-
Theursiccal HumanExal
10-
Bench
0.0
100
150
200
250
50
100
150
200
250
Namber of Tokens (1)
Number of Tobens D1)
(c)
Figure 1: Activation status and workload of experts. (a) and (b): Comparison between theoretical and actual number of activated experts (()) different datasets. (a) is for Deepseek-V2 V2-Lite-Char (p 6/62) and (b) is for 1. (p 4/60). (c): Normalized number of tokens to M-5-hoh-hhat
process per expert (Tezp) versus MoE sparsity (p) for given input token count T.
all experts are activated but remain far from being assigned adequate workloads, FFNs become memory-bound, presenting opportunity to leverage computational power almost for free through SD. To demonstrate this, we first formalize the expected number of activated experts, and then show MoE FFNs become more memorry-ounn as the model becomes sparser.
We the Bernoulli random variable X indicate the activation for experts: X; for expert being activated, 0 otherwise. For simplicity, we assume Xs are i.i.d. Then, the expected number of activated experts can be expressed as Eq. 6 where E denotes the total expert count and Pr(X;) represents the probability that the ith expert activated.
Σ x Σ Pr(X,) Pr((()
Given / tokens passed through the MoE gate, then Pr(X) is expressed as Eq. 7 K denotes the number of activated experts per token, which is architectural hyperparameter for MoE:
Pr(X)= = 1 Pr( None the / tokens activates the expert =1 E K
(7)
Therefore, the overall expression of (()) is given by Eq. Our derivation assumes uniformly activated experts, which is reasonable for well-trained MoE models. Load imbalance among experts can lead to routing collapse [5] and decrease computational efficiency in expert-parallel deploy- ment 6 state-of-the-arr MoE models are typically trained with methods like incorporating auxiliary loss [37 6 to ensure that experts have balanced loads. The experiment results also verified our theoretical derivation of (t), shown in Fig. Ialand lb
((1) = E (1-(
EI-
8
We then solve how many tokens can lead to full activation. Since N(I) asymptotically approaches E when tends to infinity, and practice N(I) should be finite integer, we deem N(I) > 7 F as almost full activation, where 7 is usually a large ratio such as 0.95. We further express K pF., where p is the sparsity of MoE, then the token threshold Tihres can be solve by:
(Tehres)) E E- (1 (1-
ZTE = Tihres [lo r)
(9)
Therefore, when B exceeds Tehress the number of activated experts saturates, causing the By tokens in verification to incur only marginally larger memory access. Having addressed the second factor (namely, extra memory loads) for (B. 's increase analyzed in Sec. 3.1 we now turn to the potential (B(,
limitations caused by the first factor of compute-boundness. boundness. If such Bs make the system compute- bound, SD would also fail to accelerate MoF effectively. Our answer to this concern is: Sparser MoEs delay the transition from memory-bound to compute-bound when input tokens count increases.
We have obtained that given & tokens, (t) experts are activated. Since each token activates K experts, the number of tokens each expert needs process on average Tezp be computed as:
Tezp t. K-K
t (pE)
pt
(10)
N
E. (1 (1 - (1))))
1
/)
5

---

Algorithm The Modeling of SD Speedup and Corresponding Fitting Method
I:
Measurement Input: A total of m measurements denoted as M. Each M:, = 1,2, .., m contains the attributes including batch size B, draft length 7, number of activated experts per token K, total number of experts E, the ratio of accepted token counts to the maximal possible accepted tokens a, Speedup for the actual speedup achieved.
Output: The optimal litting parameter params".
3: def ComputeSpeetuppprrams
B, 7, K, oy:
Compute the SD Speedup
4: bias, Yn, h2, k3, draft bias, draft k. reject_ bios, reject k, X, 5 params
Unpack parameters
5: Nur = E (1 ((E K) E) ), Ter B K/Na-
D Compute AR forward time
6: ar time biay +ki- G(B; ARP, s) + k2 Nar + K3 G(Tari XRP,
7:
Ned= F. (1 ((F) - K) BT), Ted =ed= B.y. KK/Nsd
> Compute SD forward time
8:
verify time = bias +hy- G(B;; XRP, 8) K2 Nad + K3
G(T.a; ; sRP,s)
9:
draft time draft bias draft k. G(B;ARP,s)
> Compute draft model forward time
10:
reject time reject bias reject_ k B
P Compute rejection sampling time
11:
Speedup = a (7+ 1)- draft none or Ring + verify tnne+rejestttnm time
Line
b Compute the speedup as formalized Eq.
12:
return Speedup
13: params* * argmin
ComputeSppeduupprrrnns, M. M.,, M..K, M..E, MM.))M..Spedup
Speeduup
peramss
> Decide the optimal params * by fitting the model to the measured inputs using the least squares criterion.
As proven in Appendix and shown in Figure 1c given = T 1, (Tzp(T;T)) (T; decreases with p. indicating that as MoE becomes sparser, each expert processes fewer tokens per parameter loading. Consequently, the system running sparser MoEs is more memory-bonnd, leading lower utilization of arithmetic units. The verification stage can therefore leverage these spare resources without notably increasing processing time. In contrast, dense models are extreme cases with p = 1, where the FFN consistently approaches the maximal possible arithmetic intensity of T, and the system transitions rapidly the compute-boune bound regime T increases.
We should note that our conclusion is based on a relatively large MoE FFN portion in the whole model, which holds true for current MoE models whose most parameters are experts. In hypothetical extreme case where Attention dominates and the MoE FFN is negligible, MoE's sparsity would have only limited impact overall system performance indicated by Amdahl's Law.
3.3
A Modeling Method for Speculative Decoding Speedup
Given the numerous factors affecting final speedup, quantitatively understanding each factor's 's impact challenging. Therefore, we developed a modeling method that makes SD speedup results more explainable and transparent. As demonstrated by Eq. 4. the core of modeling SD speedup lies in characterizing the model's forward pass time. Based on theoretical analysis in previous sections, we identified three primary factors affecting forward execution: (1) the roofline model effect, (2) the number of active experts, and (3) expert load. Since GPU execution is dynamic in practice, and not all operators are optimized their theoretical limits, we introduced several parameters for relaxation. The values of these parameters are then automatically determined by fitting GPU measurements. These factors and their impacts on execution time are examined as follows.
(1) The roofline model effect. It manifests as execution time increases with token counts I, with growth rate that starts slow, then accelerates, and finally stabilizes. The underlying reasons are as follows. When t is small, parameter loading time exceeds computation time, creating a memory bottleneck. Therefore, given the parameter volume, the memory access time is stable (memory- bound regime). As & increases, computation time exceeds parameter loading time and becomes the bottleneck. With fixed arithmetic units in the hardware, computation time scales linearly with computational load (compute-bound) regime). To characterize this trend, we design G(t; XRP.s) as shown in Eq. [1 where XRP represents the transition point between regimes, and s controls the increasing rate of execution time. Here, RP follows Eq. [ and À is constant less than that accounts for practical limitations in memory bandwidth utilization. G(t) exhibits gradually increasing slope before the transition point, then shifts to a linear function afterwards, maintaining first-order gradient

---

continuity at the transition.
5 XRP
GRP, SARP + +( ((daa" (=ARE)
)t- (t- ARP) = sAPP sAnP(1 (1+ (t- (t (RP)) t> XRP
(11)
(2) The number of activated experts. When it increases, the memory access volume increases. thus adding to the final processing time. We use the derived of N to characterize how workload and model architecture affects the number of activated experts.
(3) Expert load. This refers to the fact that after token distribution through the MoE gating, each expert processes only a subset of tokens Texp(t; p) rather than the entire input token count Therefore, we should use (TTerp) rather than G(t) when applying the roofline model to MoE experts. This corroborates theoretical conclusion that sparser MoEs delay the transition from memory-bound to compute-bour bound when input tokens count increases.
For the MoE target model, factors (1), (2), and (3) are all involved. We combine them in first-order style and introduce parameters bias, ki, k2, and k3 to adjust for non-ideal factors in actual GPU execution, with the full expression shown in lines 6 and 8 of Alg. [ These parameters have clear practical meanings: bias represents the time required to load fixed parameters; k2 N represents the time needed to load N activated experts; k G(t) and K3 G(Tezp) describe the incremental trend in execution time as the number of tokens increases. For the draft model, only factor (1) is involved since it usually dense, with the modeling form shown in line of Alg. T
With the expression of SD speedup determined, we fit the measurement inputs to automatically determine the relaxation parameter values, with the optimization criterion being the minimization of Mean Squared Error (MSE) between the model outputs and the ground truth, as shown in line 13 of Alg. D By applying these optimized parameters in our model (i.e., the ComputeSpeeduup function in line 3), we obtain the complete modeling for SD speedup. An illustrative diagram of this process and more fitting details are provided in Appendix
Since our theoretical analyses capture the primary tradeoffs and provide a solid foundation for the modeling, the fitting is very efficient. The fitting results with 21 measurements are displayed in Fig. 4 which show consistent trends with GPU results under various cases. These results validate the reliability of our modeling, thereby establishing it effective tool for analyzing the components of the model's s forward pass and quantitatively understanding the tradeoffs between different factors. As shown in Sec. 4.2 we explain some unexpected results with the help of the model.
3.4 Practical Values of Theoretical Findings
While previous sections focuses on theoretical analysis, this section demonstrates how these findings translate to practical speedups. Our theoretical analysis has already revealed that SD speedup for MoE is most effective at moderate batch sizes, with its trend initially increasing and then decreasing. We discuss their practical values considering both basic deployment and extended configurations.
Basic deployment. (1) Moderate batch sizes are common in private serving, which are increasingly adopted for data security, with representative applications like enterprise in-house chatbots. (2) When latency requirements are strict, large batch sizes often not feasible. LLM serving must satisfy multiple service level objectives (SLOs) 38), including time-to- -frrs----itttt (TTFT) and time-per- output-token (TPOT). Large batches reduce per-request computational resources, causing latency violations. In such cases, moderate batch sizes are common. (3) Our work actually reveals that SD on MoE relaxes the traditional latency-hrrughpul trade-off. Specifically, MoE models exhibit a regime where SD speedup increases (lower latency) alongside larger batch sizes (higher throughput).
From the model's perspective, moderate batch sizes represent an "efficiency gap" in MoEs. At this scale, all parameters must be loaded (unlike small batches with selective expert activation), yet GPU FLOPs are not fully utilized (unlike large batches). Our findings provide a novel perspective to address this efficiency challenge without compromising model quality.
Extended configurations. We consider typical system optimizations on MoE like ofloading and experi parullelism (EP). When MoE models exceed GPU memory capacity, FFN parameters are offloaded to CPU memory (39). This degrades parameter loading bandwidth from GPU memory bandwidth to much lower PCIe bandwidth, making the system more memory-bound Consequently,
7

---

humanesll tompeó, y Madil
GPU readts
CPU rando
GAU neult
2.0
broot effceeny
Boor Eiiieny
Saguc Efficoncy
1.70
Ccredin
14
1.30
100
1.0
100 1.8
100
Butch
Surch sizx
(a)
(b)
(c)
Figure 2: SD speedup (left y-axis) as a function of batch size and corresponding target efficiency values (right y-axis). Across different hardware platforms and MoE models, SD speedup first increases and then decreases, verifying our theoretical predictions. The target efficiency shows consistent trends with final speedup, validating its effectiveness.
Table I: The peak speedup (x) of SD across different datasets, temperatures, ys and models on 2xA800
Device
Temp
2
3
TAB
Tep
0
x
TAR
TsD
0
X
TAR
Tsp
X
humsneval
18.89
11.61
0.94
15.93
8.11
0.03
7.31
0.91
12.93
1.48
14.00
0.73
11.14
0.67
Qwen2
mibench
0.0
20.92
16.70
0.71
1.25
16.00
12.43
0.62
1.29
20.92
17.53
0.55
1.19
mibench
21.15
17.33
0.68
1.22
19.09
14,83
0.57
19.09
15.93
0.48
homsneval
12:47
21.00
0.66
11.69
0.58
humaneval
21.52
15.58
0.61
1.38
16.00
0.46
1.33
16.25
0.39
mibench
0.0
21.61
16.10
0.61
1.34
21.61
16.43
0.46
1.30
21.36
6..89
0.39
1.26
mibench
21.33
17.70
0.53
1.21
21.33
17.84
0.43
1.20
21.33
18.05
0.35
1.18
additional computation does not significantly increase processing time, creating favorable conditions for SD. Notably, existing optimizations like expert prefetching [23] [24] and caching [25] 26 lose efficiency under moderate batch sizes since nearly all experts are activated.
Our findings are also compatible with EP. EP, experts are distributed across multiple GPUs, which affects neither N (f) nor Teep, making our previous analyses remain valid. Since components besides MoE FFN are also parallelized, MoE FFN continues to constitute significant portion of processing time, allowing memory- memory-bounding boundnees effects to remain observable in end-to-ent performance. Notably, under extensive EP configurations, the inefficiency of SD for MoE at small batch size may vanish, considering the additional memory bandwidth offered by large amounts of EP GPUs.
4 Experiments
Models and datasets. We evaluated two MoE configurations with SD: Qwen22-57--A14--Inmstt with Qwen2-0.5P 0.5-BBBIsIsteee [40). representing famillymily drafting. and Bastral-xx7-IInttie v0.1 [41] with Eagle (7], representing specialized speculation heads. When we need to examine MoEs with different sparsity, we modify the num_ experts_per
in the model's config.
file. For comparison with dense models, we use Opt-30b and Opt-350m [42 as the target and draft models. Models are evaluated on HumanEval 43 and benhhecc 44 datasets for code generation
and conversation tasks, following previous works 7 45
11. The tokenized prompt lengths range
from 38 to 391 tokens for HumanEval and to 356 tokens for MT-bench.
Frameworks and hardware. We used the existing vllm [46] framework for our experiments to verify theoretical predictions. Vllm supports batched speculative decoding, cudagraph optimization, and reports comprehensive data such as Tp. Tr, Treject and a, thus being suitable for our experiments. To prevent unstable performance the beginning, all data were obtained by averaging the results from the last five of the total ten runs. We conducted experiments on different hardware platforms including 2xA800, 2xH800, 4xA800, 4xL40.
4.1 Speedup Trend of Speculative Decoding for MoE
Figure plots the end-to-end SD speedup (left y-axis) for MoE across various settings, validating our theoretical prediction about acceleration behavior. As batch size grows, speedup initially increases

---

Table 2: The peak speedup (x) of across different datasets, temperatures, 7s and hardware on Qwen2
Devics
Danset
Temp
3
TAR
TgD
2AR
13D
TAR
x
humuneval
0.0
15.96
15.96
2.29
2xH800
homaneval
100
17.39
12.82
0.82
1.36
13.30
5.98
0.74
1.47
13.20
7.17
0.75
1.84
mibennch
0.0
24.42
16.74
0.71
1.46
24.42
16.34
0.02
1.45
24.42
17.05
0.54
1.43
1.0
18.24
16.25
16.25
13.76
1.18
humaneval
0.0
11.20
11.20
1.90
11.20
2,08
4xA800
hamaneval
1.0
11.72
8.51
0.81
1,38
12.05
8.30
0.73
1.45
11.25
7.70
0.67
1.46
mibench
0.0
11.25
8.92
0.72
1,26
11.26
9.10
0.61
1.24
11.25
9.82
0.52
1.15
mubench
11.78
1,05
hamaneval
0.0
17.84
17.54
2.14
17.84
2.25
hamaneval
1.0
17.89
12.27
0.80
1.46
17.89
11.07
0.74
1.62
17.89
10.91
0.65
1.64
4xL40
mibench
0.0
20.40
15.87
0.71
1.29
20.40
16.22
0.62
1.26
20.40
16.33
0.54
1.25
mubench
18.11
1.17
due to expert loading saturation, and then decreases due to compute-boundness. We denote the maximal speedup across batch sizes as x and summarize the results in Table For both models, SD achieves higher acceleration with longer y for tasks with more predictable patterns (e.g., code generation) or less randomness (e.g., lower temperature), aligning with conclusions from previous research. Figure 5 in Appendix A. further presents SD speedup trends under more settings, including individual runs and their mean to show the statistical significance of our findings.
We further evaluate Qwen2-57B-A 1-0-A114--- Ins-tutt on multiple hardware platforms (Table to verify the generality of our conclusions. Combined with results of Qwen2 in Table [ two observations can be made: (1) GPUs with higher ridge points yield larger SD speedups (c.g.. 2xA800 vs. 2xH800, 4xA800 vs. 4xL40), since they provide more arithmetic units for verification. (2) Scaling from 2xA8000 to 4xA800 reduces absolute runtimes (TAR and TSD), but the SD speedups slightly degrade. This is because the large model benefits from inter-GPU parallelization, whereas the small draft model remains Single-GPU making its relative forward cost higher.
Figure 2 also highlights the effectiveness of our metric target (B..) as explained Sec. 3.1
efficiency. It is computed as (r(
humeneoel tempmn, 4
0.5
VuE 555- rotruct
where both TT(B, 1) and Tr (B,7) are obtained from vllm runtime logs. Target efficiency values are annotated on the right y-axis, showing consistent trend with the end-to-end speedup. In contrast, the acceptance rate across batch sizes merely fluctuates within a small range, unable to effectively explain the dramatic changes in speedup.
Derce mocel 100h
bygoetiice orcr
5.
5.5
We further compare MoE and dense models via target efficiency (whose validity established above) to avoid interference from acceptance rate. As shown in Figure 3 the target efficiency for MoE first increases and then decreases, while that for the dense model decreases continuously. Consequently, although SD for MoE less effective with small batches, it exhibits stronger potential across wider range of larger batch sizes. Regarding end-to-end performance, SD speedups become more pronounced for MoE when the batch size exceeds 16, as supplemented in Figure 6 Appendix(A.) A.2
60
103
Buncch
Figure 3: Comparison of target effi- ciency: MoE vs dense model.
4.2 Impact of MoE Sparsity and Validation of Modeling Method
To evaluate MoE sparsity's 's impact on SD acceleration, we varied the number of activated experts per token (K) of Qwen2-57B-A. target model's performance and speculation accuracy, so we adjust the speedup by multiplying the raw speedup with _K=t, whose rationale is exhibited by Eq. Fig. 4 shows the adjusted speedup .x alongside our modeling results for comparison. The parameters used in the modeling are decided using 21 GPU measurements, as explained Section (.3) The impact of measurement selection for parameter fitting on the modeling's reliability is supplemented in Appendix C
Directly changing K without training affects the

---

Draft length
Draft length
2. US4 y
y64.y=2
2,
y
y
3.75
Speclie
Spectep
"Secodue
1.35
Specolu
12-
1.23
1.0
1.22
6.55
2.0
40
20
63
102
45
p. y
2154,
1.
5p00040
Specdan
Spccdlu
40
20
09
103
46
2.00
p
2.0
p.e. 2
2.5
1854, y-m
1.0
52, p
Speotue
Spentta
Spccdaus
2.00
40
20
103
45
Figure 4: Comparison between GPU results and our modeling for Qwen2-57B-A114--msss with varying sparsity p and draft length y.
There are three key observations. First, the modeling consistently aligns with experiment results across varying sparsity (p) and draft length (7), validating our modeling's 's reliability.
Second, while the SD speedup in most MoEs exhibits an initial increase followed by a decrease, very sparse MoEs (K = 1, 2) show continuously decreasing speedup. This appears to conflict with the theoretical analysis, but after examining the components of our modeling, we identified the reason as follows. These very sparse MoEs have a disproportionately low ratio of FFN, thus making the memory- boundness of MoE. FFN hard to manifest systematically as indicated by Amdahl's Law. The (rre22577--144- model is designed based on K 8, but by reducing K to I or 2, we actually artificially synthesize a model where Attention dominates. In practice, however, sparser MoEs typically incorporate more FFN parameters to maintain a balanced ratio between FFN and Attention components, resulting in acceleration patterns more similar to K = 8 cases.
Finally, as MoE models become sparser, the system's 's transition from memory-bound to compute- bound delayed. This is evidenced by two phenomena in Fig. 4 With smaller p. (1) the batch size for the maximal speedup (x) becomes larger; (2) the range of batch sizes that maintain speedup above a certain decay threshold (annotated by the brown dashed line in Fig. 4 for x/v2) is wider. These validate our theoretical analysis and indicate that SD has broader applicability in sparser MoEs.
Conclusion and Limitation
This work challenges the conventional wisdom that speculative decoding (SD) cannot effectively accelerate MoE models. We demonstrate that under moderate batch sizes, sparser MoEs gain greater speedup from SD due to their memory-bound FFNs, conclusion supported by both theoretical and experimental analysis. To navigate the complex factors influencing speedup, we develop a reliable, interpretable SD model and introduce target efficiency to elucidate the impact of model architecture and workload. Our work offers new perspective for accelerating MoEs, particularly in memory-constrained or moderate-batt than parameters, the behavior when it dominates analyzed by MagicDec [34]: these two works combine to give more comprehensive view of SD across batch sizes.
serving. While our analysis assumes the KV-cache is smaller
Acknowledgments and Disclosure of Funding
This work is supported by the National Key Research and Development Program of China under Grant (2024YBB4405400).
10

---

References
[1] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[2] Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
[3] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specializa- tion in mixture-of-cxpeeee
language models. arXiv preprint arXiv:2401.0600 06066, 2024.
[4] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2. A strong. economical, and efficient mixture-of-experts language model. arXiv preprint 4rXiv:2405.04434. 2024.
[5] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 0001170013..... 2017.
[6] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[7] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077 2024.
[8] Anish Saxena, Po-An Tsai, Hritvik Taneja, Aamer Jaleel, and Moinuddin Qureshi. Utility-driven speculative decoding for mixture-of-experts, 2025.
[9] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, -Bptiite Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013/6 2023.
[10] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference Machine Learning, pages 19274-19286. PMLR, 2023.
[11) Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng. Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads, 2024. URL https://arxiv. org/abs/2401.100 10774, 2024.
[12] Rahul. What private Ilms? running large language models privately privategpt and beyond, 2024. hattpp//aalllll..commm////ttatt/
[13] Esther Julie. What is a private I1m and why should you build one?, 2024. https: //t::://wwwwww. www. inoru http:/haaiiiisssi is- private- 11m- and--and-aaa soedd-why-ssaaee sooul--ouu--y----aoa build-uni one/
com/blog/whatet
[14] Hanbo Huang, Yihan Li, Bowen Jiang, Lin Liu, Bo Jiang, Ruoyu Sun, Zhuotao Liu, and Shiyu Liang. Position: On- On-premises IIm deployment demands middle path: Preserving privacy without sacrificing model confidentiality, 2025.
[15] Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. Moe- Moe-pruner: Pruning mixture-od of expertss large language model using the hints from its router. arXiv preprint arXiv:2410. 12013, 2024.
[16] Jaeseong Lee, Aurick Qiao, Daniel F Campos, Zhewei Yao, Yuxiong He, et al. Stun: Structured- then-unstructured pruning for scalable moe pruning. arXiv preprint wrXiv:2409.06211 2024.
[17] Elias Frantar and Dan Alistarh. Qmoe: Practical sub-l-bit compression of trillion-prrameter models, arXiv preprint arXiv:210..16795. 2023.
[18] HamidReza Imani, Abdolah Amirany, and Tarek El-Ghazawi. Mixture of experts with mixture of precisions for tuning quality of service. arXiv preprint 4rXiv:2407.14417, 2024.
11

---

[19] Felipe Cruz Salinas, Kenichi Kumatani, Robert Gmyr, Linquan Liu, and Yu Shi. Knowledge distillation for mixture of experts models speech recognition. Technical report. Technical Report
Microsoft Research, May 2022. https://www ... 2022.
[20] Fangxun Shu, Yue Liao, Le Zhuo, Chenning Xu, Lei Zhang, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, et al. Llava-mod: Making llava tiny via moe knowledge distillation. arXiv preprini urXiv:2408. 15881, 2024.
[21] Cheng Yang, Yang Sui, Jingi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. Moe-i:: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition. arXiv preprint arXiv:2411.010f6 2024.
[22] Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy. arXiv preprint arXiv:2310.01334 2023.
[23] Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. Moe-infinity: Activation-awea -wware expert offloading for efficient moe serving. arXiv e-prints, pages arXiv-2401, 2024.
[24] Shuzhang Zhong, Ling Liang, Yuan Wang, Runsheng Wang, Ru Huang, and Meng Li. Adapmoe: Adaptive sensitivity baste- expert gating and management for efficient moe inference. In Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design, pages 1-9, 2024.
[25] Xin He, Shunkang Zhang, Yuxin Wang, Haiyan Yin, Zihao Zeng, Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, Ivor Tsang, and Ong Yew Soon. Expertflow: Optimized expert activation and token allocation for efficient ofxture-of-expernss inference. arXiv preprint arXhv:2410.17954 2024.
[26] Peng Tang, Jiacheng Liu, Xiaofeng Hou, Yifei Pu, Jing Wang. Pheng-Ann Heng, Chao Li, and Minyi Guo. Hobbit: A mixed precision expert offloading system for fast moe inference. arXiv preprint 1eXiv:2411.014333 2024.
[27] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 932-949, 2024.
[28] Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. Rest: speculative decoding. arXiv preprint arXiv:2311.08255 2023.
Retrieval-based
[29] Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. Specexec: Massively parallel speculative decoding for interactive I1m inference on consumer devices. Advances in Neural Information Processing Systems, 37:16342-16368, 2024.
[30] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406. 1685,, 2024.
[31] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training- -ime test. arXiv preprint wxxv:2503.00844 2025.
[32] Xiaoxuan Liu, Jongseok Park, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Chen Zhang, Kuntai Du, Xiangxi Mo, Kaichao You, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Turbospec: Closed-loop speculation control system for optimizing Ilm serving goodput, 2025.
[33] Qidong Su, Christina Giannoula, and Gennady Pekhimenko. The synergy of speculative decoding and batching in serving large language models, 2023.
[34] Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. Magicdec: Breaking the latency- throughput tradeoff for long context generation with speculative decoding. arXiv preprint arXiv:2408. 11000, 2024.
12

---

[35] Zhihang Yuan, Yuzhang Shang. Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. I.l inference unveiled: Survey and roofline model insights, 2024.
[36] Georg Ofenbeck, Ruedi Steinmann, Victoria Caparros, Daniele G Spampinato, and Markus Püschel. Applying the roofline model. 2014 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 76-85. IEEE, 2014.
[37] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022.
[38] Zhibin Wang, Shipeng Li, Yuhang Zhou, Xue Li, Rong Gu, Nguyen Cam-Tu, Chen Tian, and Sheng Zhong. Revisiting slo and goodput metries in Ilm serving, 2024.
[39] kveache ai. Ktransfromers: A flexible framework for experiencing cutting- Ilm inference optimizations, 2025.
[40] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, et al. Qwen2 technical report, 2024.
[41] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
[42] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066. 2022.
[43] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, et al. Evaluating large language models trained on code, 2021.
[44] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
[45] Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. Learning harmonized represen- tations for speculative sampling, 2025.
[46] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[47] NVIDIA Docs Hub. Matrix multiplication background user's guide. http:httttsssssnnnttee .ww.//diss..iiiaiaa............a http://pplaariiimm///rrrommree/e http:///w...aaaeeao/oo//ooooa/aa
html
[48] Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024.
[49] Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Tian Tang, Qinyu Xu, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Ziren Wang, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikei. Nanoflow: Towards optimal large language model serving throughput, 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Abstract and Introduction. The three items in the contribution list are explained in detail in Section and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section 5 The last several sentences discuss our limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Please refer to Section and Appendix in the supplementary materials. We provide the meaning of all variables and assumptions with the proof.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer to Section for experiment settings. Besides, we provide complete codes the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the complete codes and data supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.co. ccippi.cc guwdes/ddeeeee CooeeSumeeieeeeeeeeeee ssi onyllicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Please refer to Section and Appendix in supplementary materials. The experiment details can also be found in supplementary codes.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not include error bars. However, all results are averaged over 5 runs to avoid randomness as mentioned in Section 4 Besides, the Appendix in supplementary materials further discusses the impact of data selection on model fitting.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section 4 Besides, the readme file in supplementary codes also describes the computer resources needed for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our research focuses speculative for MoE, conforming with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer to Section We cite the vllm framework as its GitHub main page suggests.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20

---

(a)
humaneval, tempmm, (wenn-578-l1-BBhhnnuu
numaneval, temp-1. Owen2-578-A14-
Average
Average
Run
1.6
Spencas
1.75
Run
Spaedup
Run
Run
Run
1.50
1.2
1.25
1.0
Batch
(c)
mtbbanh,, tomp-O, 578 Instruct
mibench,
578 Instruct
3.3
Average
Meraag
Run
Run
Run
Run
Speedus
Spacdup
Run
1.0
Run
Run 5
Run
0.8
0.8
40
60
100
20
60
1.00
Batch size 8
Batch size
(e)
humeneval, temp=00 Mintrallbkk0--nnssrurtt.....
(f)
humaneval, temp=1. Mixtra-8xx8 7001in-t-.---
2.8
Average
1.4
Average
1.6
Run
1.2
Run
Speedup
Run 3
Speedup
Bun
1.0
Run
0.8
0.6
20
60
80
100
20
80
100
Batch size B
Batch size 8
(g)
mibench, temp=o, 1
mtbbench, temg-1. 1. Mixrral-xx7B.nstrrutt000
Average
Average
Run
Run
Speedup
1.0
Run
Run
Run
2.0
0.9
0.8
0.7
size
Batch size
Figure 5: SD speedup trends across more settings with individual runs and averages shown.
A Supplementary Experimental Results
This section presents additional experimental results referenced Section which are included here due to space limitations.
A.1 Trends of SD speedup under more configurations
Figure presents additional trends of SD speedup across different datasets, temperatures, and model types, serving as supplement to Figure [ The results demonstrate that SD speedup exhibits a consistent first-nncrease-thee teasss-tthen-eeeeeee pattern, which aligns well with our theoretical analysis.
To confirm the statistical significance of our findings, we also present the five individual runs that constitute the averages in Figure 5] The variance across different runs is minimal, which is expected since the random seed is fixed across all runs to ensure identical workloads.
While the overall trend follows the first-increase settreesse-heen--eee decreaee pattern, local fluctuations are ob- servable the curves. For instance, Figure (Jc) exhibits a sawtooth-like decreasing trend. This phenomenon can be attributed to GPU quantization effects, as documented in NVIDIA's documenta- tion [47] When dimensions are not evenly divisible by the GPU's native tile sizes, computational performance degrades. AR decoding is more sensitive to this effect than SD, making the time ratio of
21

---

Temperature: length:
Temperature: length:
2.7
0p-300
020-00 Geer2574400issnn
Geet 578.3200 estred
2.50
0 inaraat
2.35
2.5
Spredhon
2.6
Spocdup
2.00
Spredtup
2.75
2.0
2.4
1.5
1.0
Batch
Batch
Batch
Temperabure: 1.0, Draft length: 2
Temperature: 1,0, Draft length:
Temperature: 1.0, Draft length: 4
1.8
0y-33o
1.4
1.5
3.4-
Speeduup
Spenclup
2.2
Speesuup
3.2
2.2
2.5
0.5
3.5
50
90
120
0
20
50
80
100
20
80
50
100
Batch wiw
Batth
Batch
Figure End-to-end speedup comparison of MoE and dense models under various settings.
AR to SD (namely, SD speedup) fluctuate. Despite these local variations, the overall speedup trend follows our theoretical predictions, confirming the validity of our conclusions.
A.2 End-to-end speedup comparison between MoE and dense models
To isolate the effects of acceptance rate variations and enable clearer focus on system bottlenecks, we have compared MoE and dense models using target efficiency Section 4.1 In this section, we further compare their end-to-end speedup across various settings in Figure 6
supplement.
Two key observations emerge from the experiment results. First, while SD speedups for MoE models initially increase before declining, SD speedups for dense models continue to decrease. Consequently, SD achieves more substantial end-to-end speedups for MoE models at moderate batch sizes, which aligns with the trend in Figure 3 in Section[4]] 4.1 Second, the extent to which SD favors MoE over dense models varies across different configurations. For instance, at temperature demonstrates greater relative benefits for MoE compared to temperature 0 (first row). This variation stems from diverse acceptance rates under different settings, which can obscure the observation of systemic bottlenecks. In summary, target efficiency serves as a reliable comparison metric while controlling for the confounding effects algorithmic optimizations.
(second row), SD
B Proof of (T:p(T;;)) p)'s Trend with Varying p
(T;
In Section 3.2, Fig. I(c) demonstrates that: Given input token count T'> 1, the number of tokens each expert processes on average Texp (T: p) 1- decreases as p decreases. We prove this by
(TTp) 0 when T 1. dp
showing
((Texp(T:))
oT d( d(--(-p)T)
T( T(-pT(1 (1T(1-))T-- T1 -(1 p)² +1)
(12)
dp
dp
(1-(1- (1-11p)) 7)
Since P represents MoE sparsity E (0, 1), the original proposition is equivalent to proving:
F(p T) (1 - T> TpT +1 11 <
(13)
Note that TT) + as p - Therefore, if we can prove that F(p; I') decreases as p increases from 0 to 1, then the original proposition is proven. We demonstrate this by computing d(F(p:T))
dp
d(F(T) dp
d((1 1 T- + p)) -P 1) -1 T
(14)
dp
22

---

Thearetical analysis |Section / 3.2)
Theory
Profilnng
50 speedup expression with relaxation parameters
Runtima
Real Measurements (Norklodds SD speodup)
Optimal parameters
Fitting square criterion (Section 3.3 Aooendlx 22/ C3)
Computing speedup
[Section 3.3 Appandix C22 C.3)
speeduup
Worikeads
Figure 7: The overall diagram of the modeling method.
When T' > 1. d(F(o:)) < O. This confirms that F(p; T') decreases as p increases, which proves our original proposition: when T > 1, Texp p) decreases as p decreases.
C More details about the Modeling Method
The main design considerations and expressions of our modeling method have been presented in Section 3.3 In this section, we provide additional content on the following topics to give a more comprehensive view of the modeling method:
Description and an illustrative diagram of the modeling process. (Appendix C.I
Fitting Details of the modeling shown in Figure in Section 4.1 (Appendix
C.2
How the modeling is affected by alternative measurement selection. (Appendix
The value of our modeling is twofold. On one hand, it achieves alignment with real measurements with only small number of simple parameters, thus validating the correctness of theoretical analyses. On the other hand, it provides the decomposition of various factors in the end-to-end results, making the entire SD acceleration process explainable and transparent.
C.1 Description and Overview of the Modeling Process
Figure 7 presents the overall diagram of our modeling method. Building upon the theoretical analyses in Sections [nd 3.2] derive expression for SD speedup function of workloads. This expression contains several relaxation parameters to be determined for complete modeling. We determine these parameters through empirical profiling. We first collect small set of real measurements comprising various workloads and their corresponding SD speedups. We then perform parameter fitting using these measurements under the least squares criterion obtain optimal parameter values. Since our theoretically-derivee fundamental performance tradcoffs, the fitting process is computationally lightweight and robust, as will be demonstrated in Appendix C.2 and C.3 Once the optimal parameters are obtained, the resulting expression can predict SD speedups for arbitrary workloads.
SD speedup expression already captures the
We now explain why profiling is necessary and why we cannot derive the complete SD speedup expression purely through theoretical analysis, examining both software and hardware considerations.
Software considerations: Actual execution times can deviate significantly from theoretical pre- dictions for complex operators with diverse implementations. On GPUs, GEMM operations are indeed predictable due to their regular structure and highly optimized implementations. However, prediction becomes challenging for operators such as Attention, which involve customized kernel optimizations (c.g.. FlashAttention (incorporating various nonlinear layers positional encodings such as RoPE and its variants). To illustrate this complexity, we examine profiling results from Qwen2-57B-NN and Mixtral-88 -x (hidden size 4096). For FFN, Qwen takes shorter time than Mixtral (143us vs 226us), aligning with their relative hidden sizes. However, for Attention, Qwen takes a longer time than Mixtral (27lus 1 115us), contradicting the theoretical expectation based hidden size scaling.
eager attention, SDPA attention) and operator fusion strategies
(hidden size 3584)
Hardware considerations: GPU microarchitectures vary across different series, which can greatly impact execution times. For instance, attention efficiency depends heavily hardware-aware
23

---

programming optimizations, while different GPUs vary in cache configurations and thread memoory interaction patterns across memory hierarchy levels. By taking advantage of new capabilities in modern hardware, FlashAttention-3 successfully increases GPU utilization from 35% to 75% on H100 GPUs 48). Moreover, many critical hardware details remain undisclosed by GPU vendors, making theoretical predictions impractical.
Therefore, modeling speedup trends with pure analytical methods requires case-by-case analysis for different operator implementations and GPU microarchitectures. In contrast, the hyperparameter approach offers a more generalizable paradigm and is easy to use: all parameters possess clear physical interpretations, only minimal profiling data are required, and the computational overhead is low. Our method achieves balance between effectiivnness and practicality: on one hand, it captures the primary performance drivers (i.e. the number of activated experts and roofline trends); on the other hand, it avoids getting entangled in low-level implementation complexities. This approach is also used by other system optimization frameworks such as NanoFlow [49], which similarly adopt a two-stage strategy of profiling followed by runtime execution.
C.2 Fitting Details of the Modeling shown in Figure 4
We first explain how we select the 21 measurements. Due to GPU resource and time constraints, we obtained total of 228 GPU measurements across varying experimental settings, including 6 different numbers of activated experts per token (K); 2 draft lengths (7), and 19 batch sizes (B). These measurements are sorted first by K, then by y within each K group, and finally by B within each 7) combination, forming the total dataframe (df). We then uniformly sampled measurements from this sorted dataset with fixed stride, namely M df [begin: end: 11]. This sampling strategy enables our selected measurements to contain different settings, making the modeling more robust.
The SD speedup function (namely, ComputeSpeeduu defined in line 3 of Algorithm 1) is nonlinear. To optimize its MSE, we employed the scipy. scipy.optinns wpt.pppe..e.....m.mm wwwastessuarreeeeeeeeetesesseeee function with the Trust Region Reflective (TRR) algorithm. TRR is optimization method for bound-constrainet nonlinear least squares problems that combines trust region methods with reflection techniques. It constructs quadratic models within trust regions and uses reflection strategies near boundaries maintain feasibility while ensuring convergence. The fitting process for these 21 data points is efficient, completing in approximately 0.114 seconds. Our modeling incorporates 10 parameters requiring relaxation, with their search boundaries specified as follows:
bias: It represents the time required to load the dense parameters of the target model. We de- note the model's non-FFN parameter count as Vdense- Consequently. the theoretical minimum Vaenas xbitwidth
loading time can be calculated as biasmin = relaxation range, we set biasmaz = 5 x biasmin-
poal. memory handwidth- For the upper bound of the
k/: It adjusts the intensity of the roofline effect of dense components. It should be larger than 0 to ensure the execution time increases as the token count increases. We don't 't set a definite upper limit for k1, as its value is affected by other parameters. Given the hardware with fixed arithmetic units, the execution time grows lincarly with the token count in the compute-boune regime. As shown in line 6 of Algorithm 1, k1 appears as coefficient in the term k1 Gtt; X, s), whose gradient in the compute- bound regime is k1 In(s) s ^APP, As approaches 1, k1 needs to continuously increase to counterbalance in(s) that approaches 0.
k2: It represents the time required to load one expert. Given target model, we denote the parameter count per expert as Veap. Consequently, the theoretical minimum loading time can be calculated as k2min peak memory bundvidtfi"
Vesp xbitwidth
For the upper bound of the relaxation range, we
set k2
5 X K2minn k3: It adjusts the intensity of the roofline effect of sparse components. Similar to k/, we set k3min = 0 and k3 inf.
draft_ bias: It represents the time required to load the dense draft model. We denote the draft model's parameter count as Vdraft- Consequently, the theoretical minimum loading time can Vara fa biiwiddth
be calculated as draft bias min pook memory bundwidth For the upper bound of the relaxation
range, we set draft_ bias = 5 x draft bias biasmin- min.
draft_ k: It adjusts the intensity of the roofline effect of the dense draft model. Similar to kJ, we set d knnnn = O and drafttkkma inf.
24

---

reject bias: It represents the fixed overhead when performing rejection sampling, Vilm reports its elapsed time during SD, and we denote the maximum across measurements Troj. We then set reject fustt_bassmii
and Sifect_bis,,ue max = Traj-
reject_k: It represents the incremental processing time in rejection sampling as the input token count increases. For simplicity, reject_bias
reject_k 0 and reject_
Trej just like
X: It represents the ratio of the empirical ridge point to the theoretical ridge point. Since memory bandwidth is typically less utilized than arithmetic units, set Xmin 0.2 and Amar 1.
8: It adjusts the growing rate of execution time input token count increases. Since & serves as the base of G(t), it must exceed to ensure monotonic growth. However, & should not be too large, as would result in an excessively steep growth rate. In experiments, we set Smin 1 and 8 maa 2.
C,3 Exploration of Alternative Measurement Selection
In this section, we demonstrate the impact of varying the number (m) of measurements used for fitting the modeling results. Given that our model incorporates 10 parameters, a minimum of 10 profiling data points (m 2 10) are required to determine all parameters. We present the modeling fitting with m ranging from 10 to 228. The data selection method follows the stride-based approach described in the previous section, specifically M df [begin: end: stride]. Measurement count m and stride satisfy the following relation:
[228/ stride].
We present the MSE values of different ms and their corresponding fitting figures in Table 3 list the distinct batch sizes involved in the selected measurements, which helps explain why some configurations show inferior model fit. Due to integer division constraints. ms are not continuous at larger magnitudes. Generally, the modeling fits well with the real measurements, except for m 10, 12, 13. The reasons are as follows. When m 10, the number of measurements equals the parameter count, resulting in insufficient data for robust fitting. When m = 12 and m 13, the distribution of the measurement data biased. With stride-based selection, measurements at 12 12 does not include
We also
and = 13 demonstrate notable gaps in batch size coverage (specifically, batch sizes greater than 40, while m 13 does not include batch sizes within 1~224). Their MSE values are larger than that of m 11. despite the latter containing fewer data points for fitting. Based on this analysis, we recommend prioritizing uniform data distribution when selecting measurements, as this approach enables the development of more reliable models even with smaller datasets.
25

---

Table 3
m
stride
MSE
Figure
Batch Size Involved
25
2.216
[1, 12, 16, 20, 36, 40, 44, 60, 80, 100]
22
1.764
[1.4, 16, 20, 28, 32, 40. 44, 56, 100]
12
20
4.288
[1,2,4,8
16, 20, 28, 32, 36, 40]
13
18
2.681
21, 28, 32, 36, 40, 44, 48, 52, 56, 60, 80, 100]
14
17
2.041
[1,2, 16, 24, 32, 40, 44, 48, 52, 56, 60, 80, 100]
15
16
1.668
13
16, 24, 28, 36, 40, 48, 52, 56, 60, 80, 100)
16
15
1.508
[1,2,4,8, 16, 16, 20, 24, 36, 40, 48, 52, 56, 60, 80, 100]
14
1.563
24, 24, 11, 32, 40, 44, 48, 52, 56, 60, 80, 100]
13
1.525
[1.2.4.8. 24. 32, 36, 44, 48, 52, 56, 60, 80, 100]
19
12
2.080
11,2,4.8. 16, 20, 28,
40, 44, 52, 56, 60, 80, 100]
21
11
1.679
18
(1.2.4.8. 16. 20, 24, 32, 40, 48, 52, 52, 80, 100]
23
10
1.800
19
[1,2,4,8, 16, 20, 24, 32,
80, 100]
26
9
1.716
20
[1,2,4,8, 16, 20, 28, 36, 44, 44,
100]
29
8
1.524
21
[1,2,4,8, 16, 20, 24, 28, 32, 36, 40, 44,
100)
33
7
1,526
22
81,2,4.8, 16, 20, 24, 28, 32, 36, 44,
100]
38
1.715
23
[1,2,4.8, 16, 20, 24, 32, 36,
46
5
1.644
24
[1,2,4,8, 16, 20, 24,
32,
57
4
1.509
25
[1.2.4, 16, 20, 24, 28, 32, 40,
76
3
1.553
26
[1 16, 20, 24, 32, 36, 40, 44,
114
2
1.485
27
[. 8,
12, 16, 20, 24, 28, 32, 36, 40, 44, 600
228
1
1.523
28
[1,2,4, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 56,
p y
2, y
Mode
1.5
Hoddl
Vodklngg
Modeiig
Speedhi
Spucedao
2.22
4-4
60
160
60
100
Butch Sixe
Balch 222 (8)
5U00 226 (5))
S1tch slachslee0
4/04, y
6- 8, 3/64,
E- 4154,
K- 2, 8854 4
reuuts
Spocebap
1.3
sie
Batch
Buoch.a
stox
X= 16/04.
32154,
1.5
32/04.
CFU roaula Hode/
GPU rosella Model no
2.5
GPO rpolll
mg
970 nosta
Voselrs
Spoccuup
1.2
2.-
3.0
60
100
Figure 8: Comparison between GPU results and modeling with 10 measurements.
26

---

ym
CTU rouila Mideiin
15
GrU Modeinn
3nUrodd Vodei
Voselns
Specsup
Spoccup
Spersuu
1.0
45
80
126
40
60
60
100
20
46
60
80
160
25
46
80
320
siox
Batch-924
p. A/61,
K: 6, 8361, y
4, 6/44.
R, 0164,
Moddiing
2.8
Hodil
Modeling
1.
1.0
60
1643
60
100
Butch Sixe
Balch 2:2 (8)
Dutch 222 (59)
S0tch sizc
X- 20, p= 15/54, y
K- 32, 52154, y
K- 16, 20164, y
X- 22, 22............ 33/54, y
GFU renutt
2.0
medtts
1.0
2.5
1.0
1.
Batch size
Batch
Buoch
Sattch sioe
Figure 9: Comparison between GPU results and modeling with 11 measurements.
cPU rouls Hodsinng
grdrpds Vedaing
99 rosts Veselns
Speenn
Spccuaap
Speren
2.0
1.0
Mooiding
GFJ
45
60
00
120
40
60
00
100
20
40
20
20
100
25
40
00
320
sibe
sine
4/64,
y
1/64, y
p y
p.p
Hudld
1.5
Veaking
Modeing
2.4
2.0
1.2
12
1.0
60
130
60
20
160
20
60
320
Bitth Size
B31ch 922 (8)
5ooch 5220 (5)
Sitch sith
C= 26. p= 18/66, 2
K 32; p- 52:54, y
K-n 16, 16/64 y
E= 22. 2 32/60. y
rauuta
roauta
2.5
russa
Spccodop
Spccdda
2.0
Speeddrs
1.0
Bltch slxc
Bulch
Buoch
(5)
50tth sloc
Figure 10: Comparison between GPU results and modeling with 12 measurements.
27

---

C/U nasts Noselig
GFV nou Viddeing
Vedding
Voselis
Spersup
Spociiue
C
Spenttus
Spocciup
1.0
20
103
80
110
66
60
00
103
23
40
80
100
Butch
Butch
1055. y
K: p. EIEA, V
0..pp
1/54, V
2.2
Vodding
Noseling
Miodaing
Moceiing
1.6
Spoodis
Sceeddi
Speedio
1.50
162
100
102
110
B1 20
300c13ie (5)
Butchsae (8)
B40ch 222 (5)
<- 26, p= 18154, v
X 32,
22704, y
K. 26,
1814. v
K- 32,
32/04, y
3.0
resute
ma
1.0
2.5
Secudas
Spcuoko
1.5
1.0
Batch
Batchsiiz
Batth
Butch (25)
Figure 11: Comparison between GPU results and modeling with 13 measurements.
und reura Vodsing
0/4 nosts Noselts
ard Vodeing
188
UPU rosts Veselng
Speedn
Spocouu
Speroaup
Specodape
2.4
1.2
1.0
20
60
02
100
22
20
20
110
46
60
00
100
22
40
90
80
100
Batch uRD
size
Batch
x
4/54,
BIAA 2
4164,
S. 2/54, y
Vedlingg
Mossinng
12
Pioddnng
Moreingg
Spoutup
Spcodia
1.2
1.0
302
120
100
100
B1 20
1oze (8)
Butch 5i2e (6)
800ch 922 (5)
K-n 266 p. 1815., y
X 52,
32/64, y
K- 26, p 18194 y
K= 32, p 32/64, y
resuica
2.5
naalaa
2.5
rasu
5.0
GnU rasla
2.00
2.5
Speedda
Spreebip
Spoodhe
2.0
2.5
1.00
1.0
B-
Butch
B4tch
(5)
Figure 12: Comparison between GPU results and modeling with 14 measurements.
28

---

y
y- 3hUrodd Videiig
1.C
CTU rouila Midsiing
Modeinn
Voselis
Spareip
Spoucuup
Spertiup
1:22
2.1
12-
22
45
80
120
40
60
66
100
20
46
60
80
100
25
46
80
320
siox
Batch 922
4/61, yn
K: 6, 8.61, y
4, 4/44.
0. 0164, y
1.2
Hodsiing
1.8
Hodil
3.8
Vedalnng
Modeling
1.4
1.2
12-
1.0
1.0
60
160
20
60
100
Butch Sixe
Balch 222 (8)
Butch 222 (55
S0tch sizc
A- 20, p- 15/54, y
K- 32, 52154, y
K- 10,
20164, y
X- 22, 22...2.2..... 33/54, y
GFU renutt
madtts
renul
23
1.0
2.0
Sponddes
2.0
1.
Batch stze
Batch
Buoch
Sattch stoe
Figure 13: Comparison between GPU results and modeling with 15 measurements.
oru rou Hodsing
1.4
OPU Hodelng
gPUrpds Vedaing
99 rosts Veselns
Speren
Spccudap
2,4
Speren
22
45
60
00
120
40
60
60
100
20
40
20
20
100
25
40
80
03
320
size
Datcs
sine
4/04, y
E. 5/64, y
p 4134, y
p.p 4
Moddiing
1.A
Veokiigg
Modeing
Spcciiip
1.4
1.2
2.2
12
1.0
22
60
130
60
20
160
20
00
320
B1tth Size
B31ch S22 (8)
5o0ch 9:20 (50
Sitch size
<= 26. p= 18/66 -
K 32, p 52:64, y
K- 16, 16/64, y
E= (2.32, 2- 33/64. y
CFU rauuta
roauts
2.5
rassa
Spccobee
Spccddap
2.0
Speeddrs
3.2
1.0
Bltch sloc
Bulch
Buoch
(55
50tch-to sioc
Figure 14: Comparison between GPU results and modeling with 16 measurements.
29

---

y
y 3hUnndd Vodei
CTU reaala Modeinn
14
Gro Hodeinn
Voselis
Spaucuu
2,4
Spertiup
12-
20
45
80
126
40
60
60
100
20
46
60
80
100
25
46
80
320
stox
Batch
sloe
A/61,
K: 6,
8.61, y
4, 4/44.
R, 0164, y
Moduiing
2.40
Hodid
2.0
Vodaling
Modeling
1.0
122
1.0
20
60
1603
60
100
Butch Sixe
Bulch 2I2 (8)
Butch 222 (55
S0tch sizc
A- 20, p= 16/54, y
X- 32, 52154, y
K- 10, p 20164, y
X- 22, 22..2-2..... 33/54, y
GFU renutt
2.0
mentts
2.
renul
1.0
Spocoke
2.0
Sponddes
2.0
1.0
Batch stze
Batch
Buoch
Sattch stoe
Figure 15: Comparison between GPU results and modeling with 17 measurements.
CPU roulb Hodsinng
oPU Hodelng
gPUrpds Vedaing
99 rosts
1.4
Vedelns
Speedi
Spccudag
Spocdaup
2.0
1.0
45
60
00
120
40
60
60
100
20
40
20
20
100
25
40
80
03
320
Batch sibe
Datcs
sine
4/04, y
5/64, y
p 4134, y
p.p v
Moddiing
2.5
Hudld
1.5
Veakiigg
Modeing
Spercin
2.4
1.4
2.0
2.2
12
1.0
22
60
130
60
20
160
20
00
320
Bitth Size
B31ch S22 (8)
5ooch 5220 (50
Sitch 5ize
E= 26. p= 18/66 -
K 32, p= 52:64, y
K- 16, p 16/64, y
E= 22, 2- 33/64. y
rauuta
roauta
2.5
rassa
1.0
Spccobop
Spccddap
2.0
Speeddrs
1.0
Bltch sloc
Bulth%
Buoch
(55
50tch sioc
Figure 16: Comparison between GPU results and modeling with 18 measurements.
30

---

ym
CTU rauila Mideiino
cro Hodeinn
3hUrodd Videi
Voselns
--4
Spersuu
Spocccup
Spensuu
20
60
80
120
40
60
66
100
20
46
60
80
100
26
46
80
320
siox
Batch
4/65,
K: 6,
8.61, y
K: 4, 4/44.
0.p 0164, y
1.4
1.5
Hodsiing
1.8
Hodil
Modeling
Speadii
1.4
1.0
1.0
60
1:00
1642
60
100
Butch Sixe
Balch 222 (8)
Dutch 222 (55)
S0tch sizc
X- 20, p- 15/54, y
K- 32, 52154, y
K- 16, p 26164, y
X- 22, 22..2..22.... 33/54, y
GFU renutt
2.0
madtt
2.5
renul
1.0
2.0
1.0
Batch stze
Batch
Buoch
Satich stoe
Figure 17: Comparison between GPU results and modeling with 19 measurements.
reuta Vodsing
0/ rosts Noselns
ard Vodeing
188
UPU rosts Veselng
Spacen
Spocoou
Spereiup
Specodate
1.2
1.0
20
60
02
100
22
20
20
110
46
60
00
100
40
90
80
100
Batch uRD
Batch
4/54, y
BIAA, 2
4164,
y
S. 2/54, y
Veddingg
Mossinng
1.4
Miodding
Moreing
Spoutuu
Spcodaa
13-
1.0
102
100
103
40
100
5 20
1oze (8)
Suich-520 (6)
800ch 922 (5)
K-n 266 o- 1815. y
X 52,
32/64, y
E- 26, p 18194 w.
K= 32, p 52/64, y
1.0
resuic
naala
rasu
Gn rasla
2.00
Speedds
Spreebup
Spccdboo
1.25
2.5
1.0
B-
Butch
B4tch (5)
Figure 18: Comparison between GPU results and modeling with 21 measurements.
31

---

y
y- 3hUrodd Videiig
CTU roaala Modeinng
1.6
GrU Hodeinn
Voselis
Sparcip
Speuccup
Spertiup
2.2
12-
1.0
45
60
80
130
40
60
60
100
20
46
60
80
100
25
46
80
320
siox
Batch
4,
A/65,
6,
8,61, y
p. 6/44.
R, 0164,
1.2
Hodeiing
1.5
Hodid
3.8
Vedalnng
Modeling
1.4
2.0
12
1.0
1.0
20
60
1602
60
100
Butch Sixe
Balch 242 (8)
Butch 222 (59)
S0tch sizc
X- 20, p- 15/54, y
K- 32, 52154, y
K- 10, 20164, y
X- 22..22..2220 33/54, y
GFU renutt
medtts
23
1.0
2.0
Sponddes
2.0
1.0
Batch stze
Batch
Buoch
Sattch stoe
Figure 19: Comparison between GPU results and modeling with 23 measurements.
CPU rouls Hodeiing
1.5
oPUr Hodeln
3P gPUrpds Vedeing
99 rosts Veselns
Speenn
Spocuaap
Speeen
2.0
1.0
45
60
00
120
40
60
60
100
20
40
20
20
100
25
40
00
320
sibe
Datcs
sine
4/04,
y
1.4
K- E. 5/64, y
p 4/34, y
p.p
Mosdiingg
3.5
Huddd
Veakiing
Modeing
Spcceii
1.4
12
2.2
1.0
22
60
130
60
20
160
20
00
120
Bitth Size
B31ch S22 (8)
500ch 522 (50)
Sitch size
X= 26. p- 18/66 2
K 32, p= 52:64, y
K- 16, 16/64, y
E= 32, 2- 33/64. y
rauuta
roauts
255
rassa
Spccobee
Spccddo
2.0
Speeddrs
3.2
1.0
Bltch sloc
Bulcc
B40ch
(5)
50tch sioc
Figure 20: Comparison between GPU results and modeling with 26 measurements.
32

---

y
CTU rauila Modeiing
Gru Modeinn
3hUrodd Videiig
Voselis
Specsup
Spoccup
Spertiup
1.1
12-
1.0
20
60
80
120
40
60
66
100
20
46
60
80
100
25
46
80
32.
siox
Batch
p. A/61,
K: 6, 8.61, y
p 4/t4.
0. 0164,
Hodsiing
1.6-
Hodid
Vedal
Modeling
2.4
12-
1.0
1.0
20
60
1602
60
100
Butch Sixe
Balch 222 (8)
Butch 222 (55)
S0tch sizc
A- 20, p= 15/54, y
K- 32, 52154, y
K- 10, 20164, y
X- 22, 22..222...... 33/54, y
GFU renuts
2.0
medtts
23
renul
1.0
2.0
Sponddes
1.0
Batch size
Batch
Buoch
Sattch stoe
Figure 21: Comparison between GPU results and modeling with 29 measurements.
CPU roul Hodsingg
1.6
OPU Hodelng
(PU gPurpds Vedelng
99 rosts Veselns
Sparen
Spcuuaap
Speren
2.2
2.0
22
45
60
00
120
40
60
60
100
20
40
20
20
100
25
46
80
00
320
Batch size
Datcs
sine
4/04, ym
K= E. 5/64, y
p 4134, y
p.p 4
1.5
Moddiing
1.5
Hudld
1.
Veokiig
Modeing
Spociiup
1.2
2.2
12
1.0
22
60
130
60
20
160
20
00
320
B1tth Size
B31ch S22 (8)
500ch 522 (50
Sitch 5ize
E= 26. p- 18/66 m
K 32, p 52:64, y
K- 16, p 16/64, y
E= 22, 2- 33/64. y
CPU rauuta
2.0
roauia
2.5
rassa
Spccodeo
Spccddap
2.0
Speeddrs
3.2
1.0
Bltch slxc
Bulch
Buoch
(5)
50tchstog sioc
Figure 22: Comparison between GPU results and modeling with 33 measurements.
33

---

y
CTU rasula Mi-s
3hUrodd Videiig
Modeinn
Vodelin
Sparsip
Spoucuup
Spertiup
2.1
12-
80
120
40
60
66
100
20
46
60
80
100
25
46
80
320
stox
Batch 922
stoe
A/61, yn
K: 6, 8.61. y
p 4/44.
R. 0164,
1.4
Hodsiing
2.8
Hodil
3.8
Modeling
1.4
1.0
1:2
1.0
1.0
60
160
60
100
Butch Sixe
Balc AI2 (8)
Butch 222 (55)
S0tch sizc
A- 20, p- 15/54, y
K- 32, 52154, y
K- 10,
20164, y
X- 22, 22...-.2.... 33/54, y
GFU renutt
2.0
mautt
2.
renul
1.0
Spocoko
2.0
Sponddes
2.0
1.0
Batch size
Batch
Buoch
Sattch stoe
Figure 23: Comparison between GPU results and modeling with 38 measurements.
oru roub
1.5
OPU Hodeln
3PU gPurpds Vedeing
99 rosts Veselns
Hodsing
Sparen
Spccuaap
Speren
2.2
22
45
60
00
120
40
60
60
100
20
40
20
20
100
25
40
80
00
320
size
Datcs
sine
4/04,
y
1.5
5/64, y
p 4134, y
p.p v
Moddiing
1.5
Hudld
2.5
Veokiigg
Modeing
Spcciiup
2.2
2.2
12
1.0
22
60
130
60
20
160
20
00
320
B1tth Size
B31ch S22 (8)
500ch 5220 (50
Sitch 1ith-iee.0
X= 26. p- 18166 y
K 32, p= 52:64, y
K- 16, 16/64, y
X= 22, 2- 33/64. y
rauuta
2.0
roauta
2.5
rassa
1.8
Spccobee
2.5
Spccddo
2.0
Speeddrs
2.2
1.0
Bltch slxc
Bulch
Buoch
(5)
S0tch sizc
Figure 24: Comparison between GPU results and modeling with 46 measurements.
34

---

ym
y
CTU rauila Mideinn
GTV GrU Modeinn
3hUrodd Videi
Voselis
Specsip
Spoccuup
Spentiup
12-
1.0
20
80
120
40
60
66
100
20
46
60
80
100
25
40
320
siox
Batch 922
p. A/65,
K: 6, 8.61, y
p 4/t4.
R, 0164,
Hoddiing
2.8
Hodil
3.8
Modeling
12
1.0
1.0
60
1:00
1692
60
100
Butch Sixe
Bulch 222 (8)
Butch 222 (55)
S0tch sizc
X- 20, p- 15/54, y
K- 32, 52154, y
K- 10, 20164, y
X- 22, 2-...-.2...... 33/54, y
GFU renutt
2.0
medtts
23
renul
1.0
2.0
Sponddes
1.0
Batch size
Batch
Buoch
Satich stoe
Figure 25: Comparison between GPU results and modeling with 57 measurements.
crU rouls Hodsinng
1.5
Hodil
(PU gPurpds Vedeing
99 rosts Veselng
1,4
Spaeen
Spocuaap
Speren
2.2
2.0
45
60
00
120
40
60
60
100
20
40
20
20
100
25
40
80
03
320
sibe
Datcs
sine
4/04,
y
K= E. 5/64, y
p 4134, y
p.p v
Moddiing
2.6
Veakiigg
Modeing
Spcciiip
1.4
2.6
1.2
12
1.2
22
60
130
60
3.0
20
160
1.0
20
00
320
Bitth Size
B31ch S22 (8)
500ch 522 (50
Sitch 5ize
X= 26. p- 18/66 y
K 32, p- 52:64, y
K- 16, 16/64, y
X= 22, 2- 33/64. y
rauuta
7.0
reauts
2.5
rassa
2.8
Seccobop
Spccddo
2.0
Speeddrs
1.0
Bltch slxc
Bulch
Buoch
(5)
50tch sioc
Figure 26: Comparison between GPU results and modeling with 76 measurements.
35

---

ym
y
CTU rauila Modeiin
GTV GrU
3hUrodd Videing
Modeinn
Voselis
Specsup
Spoccuue
Spertiup
12-
80
120
40
60
66
100
20
46
60
80
100
25
46
80
32.
siox
Batch
p. A/61,
K: p 8.61, y
p 4/44.
R. 0164,
Hodsiing
2.5
Hodil
3.8
Veddlnng
Modeling
1.2
1.0
3.0
1.0
60
164
60
100
Butch sixe
Bulch 222 (8)
Butch 226 (55
S0tch sizc
A- 20, p- 15/54, y
K- 32, 52154, y
K- 10,
20164, y
X- 22..22..2220 33/54, y
GFU renutt
2.0
medtts
2.5
renul
1.8
Spocoke
2.0
Spondees
1.0
Batch size
Batch
Buoch
Sattch sioe
Figure 27: Comparison between GPU results and modeling with 114 measurements.
cPU rouls Hodsinng
1.3
oPUr Hodelng
3P gPUrpds Vedaing
99 rosts Veselng
2.4
Speeen
Spocuaap
Speren
122
2.0
45
60
00
120
40
60
60
100
20
40
20
20
100
25
40
80
03
320
sibe
Datcs
sine
4/04,
y
K= E. 5/64, y
p 4134, y
p.p v
Moddiing
2.6
3.
Veakiing
Modeing
Spcciiip
1.4
1.2
12
1.2
1.0
22
60
1:0
60
20
160
20
00
320
B1tth Size
B31ch S22 (8)
5000h 522 (50
Sitch 5i0e
X= 26. p- 18/66 y
K 32, p- 52:64, y
K- 16, 16/64, y
X= 22, 2. 33/64. y
rauuta
7.0
reauts
255
rassa
2.8
Spccobo
Spccddo
2.0
Speeddrs
1.0
Bltch slxc
B40ch
(5)
50tch-10 sloc
Figure 28: Comparison between GPU results and modeling with 228 measurements.
36


Reviews:
[{"review_id": "4ZFToApZC4", "reviewer": "Reviewer_jdmT", "summary": "This paper investigates the performance of speculative decoding for Mixture-of-Experts (MoE) models. Specifically, the paper shows that speculative decoding is not as helpful for sparse MoE models as it is for dense models for small batch sizes, when the batch size grows, the benefit substantially increases (which is counter to the trend for dense models). To support these claims, the paper provides theoretical analysis that shows that at moderate batch sizes, the number of activated experts saturates, which helps amortize the additional expert loading costs (in terms of memory bandwidth), and allows for more compute-bound inference. The paper also introduces a new intuitive metric -- target efficiency -- that measures the cost ratio for computing logits for 1 token vs. $\\gamma$ tokens (in previous speculative decoding work, this is assumed to be near 1, which is not true for MoE models). Finally, the paper empirically shows the benefits of larger batch sizes on speculative decoding for MoEs with real models.", "questions": "- Is there any way to predict what the optimal batch size might be based on your theory, vs through hyper-parameter tuning?\n- The comparison to dense models in Figure 2c is insightful but indirect, as it only shows target efficiency.  To make your claim that MoEs can benefit more from SD than dense models more concrete, could you provide the end-to-end speedup graph for the dense model, analogous to Figures 2a and 2b?\n- The performance curves in your figures show noticeable fluctuations. While you mention averaging results over five runs, can you also add error bars to Figures 2 and 3?\n- Given that MoEs require larger batch sizes, what is the effect on latency, vs throughput?", "limitations": "None", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is clearly written and presents a simple finding: sparse MoE models benefit more from speculative decoding under moderate batch sizes. The idea that the acceptance rate is not the only important measurement --- target efficiency is also very important, especially when the assumptions that it is near 1 is violated --- is also a natural additional metric to look at. This could potentially be a valuable finding for practitioners, if it was not already known.\n\nThat said, while the paper is well-executed, it is not particularly novel. The core finding, that speculative decoding benefits from instances of memory-bound workloads in MoE models, can be seen as a logical extension of established performance principles. The theoretical analysis, while effective, supports this by applying existing concepts like the roofline model and probability theory, rather than providing new theoretical constructs. There are also some minor gaps in the experimental presentation; the comparison to dense models is indirect, using the \"target efficiency\" metric instead of a direct end-to-end speedup analysis, and the plots lack error bars, making it somewhat difficult to assess the statistical significance of the results.", "was_revised": true, "final_justification": "The paper is well done and the authors adequately addressed my concerns. However, I still believe the novelty is fairly limited, so I am recommending a 4 instead of a 5.", "rebuttal": "We sincerely appreciate your careful reading and valuable suggestions for improving our manuscript. In the following response, we first answer your four questions with additional experimental results as supporting evidence, and then respond to the weaknesses you mentioned. We hope these revisions adequately address your concerns.\n\n# Question 1: Is there any way to predict what the optimal batch size might be based on your theory, vs through hyper-parameter tuning?\n\nPredicting speedup based purely on theoretical analysis without using any profiling data is challenging. On GPUs, GEMM is indeed predictable because it involves regular and highly optimized operators. However, this becomes difficult for operators like Attention that involve customized kernel optimizations (FlashAttention1/2, eager attention, SDPA attention), fusion (of different types of nonlinear layers or positional encodings like RoPE and its variants), and multi-hierarchy memory access (registers, caches, DRAM), as pointed out by previous works [1,2,3]. Consequently, other system optimization works like NanoFlow also adopt a two-stage strategy: profiling first, then runtime execution.\n\nWe take profiling results of Qwen2-57B-A14B (hidden size 3584) and Mixtral-8x7B (hidden size 4096) as an example to illustrate this. For FFN, Qwen takes a shorter time than Mixtral (143μs vs 226μs), aligning with their relative hidden sizes. However, for Attention, Qwen takes a longer time than Mixtral (271μs vs 115μs), contradicting the theoretical expectation based on hidden size scaling.\n\nTherefore, modeling speedup trends with pure analytical methods requires case-by-case analysis for different operator implementations, GPU microarchitectures, and instruction sets. Moreover, many hardware details are not disclosed by GPU vendors, making such predictions even less practical. In contrast, the hyperparameter approach offers a more general paradigm and is easy to use: all parameters have clear physical meanings, only a few profiling data points are needed, and the computational overhead is very low. Our method achieves a balance between effectiveness and practicality: on one hand, it captures the primary factors (i.e. the number of activated experts and roofline trends); on the other hand, it avoids getting lost in system implementation details.\n\n# Question 2: End-to-end speculative decoding speedup comparsion between MoE and dense models.\n\nWe compare target efficiency in Figure 2c in order to isolate the effect of acceptance rate. Given that the strong correlation between target efficiency and speedup has been demonstrated in Figures 2a and 2b, we believe this comparison provides a reliable assessment.\n\nWe also have end-to-end (E2E) speedup comparisons, as shown in the table below. As batch sizes increase (>= 16), speculative decoding (SD) indeed achieves greater speedup on MoE models compared to dense models (i.e., the last row exceeds 1). We appreciate the reviewer's constructive feedback, which helps to make our statements more direct and concrete. We will attach these E2E speedup results in the revised paper.\n\n**Table 1 Comparison of E2E SD speedup between MoE and dense models (corresponding to Figure 2c)**\n| batch size|1|2|4|8|12| 16| 20| 24| 28| 32| 36| 40| 44| 48| 52| 56| 60| 80|\n|--|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| SD speedup for dense model|3.16|2.84|2.57|2.30|2.13|1.83|1.69|1.56|1.44|1.37|1.35|1.23|1.19|1.14|1.04|1.02|1.06|0.96 |\n| SD speedup for moe model|1.60|1.73|1.91|2.18|2.07|1.85|1.88|1.81|1.80|1.97|1.77|1.44|1.45|1.38|1.26|1.27|1.28|1.09 |\n| $\\frac{\\text{SD speedup for moe model}}{\\text{SD speedup for dense model}}$|0.51|0.61|0.74|0.95|0.97|**1.01**|**1.12**|**1.16**|**1.25**|**1.44**|**1.32**|**1.18**|**1.22**|**1.21**|**1.21**|**1.25**|**1.20**|**1.13** |\n\n\n# Question 3: Addition of error bars.\n\nThe table below presents the speedup of five independent runs and their average for Figure 2a. The variance in speedup is minimal, confirming the **statistical significance** of our findings. Theoretically, the speedup should also be stable since we fixed the random seed to ensure identical workloads across all runs. Your advice is very helpful, and we will add complete error bars in the revised paper.\n\n**Table 2 Speedup of five independent runs and their average**\n| batch size|1|2|4|8|12| 16| 20| 24| 28| 32| 36| 40| 44| 48| 52| 56| 60| 80| 100  |\n|--|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| run1 speedup|1.61|1.73|1.91|2.20|2.03|1.89|1.94|1.80|1.77|1.91|1.78|1.43|1.40|1.43|1.33|1.20|1.24|1.07|0.90 |\n| run2 speedup|1.56|1.74|1.90|2.19|2.09|1.86|1.86|1.83|1.83|2.00|1.74|1.43|1.43|1.39|1.23|1.26|1.26|1.07|1.02 |\n| run3 speedup|1.61|1.73|1.90|2.20|2.07|1.87|1.79|1.91|1.82|2.00|1.81|1.46|1.50|1.31|1.23|1.32|1.27|1.07|1.00 |\n| run4 speedup|1.61|1.74|1.90|2.16|2.13|1.82|1.86|1.74|1.82|1.95|1.79|1.43|1.48|1.34|1.24|1.27|1.32|1.08|1.02 |\n| run5 speedup|1.59|1.73|1.92|2.15|2.04|1.82|1.98|1.79|1.77|2.00|1.75|1.46|1.47|1.44|1.30|1.29|1.32|1.17|0.98 |\n| speedup std |0.019|0.005|0.006|0.020|0.036|0.029|0.064|0.053|0.025|0.035|0.026|0.013|0.038|0.049|0.042|0.040|0.028|0.038|0.042 |\n\nRegarding the fluctuations in performance curve, they are attributed to GPU's *dimension quantization effects*: tensor core utilization changes with matrix dimensions. When dimensions are not divisible by the GPU's native tile sizes, computation performance degrades. This phenomenon is well-documented in NVIDIA's document [4]. Auto-regressive (AR) decoding is more sensitive to this effect, making the time ratio of AR to SD (namely, SD speedup) fluctuate. Despite these local fluctuations, the overall speedup trend follows our theoretical analysis, confirming the validity of our conclusions.  \n\n# Question 4: Latency and throughput tradeoff.\n\nThis question reveals a core difference between MoE and dense models when applying SD. Our findings actually demonstrate that for MoE models, latency and throughput can be jointly optimized, while this is hard for dense models.\n\nFor dense models, there exists **a trade-off between low latency and high throughput**: smaller batch sizes yield higher SD speedups (low latency but limited throughput), while larger batch sizes result in lower SD speedup (increased throughput but high latency). However, for MoEs, speedup can **increase with** batch size, indicating both latency and throughput are improved. Our work reveals the underlying cause of this *counter-intuitive* phenomenon and further extends it to MoEs with different sparsities.\n\nYour insightful question provides a valuable perspective that helps readers understand our work from a higher level. We will incorporate this into our revised paper.\n\n# Clarification of the novelty and impact of our work.\n\n\nWhile our work indeed builds upon established principles, we would like to respectfully highlight several aspects that demonstrate novelty and impact.\n\n**Correcting Critical Misconceptions with Quantitative Methodology:** Our work challenges two prevalent beliefs that have constrained SD research as follows:\n\n* SD is more suitable for dense models than MoEs [6,7]. → We demonstrate that at moderate batch sizes, SD is more effective for sparse MoE models.\n* SD performance degrades with larger batch sizes [8,9,10] → We demonstrate that for MoE, effectiveness may actually increase with growing batch size.\n\nOur work is the **first** to systematically refute these beliefs, addressing an important gap in current research. Beyond only *qualitatively* describing SD speedup trends as previous works have done [5,6,10], we further developed a *quantitative* model making SD more explainable, which is also **a first attempt** in SD research. Our theoretical analyses align well with experimental results, providing rigorous support for our conclusions.\n\n**Practical Impact on MoE Optimization:** Beyond theoretical findings, our work has practical impact on MoE optimization. Compared to dense models, MoEs are efficient at both small batches (fewer activated parameters) and large batches (abundant computation), but **suboptimal** at medium batches where full parameters are loaded without sufficient computation to amortize this overhead. Other MoE optimizations such as expert offloading and caching are also ineffective in this regime due to fully activated experts. In this paper, we show that SD—which has been overlooked by previous works for MoE—is effective, thus providing **a novel perspective** on MoE's deficiency at moderate batch sizes.\n\nIn summary, though based on existing theories, we have developed **novel conclusions** that help correct misconceptions and optimize MoE models, supported by a quantitative methodology unprecedented in previous SD works. Other reviewers such as kQQx, Xih7 and 8fAG also acknowledge the value of our work on this timely topic of MoE and SD.\n\n\n# Reference\n\n[1] NanoFlow: Towards Optimal Large Language Model Serving Throughput.\n\n[2] Orion: Interference-aware, fine-grained gpu sharing for ml applications.\n\n[3] Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking.\n\n[4] Since links are not allowed in rebuttal, please search for \"Matrix Multiplication Background User's Guide NVIDIA\".\n\n[5] MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding\n\n[6] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.\n\n[7] Utility-Driven Speculative Decoding for Mixture-of-Experts.\n\n[8] Optimizing speculative decoding for serving large language models using goodput.\n\n[9] The synergy of speculative decoding and batching in serving large language models.\n\n[10] Specinfer: Accelerating generative llm serving with speculative inference and token tree verification."}, {"review_id": "nwfMVBEd1y", "reviewer": "Reviewer_8fAG", "summary": "This paper applies speculative decoding (SD) to sparse Mixture-of-Experts (MoE) models for inference acceleration. The authors demonstrate that the combination yields higher speedups at moderate batch sizes. The paper introduces a new metric called target efficiency for speedup measurment in MoE models. Evaluation results using Qwen2-57B-A17B-Instruct as verifier and Qwen2-0.5B-Instruct as light-weight drafter shows up to 2.29$\\times$ speedup.", "questions": "- The authors states multiple times that they challenge the conventional belief that speculative decoding cannot effectively accelerate MoEs, where the “belief” is more of an untested assumption than a well-established empirical claim. It would be useful to cite relevant prior work on this.\n- To help illustrate the framework, the authors could consider including a system diagram to show the details of MoESD workflow and design?\n- Minor point: It would be helpful to walk-through Algo 1 in the text.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n- The paper studies the niche area at the intersection of MoE and SD, which is a timely topic - MoE brings state-of-the-art generation quality and SD is well-established for speedup at inference time.\n- The paper introduces a new SD performance metric for MoE setting, namely target efficiency, which goes beyond commonly known acceptance rate by measuring how much verifier work is converted into useful tokens.\n- The theoretical modelling seems to be sound.\n\nWeaknesses:\n\n- My main concern is that the evaluation appears to be narrow as only one sparse model was evaluated (although OPT is included for dense model case). Given this, and the standard that NeurIPS holds for submissions, I will have to request\n- The evaluation doesn’t seem to include tree-based speculative decoding e.g. Eagle, Eagle-2, which are at the top of the SD leader-board according to SpecBench (https://github.com/hemingkx/Spec-Bench/blob/main/Leaderboard.md). It would hence be useful to include analysis of such tree-based method.", "was_revised": true, "final_justification": "My concerns have been sufficiently addressed, and I am raising my score to Accept.", "rebuttal": "Thank you sincerely for your constructive feedback! We hope our responses below can address your concerns and clarify any misunderstandings about our work.\n\n# Weakness 1: Model diversity.\n\nWe have added experiments for another MoE: Mixtral-8x7B-Instruct-v0.1. We used Eagle's **speculation head** for Mixtral rather than a standalone draft model to validate our method's generality. Due to the format limitations of this year's rebuttal, we can only present the results in tables as follows. Your advice on model diversity is very valuable and we will add them to the revised paper.\n\nAs shown in tables, experimental results on another MoE also **conform to our theoretical analysis**, where the speculative decoding (SD) speedups on MoE models first increase and then decrease as batch size increases. Since our theoretical analysis places no restrictions on model architectures, this result is as expected and should hold for other MoE models.\n\n**Table 1: The SD speedup trend vs. batch size, under different settings for Mixtral-8x7B-Instruct-v0.1**\n\n|  Batch sizes   | 2    | 4    | 6    | 8    | 10   | 12   | 16   | 20   | 24   | 28   | 32   | 36   | 40   | 60   | 80   | 100  |\n|----|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| Speedup @(humaneval, temperature 0) | 0.90 | 0.96 | 1.19 | 1.04 | 1.71 | 1.67 | 1.43 | 1.31 | 1.21 | 1.18 | 1.14 | 1.11 | 1.15 | 0.97 | 0.80 | 0.72 |\n| Speedup @(humaneval, temperature 1) | 0.91 | 0.91 | 1.07 | 0.99 | 1.37 | 1.38 | 1.16 | 1.13 | 1.01 | 1.06 | 1.07 | 1.05 | 1.04 | 0.88 | 0.71 | 0.54 |\n| Speedup @(mtbench, temperature 0) | 0.84 | 0.86 | 1.00 | 0.98 | 1.34 | 1.28 | 1.23 | 1.22 | 1.14 | 1.12 | 1.07 | 1.06 | 1.04 | 0.95 | 0.88 | 0.81 |\n| Speedup @(mtbench, temperature 1) | 0.85 | 0.80 | 0.96 | 0.93 | 1.15 | 1.17 | 1.13 | 1.14 | 1.09 | 1.05 | 0.99 | 0.97 | 0.92 | 0.86 | 0.75 | 0.69 |\n\n\n**Table 2: The overview of SD speedup on Mixtral-8x7B-Instruct-v0.1**\n\n*Notation: This table follows the format of Table 1 in the original paper. $\\gamma$ represents draft token length, $T_{AR}$ denotes auto-regressive decoding time, $T_{SD}$ denotes speculative decoding time, $\\sigma$ is the ratio of actually generated tokens to theoretical maximum, and $\\mathbf{x}$ denotes maximal speedup across batch sizes.*\n\n| Dataset   | Temperature | $\\gamma=2$  |        |        |        | $\\gamma=3$  |        |        |        | $\\gamma=4$  |        |        |        |\n|-|---|--|-----|-----|-----|--|-----|-----|-----|--|-----|-----|-----|\n|    |      | $T_{AR}$    | $T_{SD}$      | $\\sigma$      | $\\mathbf{x}$  | $T_{AR}$    | $T_{SD}$      | $\\sigma$      | $\\mathbf{x}$  | $T_{AR}$    | $T_{SD}$      | $\\sigma$      | $\\mathbf{x}$  |\n| humaneval | 0.0  | 20.86       | 12.47  | 0.78   | 1.67   | 21.00       | 12.46  | 0.66   | 1.69   | 20.86       | 11.69  | 0.58   | **1.79**   |\n| humaneval | 1.0  | 21.52       | 15.58  | 0.61   | **1.38**   | 21.39       | 16.03  | 0.46   | 1.33   | 21.48       | 16.23  | 0.39   | 1.32   |\n| mtbench   | 0.0  | 21.61       | 16.10  | 0.61   | **1.34**   | 21.61       | 16.43  | 0.46   | 1.32   | 21.36       | 16.89  | 0.39   | 1.26   |\n| mtbench   | 1.0  | 21.33       | 17.70  | 0.53   | **1.21**   | 21.33       | 17.84  | 0.43   | 1.20   | 21.33       | 18.05  | 0.35   | 1.18   |\n\n\n# Weakness 2: Exclusion of tree-based speculative decoding.\n\nWe would like to point out that while tree-based speculation can achieve higher speedup at batch size = 1, they perform poorly in **batched inference**. Compared to chain-based methods, although tree-based methods accept more tokens in each iteration, they have significantly lower values of $\\frac{\\text{accepted tokens count}}{\\text{total draft tokens count}}$, since only one path in the tree can be accepted while all other branches are discarded. Specifically:\n* At batch size = 1, there are abundant idle computational resources, so verifying a large number of draft tokens does not incur significant latency overhead, making tree-based speculation schemes suitable.\n* As batch size increases, there are fewer idle computational resources, and the large number of draft tokens slows down the verification stage. Moreover, most of the verified tokens are destined to be rejected, which wastes computational power.\n\nAdditionally, tree-based methods are not engineering-friendly, as they require separate handling of attention masks for each request in the batch. Other works involving **batched** speculative decoding such as [1,2], also adopt chain-based approaches.\n\n# Question 1: Is the belief that speculative decoding cannot effectively accelerate MoEs an untested assumption or a well-established empirical claim?\n\nThis belief is supported by **experimental results rather than untested assumptions**. We have *already* provided a citation supporting this belief in line 36 of the original paper, which refers to EAGLE [3]. Table 2 and Table 3 of EAGLE shows that SD achieves 2.54× better speedup on dense models than MoE models. Additionally, [4] from Georgia Tech and Nvidia Research also explicitly states in their abstract that \"speculation is ineffective for MoEs\" and reports 1.5× SD degradation when applied to MoE. We appreciate the reviewer's constructive feedback, and will incorporate these supporting data in the revised version.\n\nBeyond empirical validation, this belief has solid theoretical grounding. Under small batch sizes, systems are memory-bound, meaning execution time is primarily determined by parameter access. For dense models, decoding one token or verifying multiple tokens loads the same amount of parameters, resulting in similar total execution time. However, for MoE models, verifying multiple tokens means activating more experts and loading additional parameters, thereby increasing latency and reducing speculative decoding efficiency. More rigorous analysis can be found in Section 3.1 in the original paper. This theoretical limitation explains why most existing speculative decoding works focus on accelerating dense models.\n\n# Question 2: Illustration of the system.\n\nThank you for your constructive suggestions on improving the clarity of our work. We will include a system diagram and a detailed textual walk-through of Algorithm 1 in the revised manuscript.\n\nAs figures are not allowed in this year's rebuttal, we provide a textual description of the MoESD framework here. Based on our analysis of roofline effects and the number of activated experts in MoE, we establish the expression of SD speedup for MoE as shown in line 11 of Algorithm 1. Due to uncertainties in operator implementation and GPU architecture, the expression includes several parameters with clear physical meanings for relaxation. Through profiling a small number of speedup data points and lightweight fitting (as detailed in Appendix B), we can determine the values of these parameters. Applying these parameters to the expression yields a complete speedup model. Our experimental results show that this modeling consistently matches real measurements, thereby validating the correctness of our theoretical analysis. This modeling approach also makes the tradeoffs involved in SD acceleration for MoE more transparent and explainable. \n\n# References\n\n[1] MagicDec: Breaking Throughput-Latency Trade-off for Long Context Generation with Speculative Decoding.\n\n[2] BASS: Batched Attention-optimized Speculative Sampling.\n\n[3] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.\n\n[4] Utility-Driven Speculative Decoding for Mixture-of-Experts."}, {"review_id": "gmcjqPD5Vt", "reviewer": "Reviewer_kQQx", "summary": "This paper studies speculative decoding (SD) for MoE-based LLMs from a new perspective: identifying the conditions under which MoE LLMs benefit most from SD. Contrary to the common assumption that SD may not accelerate MoE-based LLMs, the paper shows that SD can be effective at moderate batch sizes, where experts are typically fully utilized. The benefits become larger when the LLMs become sparser. To support the theoretical analysis, the paper introduces a new metric (i.e. target efficiency) and a modeling approach to formulate optimal SD configurations. Experimental results confirm that the theoretical predictions align closely with real-world performance across different GPUs (e.g., 2×A100, 2×H100) and tasks (e.g., HumanEval, MT-Bench).", "questions": "N.A.", "limitations": "Limitations (e.g. longer context/KV caches) have been discussed in the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "---\nStrengths\n\ni) Topic: This paper explores SD for MoE and general LLMs from a different perspective—identifying the conditions under which SD provides the most benefit. The topic is timely and interesting, addressing an important gap in current research.\n\nii) Methodology: To support the theoretical analysis, the paper introduces a new metric (i.e. target efficiency) and a modeling approach to derive optimal SD configurations. The formulation is well-motivated and technically sound.\n\niii) Experiments: Empirical results demonstrate close alignment between the theoretical predictions and real-world performance across various GPUs (e.g., 2×A800, 2×H800) and benchmarks (e.g., HumanEval, MT-Bench).\n\n---\nWeaknesses\n\nThe paper would be stronger with more comprehensive evaluations across a broader range of LLMs and hardware configurations.\n\ni) Model diversity: The current experiments are based on a single pair of target and draft models—Qwen2-57B-A17B-Instruct and Qwen2-0.5B-Instruct—which limits the generalizability of the findings. The authors are encouraged to include additional representative model pairs, ideally with varying sizes and architectures.\n\nii) Hardware settings: All experiments were conducted on 2xA800 or 2xH800 GPUs. While informative, these settings may not fully reflect real-world deployment scenarios, which often involve distributed environments with more inter-GPU communication or different compute-memory tradeoffs (e.g., H800 vs. L20).", "was_revised": true, "final_justification": "The reviewer appreciated the authors' efforts on extra experiments. The reviewer'd like to keep the postive rating of the paper.", "rebuttal": "Thank you for your valuable suggestions on diversity, which help to make our conclusion more convincing. We have supplemented experiments, hoping they can address your concerns.\n\n# Weakness 1:  Model diversity\n\nWe have added experiments for another MoE: Mixtral-8x7B-Instruct-v0.1. We used Eagle's **speculation head** for Mixtral rather than a standalone draft model to validate our method's generality. Due to the format limitations of this year's rebuttal, we can only present the results in tables as follows. Your advice on model diversity is very valuable and we will add them to the revised paper.\n\nAs shown in tables, experimental results on another MoE also **conform to our theoretical analysis**, where the speculative decoding (SD) speedups on MoE models first increase and then decrease as batch size increases. Since our theoretical analysis places no restrictions on model architectures, this result is as expected and should hold for other MoE models.\n\n**Table 1: The SD speedup trend vs. batch size, under different settings for Mixtral-8x7B-Instruct-v0.1**\n\n|  Batch sizes   | 2    | 4    | 6    | 8    | 10   | 12   | 16   | 20   | 24   | 28   | 32   | 36   | 40   | 60   | 80   | 100  |\n|----|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| Speedup @(humaneval, temperature 0) | 0.90 | 0.96 | 1.19 | 1.04 | 1.71 | 1.67 | 1.43 | 1.31 | 1.21 | 1.18 | 1.14 | 1.11 | 1.15 | 0.97 | 0.80 | 0.72 |\n| Speedup @(humaneval, temperature 1) | 0.91 | 0.91 | 1.07 | 0.99 | 1.37 | 1.38 | 1.16 | 1.13 | 1.01 | 1.06 | 1.07 | 1.05 | 1.04 | 0.88 | 0.71 | 0.54 |\n| Speedup @(mtbench, temperature 0) | 0.84 | 0.86 | 1.00 | 0.98 | 1.34 | 1.28 | 1.23 | 1.22 | 1.14 | 1.12 | 1.07 | 1.06 | 1.04 | 0.95 | 0.88 | 0.81 |\n| Speedup @(mtbench, temperature 1) | 0.85 | 0.80 | 0.96 | 0.93 | 1.15 | 1.17 | 1.13 | 1.14 | 1.09 | 1.05 | 0.99 | 0.97 | 0.92 | 0.86 | 0.75 | 0.69 |\n\n\n**Table 2: The overview of SD speedup on Mixtral-8x7B-Instruct-v0.1**\n\n*Notation: This table follows the format of Table 1 in the original paper. $\\gamma$ represents draft token length, $T_{AR}$ denotes auto-regressive decoding time, $T_{SD}$ denotes speculative decoding time, $\\sigma$ is the ratio of actually generated tokens to theoretical maximum, and $\\mathbf{x}$ denotes maximal speedup across batch sizes.*\n\n| Dataset   | Temperature | $\\gamma=2$  |        |        |        | $\\gamma=3$  |        |        |        | $\\gamma=4$  |        |        |        |\n|-|---|--|-----|-----|-----|--|-----|-----|-----|--|-----|-----|-----|\n|    |      | $T_{AR}$    | $T_{SD}$      | $\\sigma$      | $\\mathbf{x}$  | $T_{AR}$    | $T_{SD}$      | $\\sigma$      | $\\mathbf{x}$  | $T_{AR}$    | $T_{SD}$      | $\\sigma$      | $\\mathbf{x}$  |\n| humaneval | 0.0  | 20.86       | 12.47  | 0.78   | 1.67   | 21.00       | 12.46  | 0.66   | 1.69   | 20.86       | 11.69  | 0.58   | **1.79**   |\n| humaneval | 1.0  | 21.52       | 15.58  | 0.61   | **1.38**   | 21.39       | 16.03  | 0.46   | 1.33   | 21.48       | 16.23  | 0.39   | 1.32   |\n| mtbench   | 0.0  | 21.61       | 16.10  | 0.61   | **1.34**   | 21.61       | 16.43  | 0.46   | 1.32   | 21.36       | 16.89  | 0.39   | 1.26   |\n| mtbench   | 1.0  | 21.33       | 17.70  | 0.53   | **1.21**   | 21.33       | 17.84  | 0.43   | 1.20   | 21.33       | 18.05  | 0.35   | 1.18   |\n\n\n\n# Weakness 2: Hardware settings diversity\n\n\nWe add experiments for additional hardware settings including **more inter-GPU communication** and **different compute-memory tradeoff** as you required. The configurations include: 2×A800 (original setup for comparison), 4×A800 (newly added), and 4×L40 (newly added). Since L40 only has 48GB memory while the Qwen2 MoE model has 57B parameters, we use 4×L40 due to memory constraints.\n\n**More inter-GPU Communication:** 2×A800 (1st group) and 4×A800 (2nd group) configurations in Table 3 are compared. For absolute timings (i.e., $T_{AR}$ and $T_{SD}$), we observe obvious decreases due to additional devices. For SD speedups,  they maintain similar trends but with slight degradation.  This occurs because while the large model is parallelized between GPUs, the small draft model remains on a single GPU, relatively increasing its forward pass time compared to the large model's forward time.\n\n**Different compute-memory tradeoffs:** 4×A800 (2nd group) and 4×L40 (3rd group) configurations in Table 3 are compared. A800-80GB provides 312 TFlops BF16 performance with 1.94 TB/s bandwidth (roofline ridge point ≈161), while L40 offers 181 TFlops with 864 GB/s bandwidth (ridge point ≈214). L40's higher ridge point indicates more available compute capacity for SD verification, enabling higher speedups as confirmed by our experimental results.\n\n**Table 3: SD speedup on different hardware settings**\n\n*Notation: This table follows the format of Table 1 in the original paper. $\\gamma$ represents draft token length, $T_{AR}$ denotes auto-regressive decoding time, $T_{SD}$ denotes speculative decoding time, $\\sigma$ is the ratio of actually generated tokens to theoretical maximum, and $\\mathbf{x}$ denotes maximal speedup across batch sizes.*\n\n| Device  | Dataset  | Temp | $\\gamma=2$ |  |  |  | $\\gamma=3$ |  |  |  | $\\gamma=4$ |  |  |  |\n|----|-----|-|--|---|---|---|--|---|---|---|--|---|---|---|\n| |  |      | $T_{AR}$  | $T_{SD}$ | $\\sigma$ | $\\mathbf{x}$     | $T_{AR}$  | $T_{SD}$ | $\\sigma$ | $\\mathbf{x}$     | $T_{AR}$  | $T_{SD}$ | $\\sigma$ | $\\mathbf{x}$     |\n| 2xA800  | humaneval| 0.0  | 18.89     | 11.61    | 0.94     | 1.63     | 15.93     | 8.11     | 0.93     | 1.96     | 15.93     | 7.31     | 0.91     | **2.18** |\n| | humaneval| 1.0  | 19.13     | 12.93    | 0.83     | 1.48     | 21.20     | 14.09    | 0.73     | 1.50     | 19.13     | 11.14    | 0.67     | **1.72** |\n| | mtbench  | 0.0  | 20.92     | 16.70    | 0.71     | 1.25     | 16.00     | 12.43    | 0.62     | **1.29** | 20.92     | 17.53    | 0.55     | 1.19     |\n| | mtbench  | 1.0  | 21.15     | 17.33    | 0.68     | 1.22     | 19.09     | 14.83    | 0.57     | **1.29** | 19.09     | 15.93    | 0.48     | 1.20     |\n| 4xA800  | humaneval| 0.0  | 11.20     | 6.77     | 0.95     | 1.65     | 11.20     | 5.89     | 0.93     | 1.90     | 11.20     | 5.38     | 0.90     | **2.08** |\n| | humaneval| 1.0  | 11.72     | 8.51     | 0.81     | 1.38     | 12.05     | 8.30     | 0.73     | 1.45     | 11.23     | 7.70     | 0.67     | **1.46** |\n| | mtbench  | 0.0  | 11.26     | 8.92     | 0.72     | **1.26**     | 11.26     | 9.10     | 0.61     | 1.24     | 11.26     | 9.82     | 0.52     | 1.15 |\n| | mtbench  | 1.0  | 11.78     | 10.32    | 0.67     | 1.14     | 11.30     | 9.42     | 0.58     | **1.20**     | 11.78     | 11.25    | 0.47     | 1.05 |\n| 4xL40   | humaneval| 0.0  | 17.84     | 10.00    | 0.95     | 1.79     | 17.84     | 8.33     | 0.93     | 2.14     | 17.84     | 7.94     | 0.90     | **2.25** |\n| | humaneval| 1.0  | 17.89     | 12.27    | 0.80     | 1.46     | 17.89     | 11.07    | 0.74     | 1.62     | 17.89     | 10.91    | 0.65     | **1.64** |\n| | mtbench  | 0.0  | 20.40     | 15.87    | 0.71     | **1.29**     | 20.40     | 16.22    | 0.62     | 1.26     | 20.40     | 16.33    | 0.54     | 1.25 |\n| | mtbench  | 1.0  | 20.58     | 16.02    | 0.68     | **1.28**     | 18.11     | 14.75    | 0.54     | 1.23     | 18.11     | 15.54    | 0.48     | 1.17 |\n\nWe also present how speedup varies with batch size in Table 4. As shown, the speedup demonstrates a consistent trend that aligns with the theoretical analysis and experiments reported in the original paper: the speedup initially increases and then decreases, achieving peak performance at moderate batch sizes.\n\n**Table 4: SD speedup vs. batch sizes on different hardware settings**\n|  Batch sizes   | 1    | 2    | 4    | 8   | 12   | 16   | 20   | 24   | 28   | 32   | 36   | 40   | 60   | 80   | 100  |\n|----|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| 2xA800 | 1.60 | 1.73 | 1.91 | 2.18 | 2.07 | 1.85 | 1.88 | 1.81 | 1.80 | 1.97 | 1.77 | 1.44 | 1.27 | 1.09 | 0.98 |\n| 4xA800 | 1.47 | 0.57 | 1.74 | 1.87 | 2.08 | 1.55 | 1.72 | 1.56 | 1.56 | 1.45 | 1.40 | 1.45 | 1.12 | 1.04 | 0.90 |\n| 4xL40| 1.31 | 1.57 | 1.85 | 2.16 | 2.25 | 1.86 | 1.95 | 1.84 | 1.86 | 1.84 | 1.80 | 1.59 | 1.33 | 1.16 | 0.94 |\n\nIn summary, the SD speedups in scenarios with more inter-GPU communication or compute-memory tradeoffs **conforms to our theoretical analysis as expected**. We will include these results in the revised paper to strengthen the generalizability of our findings."}, {"review_id": "z6XUs5UewS", "reviewer": "Reviewer_WmFj", "summary": "In this paper, the authors challenge the common belief that speculative decoding (SD) is less efficient for MoE models. With both theoretical justification and empirical experiments, the authors show that SD can be more efficient for MoE when the batch size is moderate and MoE is sparse. \n\nFor the theoretical part, the authors focus on a commonly negligible factor, target efficiency, instead of the SD optimization, and show that SD's efficiency is more related to this target efficiency instead of acceptance rate, which also makes the acceleration process transparent and explainable. \n\nWith practical experiments on a large MoE model (Qwen2-57B-17A), the authors show a good correlation between the observation and theoretical justification. The results also show that SD is less efficient for dense models for moderate batch size.", "questions": "I'm open to raise my evaluation score if the authors could properly clarify my 2 weakness points.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Note**: I'm not very qualified for reviewing this paper, since I'm more an engineering researcher. I focus more on the engineering part.\n\n## Strengths\n- The paper is well-written with clear motivation and thorough explanation of the design choice;\n- Different from previous works that mostly focus on the algorithm optimization, this paper focuses on a commonly negligible factor when analyzing SD's efficiency, and show a very good correlation between the formula and the observation.\n\n## Weaknesses\n- The findings are very intuitive from my perspective. For a small batch size, MoE is inclined to memory-bound. For a large batch size, MoE is inclined to computation-bound. So a moderate batch size will be efficient for MoE.\n- The paper shows that SD works well for sparse MoE with a moderate batch size, hinting its limitation.", "was_revised": true, "final_justification": "In the discussion, the authors well address my misunderstanding of the counter-intuition and batch size. So I increase the score to 4 (botherline accept).", "rebuttal": "Thank you for your careful reading and thoughtful comments on our paper! We provide our responses below and hope they address your concerns.\n\n# Weakness 1: Clarify the counterintuitive part of our paper's findings.\n\nThere seem to be some misunderstandings of our work. We are not arguing that **MoE itself** favours moderate batch sizes, but rather that **SD acceleration for MoE** favours moderate batch sizes.\n\nGiven that speculative decoding (SD) leverages spare computational resources caused by memory-boundedness, SD acceleration is expected to be **more pronounced at small batch sizes**, which has been widely observed in previous SD works [1,2,3]. Therefore, your statement that \"*For a small batch size, MoE is inclined to memory-bound... So a moderate batch size will be efficient for MoE*\" is incorrect — SD should be more effective under memory-bound conditions.\n\nThe **counterintuitive** part for MoE is that SD is ineffective at small batch sizes, but becomes **more effective as batch size increases** to moderate level. This is because at small batch sizes, MoE's multi-token verification becomes significantly slower than single-token decoding, leading to degraded target efficiency (the metric we have proposed in paper). Computing multiple tokens often activates more experts, resulting in increased parameter loading and therefore prolonging execution time.\n\nBeyond the *preliminary* conclusion of SD speedup trend on MoE mentioned above, we *further* investigate the impact of MoE sparsity on SD speedup. Through theoretical analysis and experiments, we show that SD is actually **more effective for sparser MoE models**. The table below compares SD speedup between MoE models and dense models (which can be viewed as completely non-sparse MoE models) under identical settings. While MoE initially shows inferior acceleration ratios, it subsequently achieves more obvious acceleration over an extended range.\n\n**Table: Comparison of MoE speedup on the MoE model (Qwen2-57B-A14B-Instruct) and the dense model (OPT),  corresponding to Figure 2c in the original paper.**\n| batch size | 1 | 2 | 4 | 8 | 12 | 16 | 20 | 24 | 28 | 32 | 36 | 40 | 44 | 48 | 52 | 56 | 60 | 80 |\n| - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |\n| SD speedup for dense model | 1.807 | 1.418 | 1.346 | 1.261 | 1.177 | 1.085 | 1.038 | 0.947 | 0.848 | 0.839 | 0.797 | 0.756 | 0.712 | 0.698 | 0.665 | 0.625 | 0.648 | 0.585 |\n| SD speedup for moe model | 1.578 | 1.381 | 1.152 | 1.511 | 1.717 | 1.414 | 1.437 | 1.397 | 1.398 | 1.281 | 1.179 | 1.167 | 1.156 | 1.114 | 1.072 | 1.059 | 1.012 | 0.877 |\n| $\\frac{\\text{SD speedup for moe model}}{\\text{SD speedup for dense model}}$ | 0.874 | 0.974 | 0.856 | **1.198** | **1.459** | **1.303** | **1.385** | **1.475** | **1.648** | **1.527** | **1.479** | **1.543** | **1.624** | **1.596** | **1.612** | **1.695** | **1.563** | **1.498** |\n\n\nThe beliefs that SD is unsuitable for MoE and degrades with increasing batch sizes have been long-held in the community[1,2,3,4,5]. Consequently, SD research has focused primarily on dense models for edge deployment, overlooking the potential for recently popular MoE architectures. Our work breaks this conventional wisdom by identifying opportunities for SD in other regimes, thereby helping the community develop a more comprehensive understanding of SD.\n\nIn summary, our paper effectively challenges traditional viewpoints and provides **counterintuitive** findings as follows:\n* SD is more suitable for dense models than MoEs [4,5] → We demonstrate that at moderate batch sizes, SD is more effective for sparse MoE models.\n* SD effectiveness decreases with increasing batch size [1,2,3] → We demonstrate that for MoE, effectiveness may actually increase with growing batch size.\n\n\n\n# Weakness 2: Concerns about moderate batch size\n\nWe would like to clarify the practical value of our work from both application and model perspectives.\n\n**From the application perspective**\n\n* Moderate batch sizes are common in private serving scenarios, which are becoming increasingly popular due to their ability to ensure data security. For example, major AI companies now provide internal *enterprise chatbot* services for their employees.\n\n* When latency requirements are strict, large batch sizes are often not feasible. LLM serving must satisfy not only throughput requirements but also other service level objectives (SLOs) including time-to-first-token (TTFT) and time-per-output-token (TPOT). Excessively large batches limit the computational resources allocated to each request, leading to latency violations. In such cases, moderate batch sizes are necessary.\n\n* From a high-level perspective, our work reveals that applying SD on MoE can **relax the trade-off between latency and throughput**. In most cases, latency and throughput cannot be optimized simultaneously. For example, when applying SD on dense models, speedup diminishes as batch size increases. However, we have shown that for MoE models, there exists a regime where SD speedup increases (lower latency) with larger batch sizes (larger throughput).\n\n**From the model perspective**\n\nCompared to dense models, MoE models are actually *least efficient* in moderate batch sizes. At small batch sizes, MoE models use a small portion of FFN parameters. At large batch sizes, all loaded parameters are provided with abundant computation. However, at moderate batch sizes, all experts are activated while the workload provides insufficient computation, leading to underutilized computational resources. Therefore, optimizing MoE in this regime is critical. Our work demonstrates that SD is a viable solution for such an efficiency gap.\n\n\n# Reference\n\n[1] Optimizing speculative decoding for serving large language models using goodput.\n\n[2] The synergy of speculative decoding and batching in serving large language models.\n\n[3] Specinfer: Accelerating generative llm serving with speculative inference and token tree verification.\n\n[4] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.\n\n[5] Utility-Driven Speculative Decoding for Mixture-of-Experts."}, {"review_id": "7H58LYhZZE", "reviewer": "Reviewer_Xih7", "summary": "This paper revisits speculative decoding (SD) in the context of Mixture-of-Experts (MoE) models and challenges the widely held belief that SD is ineffective for sparse MoEs due to verification overhead. The authors introduce a theoretical and empirical framework that shows under moderate batch sizes, SD can offer greater acceleration for sparse MoEs than dense models. A new metric, target efficiency, is proposed to analyze SD speedup potential independent of the acceptance rate. Experimental results on Qwen2 models and two benchmarks (HumanEval and MT-bench) validate the theory and show up to 2.29× speedup.", "questions": "1. Have the authors considered hybrid metrics that combine target efficiency and acceptance rate to better predict speedup?\n2. Beyond moderate batch sizes, SD performance degrades. Are there strategies (e.g., chunked batch processing) that can extend SD benefits in such regimes?\n3. The paper uses a fixed draft model. Could training a task-specific or sparsity-aware draft model further improve the speedup or acceptance rate?", "limitations": "Yes.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The core claim that SD can outperform on sparse MoEs under moderate batch sizes is counterintuitive and well-motivated.\n2. The formulation of the target efficiency metric adds analytical clarity\n\nWeaknesses:\n1. While Qwen2 is used with MT-bench and HumanEval, additional LLMs (e.g., Llama, Mistral) would strengthen generalization claims.", "was_revised": true, "final_justification": "I think the reviewers answered my questions well. I have no further questions to this submission and will maintain my current score.", "rebuttal": "Thank you very much for your careful review and constructive feedback! Your insightful questions have provided valuable directions for future improvements and greatly inspired our thinking. Our responses are as follows, and we hope they can resolve your questions and concerns.\n\n# Weakness: Model diversity.\n\nWe have added experiments for another MoE: Mixtral-8x7B-Instruct-v0.1. We used Eagle's **speculation head** for Mixtral rather than a standalone draft model to validate our method's generality. Due to the format limitations of this year's rebuttal, we can only present the results in tables as follows. Your advice on model diversity is very valuable and we will add them to the revised paper.\n\nAs shown in tables, experimental results on another MoE also **conform to our theoretical analysis**, where the speculative decoding (SD) speedups on MoE models first increase and then decrease as batch size increases. Since our theoretical analysis places no restrictions on model architectures, this result is as expected and should hold for other MoE models.\n\n**Table 1: The SD speedup trend vs. batch size, under different settings for Mixtral-8x7B-Instruct-v0.1**\n\n|  Batch sizes   | 2    | 4    | 6    | 8    | 10   | 12   | 16   | 20   | 24   | 28   | 32   | 36   | 40   | 60   | 80   | 100  |\n|----|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| Speedup @(humaneval, temperature 0) | 0.90 | 0.96 | 1.19 | 1.04 | 1.71 | 1.67 | 1.43 | 1.31 | 1.21 | 1.18 | 1.14 | 1.11 | 1.15 | 0.97 | 0.80 | 0.72 |\n| Speedup @(humaneval, temperature 1) | 0.91 | 0.91 | 1.07 | 0.99 | 1.37 | 1.38 | 1.16 | 1.13 | 1.01 | 1.06 | 1.07 | 1.05 | 1.04 | 0.88 | 0.71 | 0.54 |\n| Speedup @(mtbench, temperature 0) | 0.84 | 0.86 | 1.00 | 0.98 | 1.34 | 1.28 | 1.23 | 1.22 | 1.14 | 1.12 | 1.07 | 1.06 | 1.04 | 0.95 | 0.88 | 0.81 |\n| Speedup @(mtbench, temperature 1) | 0.85 | 0.80 | 0.96 | 0.93 | 1.15 | 1.17 | 1.13 | 1.14 | 1.09 | 1.05 | 0.99 | 0.97 | 0.92 | 0.86 | 0.75 | 0.69 |\n\n\n**Table 2: The overview of SD speedup on Mixtral-8x7B-Instruct-v0.1**\n\n*Notation: This table follows the format of Table 1 in the original paper. $\\gamma$ represents draft token length, $T_{AR}$ denotes auto-regressive decoding time, $T_{SD}$ denotes speculative decoding time, $\\sigma$ is the ratio of actually generated tokens to theoretical maximum, and $\\mathbf{x}$ denotes maximal speedup across batch sizes.*\n\n| Dataset   | Temperature | $\\gamma=2$  |        |        |        | $\\gamma=3$  |        |        |        | $\\gamma=4$  |        |        |        |\n|-|---|--|-----|-----|-----|--|-----|-----|-----|--|-----|-----|-----|\n|    |      | $T_{AR}$    | $T_{SD}$      | $\\sigma$      | $\\mathbf{x}$  | $T_{AR}$    | $T_{SD}$      | $\\sigma$      | $\\mathbf{x}$  | $T_{AR}$    | $T_{SD}$      | $\\sigma$      | $\\mathbf{x}$  |\n| humaneval | 0.0  | 20.86       | 12.47  | 0.78   | 1.67   | 21.00       | 12.46  | 0.66   | 1.69   | 20.86       | 11.69  | 0.58   | **1.79**   |\n| humaneval | 1.0  | 21.52       | 15.58  | 0.61   | **1.38**   | 21.39       | 16.03  | 0.46   | 1.33   | 21.48       | 16.23  | 0.39   | 1.32   |\n| mtbench   | 0.0  | 21.61       | 16.10  | 0.61   | **1.34**   | 21.61       | 16.43  | 0.46   | 1.32   | 21.36       | 16.89  | 0.39   | 1.26   |\n| mtbench   | 1.0  | 21.33       | 17.70  | 0.53   | **1.21**   | 21.33       | 17.84  | 0.43   | 1.20   | 21.33       | 18.05  | 0.35   | 1.18   |\n\n\n# Question 1: Consideration of a hybrid metric that combines target efficiency and acceptance rate\n\nA hybrid metric combining target efficiency and acceptance rate would closely approximate the speedup metric itself. As demonstrated in the theoretical derivation in Section 3.1:\n\n$$Speedup = \\frac{1-\\alpha^{\\gamma+1}}{1-\\alpha} \\cdot \\frac{1}{\\gamma \\cdot \\frac{T_D(B,1)}{T_T(B,1)} + \\frac{T_T(B,\\gamma)}{T_T(B,1)} + \\frac{T_{reject}}{T_T(B,1)}}$$\n\nWhere $\\alpha$ represents the acceptance rate, $\\frac{T_T(B,1)}{T_T(B,\\gamma)}$ corresponds to our target efficiency metric, $\\gamma$ is the average draft token length (a constant). Both $T_D(B,1)$ (draft model inference time) and $T_{reject}$ (rejection sampling overhead) are significantly smaller than $T_T(B,1)$, contributing minimally compared to $\\frac{T_T(B,1)}{T_T(B,\\gamma)}$.\n\nTherefore, the hybrid metric you suggested would be nearly identical to the speedup formulation above. To avoid redundancy, we chose to directly use the speedup metric rather than introducing an additional combined metric. However, the introduction of target efficiency is critical as acceptance rate alone cannot fully explain the SD speedup trend.\n\n\n# Question 2: Discussion of chunked batch processing\n\nThank you for this excellent question. Chunked batch processing is indeed a valuable strategy that can help **jointly optimize throughput and latency**.\n\nLLM serving systems must satisfy both throughput requirements and latency constraints such as Time-To-First-Token (TTFT) and Time-Per-Output-Token (TPOT). Generally, there exists a fundamental **trade-off between low latency and high throughput**. When batch sizes are very large, each request receives limited computational resources, potentially failing to meet latency requirements.\n\nFor dense models, SD also follows this trade-off [1,2,3]: when speedup is significant, batch sizes are small (limiting throughput); when batch size increases, SD speedup degrades substantially (worse latency). However, our work demonstrates that for SD with MoE models, there actually exists a regime where speedup **increases with** batch sizes. This unique property suggests that chunked batch processing could simultaneously enhance both throughput and latency.\n\nWe believe this represents a promising direction for future work, particularly in serving environments where satisfying Service Level Objectives (SLOs) is critical.\n\n# Question 3: Training a task-specific or sparsity-aware draft model to further improve speedup\n\nRegarding task-specific draft models, this is indeed an active area of research. Previous works [4,5,6] have demonstrated that training specialized draft models can improve acceptance rates for specific tasks. Additionally, recent work [7] has explored an online approach that finetunes the draft model with real-time draft token acceptance patterns during serving. Since acceptance rate optimization has been extensively studied and is not our contribution, we chose to use the basic fixed draft model.\n\nRegarding sparsity-aware draft models, we think this approach only has limited benefits. The acceptance rate fundamentally reflects the alignment between draft and target models on a given task. While task-specific training can indeed improve this alignment, sparsity patterns in the target model don't have a direct impact on this. As long as MoE target models achieve similar performance on a task, they are identical in accepting tokens from the draft model's perspective.\n\n# References\n\n[1] Optimizing speculative decoding for serving large language models using goodput.\n\n[2] The synergy of speculative decoding and batching in serving large language models.\n\n[3] Specinfer: Accelerating generative llm serving with speculative inference and token tree verification.\n\n[4] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty.\n\n[5] DistillSpec: Improving Speculative Decoding via Knowledge Distillation.\n\n[6] Learning Harmonized Representations for Speculative Sampling.\n\n[7] Online Speculative Decoding."}]

Author rebuttal:

Meta-review: This paper studies whether speculative decoding (SD) is effective for sparse Mixture-of-Experts (MoE) models. It theoretically and empirically demonstrates that MoE models benefit significantly from SD at moderate batch sizes and benefit from SD even more than dense models. The authors introduce a new metric, "target efficiency," to provide a more comprehensive understanding of SD performance beyond the acceptance rate. The paper first introduces some theoretical concepts and then jumps to setting up a framework for establishing a simulation for speedups for target devices/ setups.

**Strengths:**
*   The paper addresses a timely problem and shows that MoEs can benefit more from SD than dense models at moderate batch sizes (kQQx, Xih7).
*   Formalizing the tradeoffs of SD and introducing a "target efficiency", offering a more nuanced way to analyze SD performance beyond simple acceptance rates (jdmT, 8fAG, WmFj).

**Weaknesses:**
*   Initial reviews pointed out a narrow experimental evaluation regarding model diversity (8fAG, kQQx, Xih7) and hardware configurations (kQQx). Some reviewers also noted a lack of direct end-to-end comparisons and statistical error bars in the original submission (jdmT).
* The presentation can be improved a bit to better connect the theory part with the modeling/simulation part. Also, there are many notations that make it hard to follow. The optimization parameters are listed in the appendix which is helpful, it will be good to have a general glossary for all the notations, what they mean, and how to obtain them (if they represent an empirical measure).

Overall the paper provides a valuable analysis with a clear message (SD benefits for MoE) and establishing a simulation framework to benchmark expected gains. The authors provided valuable clarifications and measurements during the rebuttal. Improving the presentation and also linking the code (provided in supplemental) and clear guidelines for obtaining measurements and setting up new benchmarks should provide a good resource for researchers in the field.
Decision: Accept (spotlight)