Title: MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE
Venue: NeurIPS 2025
Accepted: yes
Keywords: Speculative decoding

MoE

Inference

Acceleration
Abstract: Large Language Models (LLMs) have achieved remarkable success across many applications, with Mixture of Experts (MoE) models demonstrating great potential. Compared to traditional dense models, MoEs achieve better performance with less computation. Speculative decoding (SD) is a widely used technique to accelerate LLM inference without accuracy loss, but it has been considered efficient only for dense models. In this work, we first demonstrate that, under medium batch sizes, MoE surprisingly benefits more from SD than dense models. Furthermore, as MoE becomes sparser -- the prevailing trend in MoE designs -- the batch size range where SD acceleration is expected to be effective becomes broader. To quantitatively understand tradeoffs involved in SD, we develop a reliable modeling based on theoretical analyses. While current SD research primarily focuses on improving acceptance rates of algorithms, changes in workload and model architecture can still lead to degraded SD acceleration even with high acceptance rates. To address this limitation, we introduce a new metric 'target efficiency' that characterizes these effects, thus helping researchers identify system bottlenecks and understand SD acceleration more comprehensively. For scenarios like private serving, this work unveils a new perspective to speed up MoE inference, where existing solutions struggle. Experiments on different GPUs show up to 2.29x speedup for Qwen2-57B-A14B at medium batch sizes and validate our theoretical predictions.

MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE

Abstract

Large Language Models (LLMs) have achieved remarkable success across many applications, with Mixture of Experts (MoE) models demonstrating great potential. Compared to traditional dense models, MoEs achieve better performance with less computation. Speculative decoding (SD) is a widely used technique to accelerate LLM inference without accuracy loss, but it has been considered efficient only for dense models. In this work, we first demonstrate that, under medium batch sizes, MoE surprisingly benefits more from SD than dense models. Furthermore, as MoE becomes sparser - the prevailing trend in MoE designs - the batch size range where SD acceleration is expected to be effective becomes broader. To quantitatively understand tradeoffs involved in SD, we develop a reliable modeling based on theoretical analyses. While current SD research primarily focuses on improving acceptance rates of algorithms, changes in workload and model architecture can still lead to degraded SD acceleration even with high acceptance rates. To address this limitation, we introduce a new metric target efficiency that characterizes these effects, thus helping researchers identify system bottlenecks and understand SD acceleration more comprehensively. For scenarios like private serving, this work unveils a new perspective to speed up MoE inference, where existing solutions struggle. Experiments on different GPUs show up to 2.29x speedup for Qwen257B-A14B at medium batch sizes and validate our theoretical predictions.

1 Introduction

Recent years have witnessed remarkable success in Large Language Models (LLMs), with Mixture of Experts (MoE) architectures showing tremendous potential. Unlike dense models use a single feed-forward network (FFN) to process all inputs, MoE models replace the FFN with multiple specialized "expert" networks plus a router that selectively activates only a few experts for each input token. Such sparsity in structure enables MoEs with more parameters to achieve higher computational efficiency, and multiple state-of-the-art LLMs, such as DeepseekV3 [1] and Qwen2.5-Max [2], are all MoEs. MoE model architectures are evolving toward larger scales with increased sparsity [3, 4, 1] and more balanced workload distribution among experts [5, 6].

Speculative decoding (SD) is a lossless technique to accelerate LLM inference, but conventional wisdom suggests that its efficacy diminishes when applied to MoEs. In SD, a smaller draft model is introduced to rapidly generate multiple candidate tokens, while the larger target model verifies these predictions in parallel, preserving only correctly speculated tokens. For dense models' inference, the time taken to generate a single token and verify multiple ones is roughly the same, as both tasks require the full set of parameters to be loaded once. Therefore, SD gains acceleration through fewer forward

rounds of the target model and shorter decoding time of the draft model. However, this acceleration has been demonstrated to diminish in MoEs [7, 8], as the multiple draft tokens in verification activate more experts than a single token, leading to larger memory access and significantly longer verification time compared to a standard decoding step.

In this work, we challenge the conventional belief and demonstrate that, under a moderate batch size, SD can be more effective for MoEs than for dense models. Our key insight is that when the batch size is moderate such that all experts are already activated in a single decoding step, verifying multiple draft tokens will not incur additional expert parameter loading costs. Furthermore, as the MoE becomes sparser, each expert processes fewer tokens per parameter loading, leading to lower utilization of arithmetic units and thereby creating greater acceleration opportunities for SD.

The insight above is supported by comprehensive theoretical analyses, through which we identify a new metric target efficiency to quantify how systemic factors (such as workload and target model architecture) affect SD speedup. In contrast to existing SD works that use acceptance rate [9, 10, 7, 11], an algorithmic metric to evaluate how accurately the draft model speculates the target model, our proposed target efficiency isolates extrinsic factors like algorithm selection and focuses on intrinsic system bottlenecks caused by the target model's computational and memory access requirements. As demonstrated in the following sections, even with similar acceptance rates, systemic factors can greatly impact SD effectiveness, making our metric target efficiency necessary for a comprehensive understanding of SD acceleration.

As a further step, we build a quantitative modeling of SD speedup for MoE based on these theoretical analyses. The consistent matching between our modeling and experiment results confirms the reliability of our analyses. Additionally, the modeling itself provides an approach for analyzing the execution time of different components, making the end-to-end SD acceleration results more transparent and explainable.

Our work offers a new perspective for lossless MoE acceleration, particularly well-suited for private serving scenarios [12, 13, 14]. Private serving has gained popularity among enterprises seeking to safeguard data and model security, with typical applications such as in-house chatbots. These environments typically process moderate batches containing tens of requests. Additionally, our findings can be applied to latency-critical scenarios where large batch sizes are infeasible, or memoryrestricted environments where MoEs exceed GPU capacity.

In Summary, the main contributions of our work are:

We refine the conventional belief that speculative decoding cannot effectively accelerate MoEs, demonstrating that under moderate batch sizes, SD is actually more effective in a wider range of batch sizes for sparser MoEs than dense models.

Based on theoretical analysis, we developed a reliable modeling for SD speedup, thus making the acceleration process transparent and explainable. Existing metrics only assess algorithmic optimization efficiency and cannot fully explain SD speedup, so we introduce a new systemic metric target efficiency that reveals speedup opportunities inherent in the target model.

Our findings can be applied to accelerate scenarios like private serving. Experiments on various GPUs with the Qwen2-57B-14A-Instruct model demonstrate that SD achieves the highest speedup at the moderate batch size, reaching 2.29x. These experiments also validate our theoretical prediction that SD is more favorable for sparser MoEs.

2 Related Work

MoE acceleration. MoE has emerged as a promising LLM architecture, and many techniques optimize its inference. Model compression methods, including pruning [15, 16], quantization [17, 18], distillation [19, 20], and decomposition [21, 22], have been applied to MoEs and achieved great acceleration. They sacrifice model quality for speedup, as in dense models. When MoEs are too large to fit in GPU memories and offloading becomes a necessity, several system-level approaches have emerged to optimize inference latency through improved scheduling techniques. Expert prefetching [23, 24] predicts and pre-loads experts for upcoming layers based on previous activation patterns, thus overlapping expert loading with current layer computation. Expert caching [25, 26] caches most frequently activated experts in GPU memory, leveraging expert locality to reduce expensive

offloading. Compared to them, our work unveils a new perspective for MoE acceleration that is lossless and doesn't depend on expert imbalance.

Speculative Decoding. Speculative decoding (SD), initially proposed by [10] and [9], has emerged as a widely adopted technique for accelerating LLM inference without sacrificing generation quality. Basic SD employs a smaller model to rapidly generate draft tokens, which are then verified in parallel by the target model that needs to be accelerated. Afterwards, more algorithms are developed to lift the acceptance rate of draft tokens. [27, 11, 28, 29, 7, 30, 31] adopt tree-structured generation patterns rather than chains to explore a broader range of potential completions. [1, 11, 7, 30, 31] propose to replace draft models with specifically trained speculative heads integrated in the target model.

Despite advances in SD algorithms, it has long been considered ineffective for large batches [32, 33, 27] or MoE [7, 8], since the verification time in these cases significantly increases. Until recently, MagicDec [34] first challenged that in long-sequence regimes, SD can effectively accelerate large batches , primarily due to the significantly increased KV cache altering the computation-to-memory access ratio of the model. However, SD research for MoE remains unexplored. In response, our work fills this gap, unveiling that under certain conditions, SD can effectively accelerate MoE models.

3 Theoretical Analysis

In this section, we present the theoretical analyses supporting our conclusion that SD can be more effective for MoE than dense models at moderate batch sizes. We begin by formalizing general SD speedup and introducing our new metric target efficiency (Sec. 3.1). Then, we focus on MoEs, analyzing how workload and MoE sparsity collectively affect the number of activated experts and SD speedup (Sec. 3.2). Based on these analyses, we develop a performance model that aligns with GPU results (Sec. 3.3). We further discuss the practical value of our theoretical findings (Sec. 3.4).

Preliminaries. LLM inference time is collaboratively determined by computation and memory access. When an operator is processed on a GPU, memory access and computation operations are pipelined and overlapped, causing the more time-consuming operation to become the bottleneck and determine the overall processing time, as depicted by the roofline model [35, 36]. The roofline model ridge point (RP) of hardware and the arithmetic intensity (AI) of software are defined as Eq. 1. When AI < RP, the system is memory-bound , and adding more computation will not significantly increase processing time. When AI > RP, the system is compute-bound , and increases in computation will directly reflect in processing time. In this paper, when we describe a system as "more memory-bound", we mean AI RP is smaller.

3.1 Formulation of Speculative Decoding Speedup and Target Efficiency

We first formalize the processing time of speculative decoding, denoted as T SD . To generate a sequence of length S , SD goes through R rounds, each containing three stages: ① the draft model proposes γ tokens as specified by the speculation strategy; ② the target model verifies these tokens; ③ rejection sampling [9] discards incorrectly predicted tokens based on logits from target and draft models. We use T T ( b, s ) and T D ( b, s ) to represent the time for once forwarding of the target and draft model, respectively, where b and s are the formal arguments for batch size and the number of tokens to process. Therefore, the time for processing a batch containing B requests is given by:

Then the speedup of SD to normal auto-regression decoding T AR is given by:

S R represents the average length of accepted tokens per SD round, which can be further expressed as σ × ( γ +1) . Here, σ is the ratio of actually generated tokens to the theoretical maximum if all draft tokens were accepted. We note that σ differs from the acceptance rate α commonly referenced in previous works [10, 9, 7], which represents the probability of the target model accepting a new draft token given the prefix. σ can be computed from α as shown in Eq. 5. The numerator follows from [10], and the denominator accounts for all γ draft tokens accepted, plus a bonus token generated during the forward verification pass.

The denominator of Eq. 4 consists of three terms. T D ( B, 1) T T ( B, 1) is the ratio of draft-model forward time over target-model forward time, reflecting the relative volume of draft and target models. It is also kept small (usually less than 1/10 [27, 7, 10]) to ensure the speculation is efficient. T reject T T ( B, 1) is even smaller, since T reject only involves sampling rather than model inference. T T ( B,γ ) T T ( B, 1) , which is the ratio of multi-token forward time over single-token forward time, has the biggest value among these three items and significantly affects the final speedup. As indicated by Eq. 4, its increase causes speedup reduction. Two different factors drive its increase, explaining SD's ineffectiveness under (1) large batches for both dense models and MoE, and (2) MoE with small batches, respectively:

The compute-boundness. The model's T T ( B,γ ) T T ( B, 1) approaches 1 when more memory-bound (smaller batch size B ) but increases to γ when more compute-bound (larger batch size B ).

The extra memory loads. For small B s, T T ( B,γ ) is notably greater than T T ( B, 1) as more experts are activated and need to be loaded. Since the system is still memory-bound at small B s, memory load profoundly determines the processing time.

Therefore, we define target efficiency as T T ( B, 1) T T ( B,γ ) , which helps understand the systemic causes of SD acceleration degradation as listed above. Our experiments shown in Fig. 2 demonstrate that target efficiency consistently reflects the trend of SD speedup variations. Despite the importance of this value, previous works have rarely noticed it, primarily due to differences in research focus. Previous SD research mainly addresses the question by lifting the acceptance rate:

Given the target model, which draft model or algorithm achieve better speedups?

In contrast, our work focuses on the following question by examining target efficiency:

Under the same level of algorithmic optimization, which types of target models or workloads are more favorable for SD?

We believe target efficiency help researchers understand SD more comprehensively. Even when target-draft pairs have the same acceptance rate α , changes in the target model's architecture and the workload can significantly affect overall speedup. By introducing target efficiency, we can decouple algorithmic optimization from systemic optimization, thus helping to identify the systemic acceleration bottlenecks and assess potential speedup.

3.2 Moderate Batch Size Enables Speculative Decoding Speedup for MoE

Although SD is ineffective for MoE under small batches, we demonstrate in this subsection that at moderate batch sizes-an overlooked regime in previous studies-SD speedup increases and benefits more from MoE with higher sparsity. Essentially, when the batch size falls within ranges where

)

T

) /

0

50

100

150

200

250

Number of Tokens (t)

10

20

30

40

50

60

Number of Activated Experts (N(t))

Theoretical

HumanEval

MT-Bench

(a)

0

50

100

150

200

250

Number of Tokens (t)

10

20

30

40

50

60

Number of Activated Experts (N(t))

Theoretical

HumanEval

MT-Bench

(b)

0.0

0.2

0.4

0.6

0.8

1.0

MoE sparsity (

)

0.0

0.2

0.4

0.6

0.8

1.0

Normalized tokens per expert (

Texp

(

T

;

T=1

T=2

T=4

T=8

T=16

T=32

T=64

(c)

all experts are activated but remain far from being assigned adequate workloads, FFNs become memory-bound, presenting an opportunity to leverage computational power almost for free through SD. To demonstrate this, we first formalize the expected number of activated experts, and then show MoE FFNs become more memory-bound as the model becomes sparser.

We use the Bernoulli random variable X to indicate the activation for experts: X i = 1 for expert i being activated, 0 otherwise. For simplicity, we assume X s are i.i.d. Then, the expected number of activated experts N can be expressed as Eq. 6, where E denotes the total expert count and Pr ( X i ) represents the probability that the i th expert is activated.

Given t tokens passed through the MoE gate, then Pr ( X ) is expressed as Eq. 7. K denotes the number of activated experts per token, which is an architectural hyperparameter for MoE:

Therefore, the overall expression of N ( t ) is given by Eq. 8. Our derivation assumes uniformly activated experts, which is reasonable for well-trained MoE models. Load imbalance among experts can lead to routing collapse [5] and decrease computational efficiency in expert-parallel deployment [6], so state-of-the-art MoE models are typically trained with methods like incorporating auxiliary loss [37, 6] to ensure that experts have balanced loads. The experiment results also verified our theoretical derivation of N ( t ) , as shown in Fig. 1a and 1b.

We then solve how many tokens can lead to full activation. Since N ( t ) asymptotically approaches E when t tends to infinity, and in practice N ( t ) should be a finite integer, we deem N ( t ) > τE as almost full activation, where τ is usually a large ratio such as 0.95. We further express K = ρE , where ρ is the sparsity of MoE, then the token threshold T thres can be solve by:

Therefore, when B exceeds T thres , the number of activated experts saturates, causing the Bγ tokens in verification to incur only marginally larger memory access. Having addressed the second factor (namely, extra memory loads) for T T ( B,γ ) T T ( B, 1) 's increase analyzed in Sec. 3.1, we now turn to the potential limitations caused by the first factor of compute-boundness. If such B s make the system computebound, SD would also fail to accelerate MoE effectively. Our answer to this concern is: Sparser MoEs delay the transition from memory-bound to compute-bound when input tokens count increases.

We have obtained that given t tokens, N ( t ) experts are activated. Since each token activates K experts, the number of tokens each expert needs to process on average T exp can be computed as:

Algorithm 1 The Modeling of SD Speedup and Corresponding Fitting Method

1: Measurement Input : A total of m measurements denoted as M . Each M i , i = 1 , 2 , ..., m contains the attributes including batch size B , draft length γ , number of activated experts per token K , total number of experts E , the ratio of accepted token counts to the maximal possible accepted tokens σ , Speedup for the

As proven in Appendix and shown in Figure 1c, given t = T > 1 , T exp ( T ; ρ ) decreases with ρ , indicating that as MoE becomes sparser, each expert processes fewer tokens per parameter loading. Consequently, the system running sparser MoEs is more memory-bound , leading to lower utilization of arithmetic units. The verification stage can therefore leverage these spare resources without notably increasing processing time. In contrast, dense models are extreme cases with ρ = 1 , where the FFN consistently approaches the maximal possible arithmetic intensity of T , and the system transitions rapidly to the compute-bound regime as T increases.

We should note that our conclusion is based on a relatively large MoE FFN portion in the whole model, which holds true for current MoE models whose most parameters are experts. In a hypothetical extreme case where Attention dominates and the MoE FFN is negligible, MoE's sparsity would have only a limited impact on overall system performance as indicated by Amdahl's Law.

3.3 A Modeling Method for Speculative Decoding Speedup

Given the numerous factors affecting final speedup, quantitatively understanding each factor's impact is challenging. Therefore, we developed a modeling method that makes SD speedup results more explainable and transparent . As demonstrated by Eq. 4, the core of modeling SD speedup lies in characterizing the model's forward pass time. Based on theoretical analysis in previous sections, we identified three primary factors affecting forward execution: (1) the roofline model effect, (2) the number of active experts, and (3) expert load. Since GPU execution is dynamic in practice, and not all operators are optimized to their theoretical limits, we introduced several parameters for relaxation. The values of these parameters are then automatically determined by fitting GPU measurements. These factors and their impacts on execution time are examined as follows.

(1) The roofline model effect. It manifests as execution time increases with token counts t , with a growth rate that starts slow, then accelerates, and finally stabilizes. The underlying reasons are as follows. When t is small, parameter loading time exceeds computation time, creating a memory bottleneck. Therefore, given the parameter volume, the memory access time is stable (memorybound regime). As t increases, computation time exceeds parameter loading time and becomes the bottleneck. With fixed arithmetic units in the hardware, computation time scales linearly with computational load (compute-bound regime). To characterize this trend, we design G ( t ; λRP,s ) as shown in Eq. 11, where λRP represents the transition point between regimes, and s controls the increasing rate of execution time. Here, RP follows Eq. 1, and λ is a constant less than 1 that accounts for practical limitations in memory bandwidth utilization. G ( t ) exhibits a gradually increasing slope before the transition point, then shifts to a linear function afterwards, maintaining first-order gradient

continuity at the transition.

(2) The number of activated experts. When it increases, the memory access volume increases, thus adding to the final processing time. We use the derived Eq. 8 of N to characterize how workload and model architecture affects the number of activated experts.

(3) Expert load. This refers to the fact that after token distribution through the MoE gating, each expert processes only a subset of tokens T exp ( t ; ρ ) rather than the entire input token count t . Therefore, we should use G ( T exp ) rather than G ( t ) when applying the roofline model to MoE experts. This corroborates our theoretical conclusion that sparser MoEs delay the transition from memory-bound to compute-bound when input tokens count increases.

For the MoE target model, factors (1), (2), and (3) are all involved. We combine them in a first-order style and introduce parameters bias , k 1 , k 2 , and k 3 to adjust for non-ideal factors in actual GPU execution, with the full expression shown in lines 6 and 8 of Alg. 1. These parameters have clear practical meanings: bias represents the time required to load fixed parameters; k 2 · N represents the time needed to load N activated experts; k 1 · G ( t ) and k 3 · G ( T exp ) describe the incremental trend in execution time as the number of tokens increases. For the draft model, only factor (1) is involved since it is usually dense, with the modeling form shown in line 9 of Alg. 1.

With the expression of SD speedup determined, we fit the measurement inputs to automatically determine the relaxation parameter values, with the optimization criterion being the minimization of Mean Squared Error (MSE) between the model outputs and the ground truth, as shown in line 13 of Alg. 1. By applying these optimized parameters in our model (i.e., the ComputeSpeedup function in line 3), we obtain the complete modeling for SD speedup. An illustrative diagram of this process and more fitting details are provided in Appendix C.

Since our theoretical analyses capture the primary tradeoffs and provide a solid foundation for the modeling, the fitting is very efficient. The fitting results with 21 measurements are displayed in Fig. 4, which show consistent trends with GPU results under various cases. These results validate the reliability of our modeling, thereby establishing it as an effective tool for analyzing the components of the model's forward pass and quantitatively understanding the tradeoffs between different factors. As shown in Sec. 4.2, we explain some unexpected results with the help of the model.

3.4 Practical Values of Theoretical Findings

While previous sections focuses on theoretical analysis, this section demonstrates how these findings translate to practical speedups. Our theoretical analysis has already revealed that SD speedup for MoE is most effective at moderate batch sizes, with its trend initially increasing and then decreasing. We discuss their practical values considering both basic deployment and extended configurations.

Basic deployment. (1) Moderate batch sizes are common in private serving, which are increasingly adopted for data security, with representative applications like enterprise in-house chatbots. (2) When latency requirements are strict, large batch sizes are often not feasible. LLM serving must satisfy multiple service level objectives (SLOs) [38], including time-to-first-token (TTFT) and time-peroutput-token (TPOT). Large batches reduce per-request computational resources, causing latency violations. In such cases, moderate batch sizes are common. (3) Our work actually reveals that SD on MoE relaxes the traditional latency-throughput trade-off . Specifically, MoE models exhibit a regime where SD speedup increases (lower latency) alongside larger batch sizes (higher throughput).

From the model's perspective, moderate batch sizes represent an " efficiency gap " in MoEs. At this scale, all parameters must be loaded (unlike small batches with selective expert activation), yet GPU FLOPs are not fully utilized (unlike large batches). Our findings provide a novel perspective to address this efficiency challenge without compromising model quality.

Extended configurations. We consider typical system optimizations on MoE like offloading and expert parallelism (EP) . When MoE models exceed GPU memory capacity, FFN parameters are offloaded to CPU memory [39]. This degrades parameter loading bandwidth from GPU memory bandwidth to much lower PCIe bandwidth, making the system more memory-bound. Consequently,

0

20

40

60

80

100

Batch size

B

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

A800, humaneval, temp=0,

GPU results

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

Target Efficiency

Target Efficiency

(a)

0

20

40

60

80

100

Batch size

B

0.8

1.0

1.2

1.4

1.6

1.8

2.0

2.2

2.4

Speedup

H800, humaneval, temp=0,

GPU results

0.4

0.5

0.6

0.7

0.8

Target Efficiency

Target Efficiency

(b)

0

20

40

60

80

100

Batch size

B

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

A800, humaneval, temp=0,

=4, Mixtral

GPU results

0.4

0.5

0.6

0.7

0.8

0.9

Target Efficiency

Target Efficiency

(c)

additional computation does not significantly increase processing time, creating favorable conditions for SD. Notably, existing optimizations like expert prefetching [23, 24] and caching [25, 26] lose efficiency under moderate batch sizes since nearly all experts are activated.

Our findings are also compatible with EP. In EP, experts are distributed across multiple GPUs, which affects neither N ( t ) nor T exp , making our previous analyses remain valid. Since components besides MoE FFN are also parallelized, MoE FFN continues to constitute a significant portion of processing time, allowing memory-boundness effects to remain observable in end-to-end performance. Notably, under extensive EP configurations, the inefficiency of SD for MoE at a small batch size may vanish, considering the additional memory bandwidth offered by large amounts of EP GPUs.

4 Experiments

Models and datasets. We evaluated two MoE configurations with SD: Qwen2-57B-A14B-Instruct with Qwen2-0.5B-Instruct [40], representing same-family drafting, and Mixtral-8x7B-Instructv0.1 [41] with Eagle [7], representing specialized speculation heads. When we need to examine MoEs with different sparsity, we modify the num_experts_per_token in the model's config.json file. For comparison with dense models, we use Opt-30b and Opt-350m [42] as the target and draft models. Models are evaluated on HumanEval [43] and MT-bench [44] datasets for code generation and conversation tasks, following previous works [7, 45, 11]. The tokenized prompt lengths range from 38 to 391 tokens for HumanEval and 5 to 356 tokens for MT-bench.

Frameworks and hardware. We used the existing vllm [46] framework for our experiments to verify theoretical predictions. Vllm supports batched speculative decoding, cudagraph optimization, and reports comprehensive data such as T D , T T , T reject and σ , thus being suitable for our experiments. To prevent unstable performance at the beginning, all data were obtained by averaging the results from the last five of the total ten runs. We conducted experiments on different hardware platforms including 2xA800, 2xH800, 4xA800, 4xL40.

4.1 Speedup Trend of Speculative Decoding for MoE

Figure 2 plots the end-to-end SD speedup (left y-axis) for MoE across various settings, validating our theoretical prediction about acceleration behavior. As batch size grows, speedup initially increases

due to expert loading saturation, and then decreases due to compute-boundness. We denote the maximal speedup across batch sizes as x and summarize the results in Table 1. For both models, SD achieves higher acceleration with longer γ for tasks with more predictable patterns (e.g., code generation) or less randomness (e.g., lower temperature), aligning with conclusions from previous research. Figure 5 in Appendix A.1 further presents SD speedup trends under more settings, including individual runs and their mean to show the statistical significance of our findings.

We further evaluate Qwen2-57B-A14B-Instruct on multiple hardware platforms (Table 2) to verify the generality of our conclusions. Combined with results of Qwen2 in Table 1, two observations can be made: (1) GPUs with higher ridge points yield larger SD speedups (e.g., 2×A800 vs. 2×H800, 4×A800 vs. 4×L40), since they provide more arithmetic units for verification. (2) Scaling from 2×A800 to 4×A800 reduces absolute runtimes ( T AR and T SD ), but the SD speedups slightly degrade. This is because the large model benefits from inter-GPU parallelization, whereas the small draft model remains single-GPU, making its relative forward cost higher.

0

20

40

60

80

100

Batch size

B

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Target Efficiency

humaneval, temp=0,  =4

MoE (Qwen2-57B-A14B-Instruct)

Dense model (Opt-30b)

Figure 2 also highlights the effectiveness of our metric target efficiency. It is computed as T T ( B, 1) T T ( B,γ ) as explained in Sec. 3.1, where both T T ( B, 1) and T T ( B,γ ) are obtained from vllm runtime logs. Target efficiency values are annotated on the right y-axis, showing a consistent trend with the end-to-end speedup. In contrast, the acceptance rate across batch sizes merely fluctuates within a small range , unable to effectively explain the dramatic changes in speedup.

We further compare MoE and dense models via target efficiency (whose validity is established above) to avoid interference from acceptance rate. As shown in Figure 3, the target efficiency for MoE first increases and then decreases, while that for the dense model decreases continuously. Consequently, although SD for MoE is less effective with small batches, it exhibits stronger potential across a wider range of larger batch sizes.

Regarding end-to-end performance, SD speedups become more pronounced for MoE when the batch size exceeds 16, as supplemented in Figure 6 in Appendix A.2.

4.2 Impact of MoE Sparsity and Validation of Modeling Method

To evaluate MoE sparsity's impact on SD acceleration, we varied the number of activated experts per token ( K ) of Qwen2-57B-A14B-Instruct. Directly changing K without training affects the target model's performance and speculation accuracy, so we adjust the speedup by multiplying the raw speedup with σ K =8 σ K , whose rationale is exhibited by Eq. 4. Fig. 4 shows the adjusted speedup alongside our modeling results for comparison. The parameters used in the modeling are decided using 21 GPU measurements, as explained in Section 3.3. The impact of measurement selection for parameter fitting on the modeling's reliability is supplemented in Appendix C.

𝜸 = 𝟐

𝜸 = 𝟒

There are three key observations. First, the modeling consistently aligns with experiment results across varying sparsity ( ρ ) and draft length ( γ ), validating our modeling's reliability.

Second, while the SD speedup in most MoEs exhibits an initial increase followed by a decrease, very sparse MoEs ( K = 1 , 2 ) show continuously decreasing speedup. This appears to conflict with the theoretical analysis, but after examining the components of our modeling, we identified the reason as follows. These very sparse MoEs have a disproportionately low ratio of FFN, thus making the memory-boundness of MoE FFN hard to manifest systematically as indicated by Amdahl's Law. The Qwen2-57B-A14B model is designed based on K = 8 , but by reducing K to 1 or 2, we actually artificially synthesize a model where Attention dominates. In practice, however, sparser MoEs typically incorporate more FFN parameters to maintain a balanced ratio between FFN and Attention components, resulting in acceleration patterns more similar to K = 8 cases.

Finally, as MoE models become sparser, the system's transition from memory-bound to computebound is delayed. This is evidenced by two phenomena in Fig. 4: With smaller ρ , (1) the batch size for the maximal speedup ( x ) becomes larger; (2) the range of batch sizes that maintain speedup above a certain decay threshold (annotated by the brown dashed line in Fig. 4 for x / √ 2 ) is wider. These validate our theoretical analysis and indicate that SD has broader applicability in sparser MoEs.

5 Conclusion and Limitation

This work challenges the conventional wisdom that speculative decoding (SD) cannot effectively accelerate MoE models. We demonstrate that under moderate batch sizes, sparser MoEs gain greater speedup from SD due to their memory-bound FFNs, a conclusion supported by both theoretical and experimental analysis. To navigate the complex factors influencing speedup, we develop a reliable, interpretable SD model and introduce target efficiency to elucidate the impact of model architecture and workload. Our work offers a new perspective for accelerating MoEs, particularly in memory-constrained or moderate-batch serving. While our analysis assumes the KV-cache is smaller than parameters, the behavior when it dominates is analyzed by MagicDec [34]; these two works combine to give a more comprehensive view of SD across batch sizes.

Acknowledgments and Disclosure of Funding

This work is supported by the National Key Research and Development Program of China under Grant (2024YFB4405400).

References

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 , 2024.

Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 , 2024.

Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066 , 2024.

Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 , 2024.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 , 2017.

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 , 2020.

Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077 , 2024.

Anish Saxena, Po-An Tsai, Hritvik Taneja, Aamer Jaleel, and Moinuddin Qureshi. Utility-driven speculative decoding for mixture-of-experts, 2025.

Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318 , 2023.

Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning , pages 19274-19286. PMLR, 2023.

Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads, 2024. URL https://arxiv. org/abs/2401.10774 , 2024.

Rahul. What are private llms? running large language models privately - privategpt and beyond, 2024. https://zilliz.com/learn/what-are-private-llms .

Esther Julie. What is a private llm and why should you build one?, 2024. https://www.inoru. com/blog/what-is-a-private-llm-and-why-should-you-build-one/ .

Hanbo Huang, Yihan Li, Bowen Jiang, Lin Liu, Bo Jiang, Ruoyu Sun, Zhuotao Liu, and Shiyu Liang. Position: On-premises llm deployment demands a middle path: Preserving privacy without sacrificing model confidentiality, 2025.

Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router. arXiv preprint arXiv:2410.12013 , 2024.

Jaeseong Lee, Aurick Qiao, Daniel F Campos, Zhewei Yao, Yuxiong He, et al. Stun: Structuredthen-unstructured pruning for scalable moe pruning. arXiv preprint arXiv:2409.06211 , 2024.

Elias Frantar and Dan Alistarh. Qmoe: Practical sub-1-bit compression of trillion-parameter models. arXiv preprint arXiv:2310.16795 , 2023.

HamidReza Imani, Abdolah Amirany, and Tarek El-Ghazawi. Mixture of experts with mixture of precisions for tuning quality of service. arXiv preprint arXiv:2407.14417 , 2024.

Felipe Cruz Salinas, Kenichi Kumatani, Robert Gmyr, Linquan Liu, and Yu Shi. Knowledge distillation for mixture of experts models in speech recognition. Technical report, Technical Report MSR-TR-2022-6, Microsoft Research, May 2022. https://www . . . , 2022.

Fangxun Shu, Yue Liao, Le Zhuo, Chenning Xu, Lei Zhang, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, et al. Llava-mod: Making llava tiny via moe knowledge distillation. arXiv preprint arXiv:2408.15881 , 2024.

Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. Moe-i 2 : Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition. arXiv preprint arXiv:2411.01016 , 2024.

Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy. arXiv preprint arXiv:2310.01334 , 2023.

Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. Moe-infinity: Activation-aware expert offloading for efficient moe serving. arXiv e-prints , pages arXiv-2401, 2024.

Shuzhang Zhong, Ling Liang, Yuan Wang, Runsheng Wang, Ru Huang, and Meng Li. Adapmoe: Adaptive sensitivity-based expert gating and management for efficient moe inference. In Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design , pages 1-9, 2024.

Xin He, Shunkang Zhang, Yuxin Wang, Haiyan Yin, Zihao Zeng, Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, Ivor Tsang, and Ong Yew Soon. Expertflow: Optimized expert activation and token allocation for efficient mixture-of-experts inference. arXiv preprint arXiv:2410.17954 , 2024.

Peng Tang, Jiacheng Liu, Xiaofeng Hou, Yifei Pu, Jing Wang, Pheng-Ann Heng, Chao Li, and Minyi Guo. Hobbit: A mixed precision expert offloading system for fast moe inference. arXiv preprint arXiv:2411.01433 , 2024.

Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 , pages 932-949, 2024.

Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. Rest: Retrieval-based speculative decoding. arXiv preprint arXiv:2311.08252 , 2023.

Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. Specexec: Massively parallel speculative decoding for interactive llm inference on consumer devices. Advances in Neural Information Processing Systems , 37:16342-16368, 2024.

Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858 , 2024.

Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprint arXiv:2503.01840 , 2025.

Xiaoxuan Liu, Jongseok Park, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Chen Zhang, Kuntai Du, Xiangxi Mo, Kaichao You, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Turbospec: Closed-loop speculation control system for optimizing llm serving goodput, 2025.

Qidong Su, Christina Giannoula, and Gennady Pekhimenko. The synergy of speculative decoding and batching in serving large language models, 2023.

Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. Magicdec: Breaking the latencythroughput tradeoff for long context generation with speculative decoding. arXiv preprint arXiv:2408.11049 , 2024.

Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. Llm inference unveiled: Survey and roofline model insights, 2024.

Georg Ofenbeck, Ruedi Steinmann, Victoria Caparros, Daniele G Spampinato, and Markus Püschel. Applying the roofline model. In 2014 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) , pages 76-85. IEEE, 2014.

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022.

Zhibin Wang, Shipeng Li, Yuhang Zhou, Xue Li, Rong Gu, Nguyen Cam-Tu, Chen Tian, and Sheng Zhong. Revisiting slo and goodput metrics in llm serving, 2024.

kvcache ai. Ktransfromers: A flexible framework for experiencing cutting-edge llm inference optimizations, 2025. https://github.com/kvcache-ai/ktransformers/tree/main .

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, et al. Qwen2 technical report, 2024.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, et al. Evaluating large language models trained on code, 2021.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.

Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. Learning harmonized representations for speculative sampling, 2025.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023.

NVIDIA Docs Hub. Matrix multiplication background user's guide. https://docs.nvidia. com/deeplearning/performance/dl-performance-matrix-multiplication/index. html .

Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024.

Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Tian Tang, Qinyu Xu, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Ziren Wang, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. Nanoflow: Towards optimal large language model serving throughput, 2025.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to Abstract and Introduction. The three items in the contribution list are explained in detail in Section 3 and 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Section 5. The last several sentences discuss our limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please refer to Section 3 and Appendix in the supplementary materials. We provide the meaning of all variables and assumptions with the proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to Section 4 for experiment settings. Besides, we provide complete codes as the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the complete codes and data as supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Section 4 and Appendix in supplementary materials. The experiment details can also be found in supplementary codes.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper does not include error bars. However, all results are averaged over 5 runs to avoid randomness as mentioned in Section 4. Besides, the Appendix in supplementary materials further discusses the impact of data selection on model fitting.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to Section 4. Besides, the readme file in supplementary codes also describes the computer resources needed for reproduction.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research focuses on speculative for MoE, conforming with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Please refer to Section 4. We cite the vllm framework as its GitHub main page suggests.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

B-lnstruct

Average

Run 1

Run2

Run 3

Run 4

C

D

Run 5

S

4B-Instruct

Average

Run 1

Run 2

Run 3

Run 4

Run5

B-lnstruct

Average

Run 1

Run 2

Run3

Run 4

Run 5

S

B-lnstruct

Average

Run 1

Run 2

Run 3

Run 4

Run 5

struct-v0.1

Average

Run 1

Run 2

Run3

Run 4

C

Run 5

S

struct-v0.1

Average

Run 1

Run2

Run 3

Run 4

Run5

ruct-vo.1

Average

Run 1

Run2

Run 3

Run 4

Run5

C

a

S

ruct-v0.1

Average

Run 1

Run 2

Run3

Run 4

Run5

A Supplementary Experimental Results

This section presents additional experimental results referenced in Section 4, which are included here due to space limitations.

A.1 Trends of SD speedup under more configurations

Figure 5 presents additional trends of SD speedup across different datasets, temperatures, and model types, serving as a supplement to Figure 2. The results demonstrate that SD speedup exhibits a consistent first-increase-then-decrease pattern, which aligns well with our theoretical analysis.

To confirm the statistical significance of our findings, we also present the five individual runs that constitute the averages in Figure 5. The variance across different runs is minimal, which is expected since the random seed is fixed across all runs to ensure identical workloads.

While the overall trend follows the first-increase-then-decrease pattern, local fluctuations are observable in the curves. For instance, Figure 5(c) exhibits a sawtooth-like decreasing trend. This phenomenon can be attributed to GPU quantization effects , as documented in NVIDIA's documentation [47]. When dimensions are not evenly divisible by the GPU's native tile sizes, computational performance degrades. AR decoding is more sensitive to this effect than SD, making the time ratio of

0

20

40

60

80

100

Batch size

B

0.8

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

Temperature: 0.0, Draft length: 2

Opt-30b

Qwen2-57B-A14B-Instruct

0

20

40

60

80

100

Batch size

B

1.00

1.25

1.50

1.75

2.00

2.25

2.50

2.75

Speedup

Temperature: 0.0, Draft length: 3

Opt-30b

Qwen2-57B-A14B-Instruct

0

20

40

60

80

100

Batch size

B

1.0

1.5

2.0

2.5

3.0

Speedup

Temperature: 0.0, Draft length: 4

Opt-30b

Qwen2-57B-A14B-Instruct

0

20

40

60

80

100

Batch size

B

0.8

1.0

1.2

1.4

1.6

Speedup

Temperature: 1.0, Draft length: 2

Opt-30b

Qwen2-57B-A14B-Instruct

0

20

40

60

80

100

Batch size

B

0.6

0.8

1.0

1.2

1.4

1.6

Speedup

Temperature: 1.0, Draft length: 3

Opt-30b

Qwen2-57B-A14B-Instruct

0

20

40

60

80

100

Batch size

B

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

Temperature: 1.0, Draft length: 4

Opt-30b

Qwen2-57B-A14B-Instruct

AR to SD (namely, SD speedup) fluctuate. Despite these local variations, the overall speedup trend follows our theoretical predictions, confirming the validity of our conclusions.

A.2 End-to-end speedup comparison between MoE and dense models

To isolate the effects of acceptance rate variations and enable a clearer focus on system bottlenecks, we have compared MoE and dense models using target efficiency in Section 4.1. In this section, we further compare their end-to-end speedup across various settings in Figure 6 as a supplement.

Two key observations emerge from the experiment results. First, while SD speedups for MoE models initially increase before declining, SD speedups for dense models continue to decrease. Consequently, SD achieves more substantial end-to-end speedups for MoE models at moderate batch sizes, which aligns with the trend in Figure 3 in Section 4.1. Second, the extent to which SD favors MoE over dense models varies across different configurations. For instance, at temperature = 1 (second row), SD demonstrates greater relative benefits for MoE compared to temperature = 0 (first row). This variation stems from diverse acceptance rates under different settings, which can obscure the observation of systemic bottlenecks. In summary, target efficiency serves as a reliable comparison metric while controlling for the confounding effects of algorithmic optimizations.

B Proof of T exp ( T ; ρ ) 's Trend with Varying ρ

In Section 3.2, Fig. 1(c) demonstrates that: Given input token count t = T > 1 , the number of tokens each expert processes on average T exp ( T ; ρ ) = ρT 1 -(1 -ρ ) T decreases as ρ decreases. We prove this by showing d( T exp ( T ; ρ )) d ρ > 0 when T > 1 .

Since ρ represents MoE sparsity ∈ (0 , 1) , the original proposition is equivalent to proving:

Note that F ( ρ ; T ) → 1 as ρ → 0 . Therefore, if we can prove that F ( ρ ; T ) decreases as ρ increases from 0 to 1, then the original proposition is proven. We demonstrate this by computing d( F ( ρ ; T )) d ρ :

Theoretical analysis

(Section 3.1 / 3.2)

Fitting under least square criterion

(Section 3.3 / Appendix C.2 / C.3)

Computing the SD speedup

(Section 3.3 / Appendix C.2 / C.3)

SD speedup expression with relaxation parameters

Real Measurements

(Workloads & SD speedup)

Optimal parameters

Workloads

SD speedup

Theory

Profiling

Runtime

When T > 1 , d( F ( ρ ; T )) d ρ < 0 . This confirms that F ( ρ ; T ) decreases as ρ increases, which proves our original proposition: when T > 1 , T exp ( T ; ρ ) decreases as ρ decreases.

C More details about the Modeling Method

The main design considerations and expressions of our modeling method have been presented in Section 3.3. In this section, we provide additional content on the following topics to give a more comprehensive view of the modeling method:

Description and an illustrative diagram of the modeling process. (Appendix C.1)

Fitting Details of the modeling shown in Figure 4 in Section 4.1. (Appendix C.2)

How the modeling is affected by alternative measurement selection. (Appendix C.3)

The value of our modeling is twofold. On one hand, it achieves alignment with real measurements with only a small number of simple parameters, thus validating the correctness of our theoretical analyses. On the other hand, it provides the decomposition of various factors in the end-to-end results, making the entire SD acceleration process explainable and transparent.

C.1 Description and Overview of the Modeling Process

Figure 7 presents the overall diagram of our modeling method. Building upon the theoretical analyses in Sections 3.1 and 3.2, we derive an expression for SD speedup as a function of workloads. This expression contains several relaxation parameters to be determined for complete modeling. We determine these parameters through empirical profiling. We first collect a small set of real measurements comprising various workloads and their corresponding SD speedups. We then perform parameter fitting using these measurements under the least squares criterion to obtain optimal parameter values. Since our theoretically-derived SD speedup expression already captures the fundamental performance tradeoffs, the fitting process is computationally lightweight and robust, as will be demonstrated in Appendix C.2 and C.3. Once the optimal parameters are obtained, the resulting expression can predict SD speedups for arbitrary workloads.

We now explain why profiling is necessary and why we cannot derive the complete SD speedup expression purely through theoretical analysis, examining both software and hardware considerations.

Software considerations: Actual execution times can deviate significantly from theoretical predictions for complex operators with diverse implementations. On GPUs, GEMM operations are indeed predictable due to their regular structure and highly optimized implementations. However, prediction becomes challenging for operators such as Attention, which involve customized kernel optimizations (e.g., FlashAttention1/2, eager attention, SDPA attention) and operator fusion strategies (incorporating various nonlinear layers or positional encodings such as RoPE and its variants). To illustrate this complexity, we examine profiling results from Qwen2-57B-A14B (hidden size 3584) and Mixtral-8x7B (hidden size 4096). For FFN, Qwen takes a shorter time than Mixtral (143us vs 226us), aligning with their relative hidden sizes. However, for Attention, Qwen takes a longer time than Mixtral (271us vs 115us), contradicting the theoretical expectation based on hidden size scaling.

Hardware considerations: GPU microarchitectures vary across different series, which can greatly impact execution times. For instance, attention efficiency depends heavily on hardware-aware

programming optimizations, while different GPUs vary in cache configurations and thread-memory interaction patterns across memory hierarchy levels. By taking advantage of new capabilities in modern hardware, FlashAttention-3 successfully increases GPU utilization from 35% to 75% on H100 GPUs [48]. Moreover, many critical hardware details remain undisclosed by GPU vendors, making theoretical predictions impractical.

Therefore, modeling speedup trends with pure analytical methods requires case-by-case analysis for different operator implementations and GPU microarchitectures. In contrast, the hyperparameter approach offers a more generalizable paradigm and is easy to use: all parameters possess clear physical interpretations, only minimal profiling data are required, and the computational overhead is low. Our method achieves a balance between effectiveness and practicality: on one hand, it captures the primary performance drivers (i.e. the number of activated experts and roofline trends); on the other hand, it avoids getting entangled in low-level implementation complexities. This approach is also used by other system optimization frameworks such as NanoFlow [49], which similarly adopt a two-stage strategy of profiling followed by runtime execution.

C.2 Fitting Details of the Modeling shown in Figure 4

We first explain how we select the 21 measurements. Due to GPU resource and time constraints, we obtained a total of 228 GPU measurements across varying experimental settings, including 6 different numbers of activated experts per token ( K ), 2 draft lengths ( γ ), and 19 batch sizes ( B ). These measurements are sorted first by K , then by γ within each K group, and finally by B within each ( K,γ ) combination, forming the total dataframe ( df ). We then uniformly sampled measurements from this sorted dataset with a fixed stride, namely M = df[begin:end:11] . This sampling strategy enables our selected measurements to contain different settings, making the modeling more robust.

The SD speedup function (namely, ComputeSpeedup defined in line 3 of Algorithm 1) is nonlinear. To optimize its MSE, we employed the scipy.optimize.least_squares function with the Trust Region Reflective (TRR) algorithm. TRR is an optimization method for bound-constrained nonlinear least squares problems that combines trust region methods with reflection techniques. It constructs quadratic models within trust regions and uses reflection strategies near boundaries to maintain feasibility while ensuring convergence. The fitting process for these 21 data points is efficient, completing in approximately 0.114 seconds. Our modeling incorporates 10 parameters requiring relaxation, with their search boundaries specified as follows:

bias : It represents the time required to load the dense parameters of the target model. We denote the model's non-FFN parameter count as V dense . Consequently, the theoretical minimum loading time can be calculated as bias min = V dense × bitwidth peak memory bandwidth . For the upper bound of the relaxation range, we set bias max = 5 × bias min .

k1 : It adjusts the intensity of the roofline effect of dense components. It should be larger than 0 to ensure the execution time increases as the token count increases. We don't set a definite upper limit for k 1 , as its value is affected by other parameters. Given the hardware with fixed arithmetic units, the execution time grows linearly with the token count in the compute-bound regime. As shown in line 6 of Algorithm 1, k 1 appears as a coefficient in the term k 1 · G ( t ; λ, s ) , whose gradient in the compute-bound regime is k 1 · ln ( s ) · s λRP . As s approaches 1, k 1 needs to continuously increase to counterbalance ln ( s ) that approaches 0.

k2 : It represents the time required to load one expert. Given a target model, we denote the parameter count per expert as V exp . Consequently, the theoretical minimum loading time can be calculated as k2 min = V exp × bitwidth peak memory bandwidth . For the upper bound of the relaxation range, we set k2 max = 5 × k 2 min .

k3 : It adjusts the intensity of the roofline effect of sparse components. Similar to k1 , we set k3 min = 0 and k3 max = inf .

draft_bias : It represents the time required to load the dense draft model. We denote the draft model's parameter count as V draft . Consequently, the theoretical minimum loading time can be calculated as draft_bias min = V draft × bitwidth peak memory bandwidth . For the upper bound of the relaxation range, we set draft_bias max = 5 × draft_bias min .

draft_k : It adjusts the intensity of the roofline effect of the dense draft model. Similar to k1 , we set draft_k min = 0 and draft_k max = inf .

reject_bias : It represents the fixed overhead when performing rejection sampling. Vllm reports its elapsed time during SD, and we denote the maximum across measurements as T rej . We then set reject_bias min = 0 and reject_bias max = T rej .

reject_k : It represents the incremental processing time in rejection sampling as the input token count increases. For simplicity, we set reject_k min = 0 and reject_k max = T rej just like reject_bias .

λ : It represents the ratio of the empirical ridge point to the theoretical ridge point. Since memory bandwidth is typically less utilized than arithmetic units, we set λ min = 0 . 2 and λ max = 1 .

s : It adjusts the growing rate of execution time as input token count increases. Since s serves as the base of G ( t ) , it must exceed 1 to ensure monotonic growth. However, s should not be too large, as it would result in an excessively steep growth rate. In experiments, we set s min = 1 and s max = 2 .

C.3 Exploration of Alternative Measurement Selection

In this section, we demonstrate the impact of varying the number ( m ) of measurements used for fitting on the modeling results. Given that our model incorporates 10 parameters, a minimum of 10 profiling data points ( m ≥ 10 ) are required to determine all parameters. We present the modeling fitting with m ranging from 10 to 228. The data selection method follows the stride-based approach described in the previous section, specifically M = df[begin:end: stride ] . Measurement count m and stride satisfy the following relation: m = ⌈ 228 /stride ⌉ .

We present the MSE values of different m s and their corresponding fitting figures in Table 3. We also list the distinct batch sizes involved in the selected measurements, which helps explain why some configurations show inferior model fit. Due to integer division constraints, m s are not continuous at larger magnitudes. Generally, the modeling fits well with the real measurements, except for m = 10 , 12 , 13 . The reasons are as follows. When m = 10 , the number of measurements equals the parameter count, resulting in insufficient data for robust fitting. When m = 12 and m = 13 , the distribution of the measurement data is biased. With stride-based selection, measurements at m = 12 and m = 13 demonstrate notable gaps in batch size coverage (specifically, m = 12 does not include batch sizes greater than 40, while m = 13 does not include batch sizes within 1 ∼ 24). Their MSE values are larger than that of m = 11 , despite the latter containing fewer data points for fitting. Based on this analysis, we recommend prioritizing uniform data distribution when selecting measurements, as this approach enables the development of more reliable models even with smaller datasets.

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

1.6

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

1.6

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.00

1.25

1.50

1.75

2.00

2.25

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.00

1.25

1.50

1.75

2.00

2.25

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

1.6

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

1.6

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

1.6

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.00

1.25

1.50

1.75

2.00

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

1.6

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

1.6

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

1.6

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

1.6

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 1,

= 1/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 2,

= 2/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

0.8

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 1,

= 1/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 2,

= 2/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.1

1.2

1.3

1.4

1.5

Speedup

K

= 4,

= 4/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

Speedup

K

= 8,

= 8/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

Speedup

K

= 4,

= 4/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Speedup

K

= 8,

= 8/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 16,

= 16/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.2

1.4

1.6

1.8

2.0

Speedup

K

= 32,

= 32/64,

= 2

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

Speedup

K

= 16,

= 16/64,

= 4

GPU results

Modeling

0

20

40

60

80

100

Batch size (

B

)

1.0

1.5

2.0

2.5

3.0

Speedup

K

= 32,

= 32/64,

= 4

GPU results

Modeling

Peer reviews:
comments: SUMMARY:
This paper revisits speculative decoding (SD) in the context of Mixture-of-Experts (MoE) models and challenges the widely held belief that SD is ineffective for sparse MoEs due to verification overhead. The authors introduce a theoretical and empirical framework that shows under moderate batch sizes, SD can offer greater acceleration for sparse MoEs than dense models. A new metric, target efficiency, is proposed to analyze SD speedup potential independent of the acceptance rate. Experimental results on Qwen2 models and two benchmarks (HumanEval and MT-bench) validate the theory and show up to 2.29× speedup.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The core claim that SD can outperform on sparse MoEs under moderate batch sizes is counterintuitive and well-motivated.
2. The formulation of the target efficiency metric adds analytical clarity

Weaknesses:
1. While Qwen2 is used with MT-bench and HumanEval, additional LLMs (e.g., Llama, Mistral) would strengthen generalization claims.

QUESTIONS:
1. Have the authors considered hybrid metrics that combine target efficiency and acceptance rate to better predict speedup?
2. Beyond moderate batch sizes, SD performance degrades. Are there strategies (e.g., chunked batch processing) that can extend SD benefits in such regimes?
3. The paper uses a fixed draft model. Could training a task-specific or sparsity-aware draft model further improve the speedup or acceptance rate?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors challenge the common belief that speculative decoding (SD) is less efficient for MoE models. With both theoretical justification and empirical experiments, the authors show that SD can be more efficient for MoE when the batch size is moderate and MoE is sparse. \n\nFor the theoretical part, the authors focus on a commonly negligible factor, target efficiency, instead of the SD optimization, and show that SD's efficiency is more related to this target efficiency instead of acceptance rate, which also makes the acceleration process transparent and explainable. \n\nWith practical experiments on a large MoE model (Qwen2-57B-17A), the authors show a good correlation between the observation and theoretical justification. The results also show that SD is less efficient for dense models for moderate batch size.\n\nSTRENGTHS AND WEAKNESSES:\n**Note**: I'm not very qualified for reviewing this paper, since I'm more an engineering researcher. I focus more on the engineering part.\n\n## Strengths\n- The paper is well-written with clear motivation and thorough explanation of the design choice;\n- Different from previous works that mostly focus on the algorithm optimization, this paper focuses on a commonly negligible factor when analyzing SD's efficiency, and show a very good correlation between the formula and the observation.\n\n## Weaknesses\n- The findings are very intuitive from my perspective. For a small batch size, MoE is inclined to memory-bound. For a large batch size, MoE is inclined to computation-bound. So a moderate batch size will be efficient for MoE.\n- The paper shows that SD works well for sparse MoE with a moderate batch size, hinting its limitation.\n\nQUESTIONS:\nI'm open to raise my evaluation score if the authors could properly clarify my 2 weakness points.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies speculative decoding (SD) for MoE-based LLMs from a new perspective: identifying the conditions under which MoE LLMs benefit most from SD. Contrary to the common assumption that SD may not accelerate MoE-based LLMs, the paper shows that SD can be effective at moderate batch sizes, where experts are typically fully utilized. The benefits become larger when the LLMs become sparser. To support the theoretical analysis, the paper introduces a new metric (i.e. target efficiency) and a modeling approach to formulate optimal SD configurations. Experimental results confirm that the theoretical predictions align closely with real-world performance across different GPUs (e.g., 2×A100, 2×H100) and tasks (e.g., HumanEval, MT-Bench).\n\nSTRENGTHS AND WEAKNESSES:\n---\nStrengths\n\ni) Topic: This paper explores SD for MoE and general LLMs from a different perspective—identifying the conditions under which SD provides the most benefit. The topic is timely and interesting, addressing an important gap in current research.\n\nii) Methodology: To support the theoretical analysis, the paper introduces a new metric (i.e. target efficiency) and a modeling approach to derive optimal SD configurations. The formulation is well-motivated and technically sound.\n\niii) Experiments: Empirical results demonstrate close alignment between the theoretical predictions and real-world performance across various GPUs (e.g., 2×A800, 2×H800) and benchmarks (e.g., HumanEval, MT-Bench).\n\n---\nWeaknesses\n\nThe paper would be stronger with more comprehensive evaluations across a broader range of LLMs and hardware configurations.\n\ni) Model diversity: The current experiments are based on a single pair of target and draft models—Qwen2-57B-A17B-Instruct and Qwen2-0.5B-Instruct—which limits the generalizability of the findings. The authors are encouraged to include additional representative model pairs, ideally with varying sizes and architectures.\n\nii) Hardware settings: All experiments were conducted on 2xA800 or 2xH800 GPUs. While informative, these settings may not fully reflect real-world deployment scenarios, which often involve distributed environments with more inter-GPU communication or different compute-memory tradeoffs (e.g., H800 vs. L20).\n\nQUESTIONS:\nN.A.\n\nLIMITATIONS:\nLimitations (e.g. longer context/KV caches) have been discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper applies speculative decoding (SD) to sparse Mixture-of-Experts (MoE) models for inference acceleration. The authors demonstrate that the combination yields higher speedups at moderate batch sizes. The paper introduces a new metric called target efficiency for speedup measurment in MoE models. Evaluation results using Qwen2-57B-A17B-Instruct as verifier and Qwen2-0.5B-Instruct as light-weight drafter shows up to 2.29$\\times$ speedup.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n- The paper studies the niche area at the intersection of MoE and SD, which is a timely topic - MoE brings state-of-the-art generation quality and SD is well-established for speedup at inference time.\n- The paper introduces a new SD performance metric for MoE setting, namely target efficiency, which goes beyond commonly known acceptance rate by measuring how much verifier work is converted into useful tokens.\n- The theoretical modelling seems to be sound.\n\nWeaknesses:\n\n- My main concern is that the evaluation appears to be narrow as only one sparse model was evaluated (although OPT is included for dense model case). Given this, and the standard that NeurIPS holds for submissions, I will have to request\n- The evaluation doesn’t seem to include tree-based speculative decoding e.g. Eagle, Eagle-2, which are at the top of the SD leader-board according to SpecBench (https://github.com/hemingkx/Spec-Bench/blob/main/Leaderboard.md). It would hence be useful to include analysis of such tree-based method.\n\nQUESTIONS:\n- The authors states multiple times that they challenge the conventional belief that speculative decoding cannot effectively accelerate MoEs, where the “belief” is more of an untested assumption than a well-established empirical claim. It would be useful to cite relevant prior work on this.\n- To help illustrate the framework, the authors could consider including a system diagram to show the details of MoESD workflow and design?\n- Minor point: It would be helpful to walk-through Algo 1 in the text.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the performance of speculative decoding for Mixture-of-Experts (MoE) models. Specifically, the paper shows that speculative decoding is not as helpful for sparse MoE models as it is for dense models for small batch sizes, when the batch size grows, the benefit substantially increases (which is counter to the trend for dense models). To support these claims, the paper provides theoretical analysis that shows that at moderate batch sizes, the number of activated experts saturates, which helps amortize the additional expert loading costs (in terms of memory bandwidth), and allows for more compute-bound inference. The paper also introduces a new intuitive metric -- target efficiency -- that measures the cost ratio for computing logits for 1 token vs. $\\gamma$ tokens (in previous speculative decoding work, this is assumed to be near 1, which is not true for MoE models). Finally, the paper empirically shows the benefits of larger batch sizes on speculative decoding for MoEs with real models.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is clearly written and presents a simple finding: sparse MoE models benefit more from speculative decoding under moderate batch sizes. The idea that the acceptance rate is not the only important measurement --- target efficiency is also very important, especially when the assumptions that it is near 1 is violated --- is also a natural additional metric to look at. This could potentially be a valuable finding for practitioners, if it was not already known.\n\nThat said, while the paper is well-executed, it is not particularly novel. The core finding, that speculative decoding benefits from instances of memory-bound workloads in MoE models, can be seen as a logical extension of established performance principles. The theoretical analysis, while effective, supports this by applying existing concepts like the roofline model and probability theory, rather than providing new theoretical constructs. There are also some minor gaps in the experimental presentation; the comparison to dense models is indirect, using the \"target efficiency\" metric instead of a direct end-to-end speedup analysis, and the plots lack error bars, making it somewhat difficult to assess the statistical significance of the results.\n\nQUESTIONS:\n- Is there any way to predict what the optimal batch size might be based on your theory, vs through hyper-parameter tuning?\n- The comparison to dense models in Figure 2c is insightful but indirect, as it only shows target efficiency.  To make your claim that MoEs can benefit more from SD than dense models more concrete, could you provide the end-to-end speedup graph for the dense model, analogous to Figures 2a and 2b?\n- The performance curves in your figures show noticeable fluctuations. While you mention averaging results over five runs, can you also add error bars to Figures 2 and 3?\n- Given that MoEs require larger batch sizes, what is the effect on latency, vs throughput?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}