Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference
Authors: Nadav Timor
Jonathan Mamou
Daniel Korat
Moshe Berchansky
Oren Pereg
Moshe Wasserblat
Tomer Galanti
Michal Gordon-Kiwkowitz
David Harel
Venue: ICLR 2025 Poster (2025)
Keywords: inference algorithms for generative models
LLM inference
speculative decoding
Primary area: generative models

Abstract: This paper introduces *distributed speculative inference (DSI)*, a novel inference algorithm that is provably faster than speculative inference (SI) [leviathan2023, chen2023, miao2024, sun2025, timor2025] and standard autoregressive inference (non-SI). Like other SI algorithms, DSI operates on frozen language models (LMs), requiring no training or architectural modifications, and it preserves the target distribution. Prior studies on SI have demonstrated empirical speedups over non-SI—but rely on sufficiently fast and accurate drafters, which are often unavailable in practice. We identify a gap where SI can be slower than non-SI if drafters are too slow or inaccurate. We close this gap by proving that DSI is faster than both SI and non-SI—given any drafters. DSI is therefore not only faster than SI, but also unlocks the acceleration of LMs for which SI fails. DSI leverages *speculation parallelism (SP)*, a novel type of task parallelism, to orchestrate target and drafter instances that overlap in time, establishing a new foundational tradeoff between computational resources and latency. Our simulations show that DSI is 1.29-1.92x faster than SI in single-node setups for various off-the-shelf LMs and tasks. We open-source all our code.

Published as a conference paper at ICLR 2025
DISTRIBUTED SPECULATIVE INFERENCE (DSI):
SPECULATION PARALLELISM FOR PROVABLY FASTER LOSSLESS LANGUAGE MODEL INFERENCE
Nadav Timor*
Jonathan Mamou
Daniel Korat
Moshe Berchansky Intel Labs
Weizmann Institute of Science
Intel Labs
Intel Labs
Oren Pereg Intel Labs
Moshe Wasserblat
Tomer Galanti
Michal Gordon
Intel Labs
Texas A&M University
Weizmann Institute of Science
David Harel
Weizmann Institute of Science
ABSTRACT
This paper introduces distributed speculative inference (DSI), novel inference algorithm that is provably faster than speculative inference (SI) (Leviathan et al.
2023; Chen al. 2023 Miao et al. 2024 Sun al. 2025 Timor al.] 2025 and standard autoregressive inference (non-SI). Like other SI algorithms, DSI operates frozen language models (LMs),, requiring training architectural modifications, and it preserves the target distribution. Prior studies on SI have demonstrated empirical speedups over non-SI-but rely on sufficiently fast and accurate drafters, which are often unavailable in practice. We identify a gap where SI can be slower than non-SI if drafters are too slow or inaccurate. We close this gap by proving that DSI is faster than both SI and non-SI-giveen any drafters. DSI is therefore not only faster than SI, but also unlocks the acceleration LMS for which SI fails. DSI leverages speculation parallelism (SP), novel type of task parallelism, to orchestrate target and drafter instances that overlap in time, establishing new foundational tradeoff between computational resources and latency. Our simulations show that DSI is 1 1.29-1.92x faster than SI in single-node setups for various off shel LMs and tasks. We open-source all our code. |
INTRODUCTION
Generative language models (LMs) have demonstrated unprecedented success across various tasks OOpenAl et al. 2023 Li et al. 2023a Andreas 2022, Bubeck et al. 2023). Reducing the inference latency of these models is critical challenge for improving downstream applications and enabling further test-time scaling (OpenAl al., 2024 Muennighoff al., (025). Faster inference can also drive broader adoption by real-time applications, which often prioritize low latency over other objectives. With the growing availability of hardware and decreasing costs, effectively utilizing more computing power for faster inference is becoming increasingly important.
promising approach to redueing LM inference latency speculative inference (SI), built upon the principles of Burton (1985). SI employs faster drafter models to predict likely token continuations, which are then verifed concurrently using modern hardware's 's data parallelism, known as batching (e.g., on GPUs) (Stern et al. 2018). SI has been widely adopted in practice thanks to novel lossless verification methods, demonstrating empirical speedups of up to 4x over standard autoregressive inference (Leviathan et al. 2023 Chen al. 20233 Miao et al.] 2024 [Snn et al. 2025 Timor
et al. 2025) and increasing throughput multi-request settings (Sadhukhan et al. 2025). The core
www.utdddd-peeuttteeeeeeeeeee wwwpllattveee-eeeeeeeee inference

---

Published as a conference paper at ICLR 2025
advantage of SI is that it can generate more than one token per forward pass of a given target language model.
However, SI relies on sequential draft-then-veriis process, where each verification must be com- pleted before drafting new tokens. As a result, SI is beneficial only if the drafters are sufficiently fast and accurate. If the drafters too slow inaccurate, SI fails to provide speedup- is even slower than standard autoregressive inference. This fundamental limitation of SI has not been addressed in prior work.
Contributions. To overcome the fundamental limitation of speculative inference (SI) as a sequential algorithm, we introduce distributed speculative inference (DSI), novel inference algorithm that parallelizes SI by leveraging speculation parallelism (SP), a new type of task parallelism. Unlike SI, which blocks drafting until verification is complete, DSI overlaps verification with drafting, transforming SI into a non-blocking algorithm and effectively hiding verification latency.
Our key contributions are:
Introducing speculation parallelism (SP): A novel type of task parallelism that eliminates the blocking nature of SI by enabling concurrent verification and drafting using multiple instances of the target and drafter models.
Provable speedup over SI and non-SI: We prove that DSI is always at least as fast as non-SI and strictly faster than both SI and non-SI in expectation.
Broader applicability: DSI accelerates inference making it effective for a wider range of LMs.
with drafters for which SI fails,
Scalability to available hardware: DSI can orchestrate an arbitrary number of GPUs (2 2) by adjusting its lookahead hyperparameter.
Empirical validation: Our simulations show that DSI is 1.29-1.922 faster than SI across various models and tasks in realistic single-node multi-GPU setups.
2
PRELIMINARIES
Below we describe speculative inference and how measure latency. For rigorous definitions of autoregressive language models (LMs) and next-token prediction, we refer the reader to Appendix B
Speculative Inference (SI) is an approach for accelerating the inference of target LM fm (c.g.. member of the GPT series). Such methods use faster LMs fi...., that approximate the target model fm in order to reduce the total inference time. For example, Leviathan et al. (2023);
Chen al. (2023) may reduce the amount of time it takes to generate N > tokens from target model given prompt zigo by using batching as follows. The inference starts by drafting tokens ar} : [1 (1 = fi(rço @ x @ @ x for i E [k]] and 1 < k < N using a faster drafter model f1. Then, the algorithm sends the prompts { { altogether as one input batch to the target model /2. The idea is take advantage of the data parallelism that modern GPUs offer to compute the logits corresponding to the prompts {
1 in parallel, hence faster than computing these k + 1 individual logits sequentially. Given the logits, the algorithm generates [1, k + 1] tokens without additional forward passes. By repeating this process, the algorithm can generate N > + 1 tokens.
Straightforward algorithms of speculative inference are typically lossless in expectation, i.e., they generate tokens from the same distributions the target would generate without speculation. Naive algorithms of speculation guarantee returning the same tokens as the target (Gante 2023; Spector &
Re| 2023 Timor al., 2025). More sophisticated algorithms of speculation might generate different tokens, but their generated tokens follow the distribution of the target (Leviathan et al. 2023 Chen
ctt al. 2023 Miao et al., 2024: Sun et al. 2025 Timor et al. 2025).
To implement distributed algorithms for speculative inference, use multiple processors servers, which are hardware components capable of executing threads. Processors can compute forward passes and sample tokens from the output probability vectors and we assume that threads run in parallel. When using DSI we will run sequences drafter models f ...,.................... Sin+ where the first model takes Zço and returns some token æ the second takes xgço @ x²' as prompt and returns 2 J2 and As such, in order to denote that given thread is computing the output of fj. on
2

---

Published as a conference paper at ICLR 2025
sequence x Jo- <k-1 When thread C5 computes an LMM, we denote the output probability vector by Cy[prob]. If C samples a new token from Cppprob], we denote this token by Cj[new]. For example, thread Cl implementing equation will have
x @ .... - we denote CJ, where J (fi..... jk).
Culprompt] == Igi, Cylprob] == f (C>[prompt)) and Cy[new] ~ Cylprob].
Once thread C. finishes sampling a new token, the thread outputs the concatenation of Cylprompt] and C2 [ne]]. Following the example in equation 3
we have
Cy[retur]] == Cppprompt] @ (C][new]) == (I L1, (1,...,;(++))
A new thread that was initiated by Cy is denoted by Cræci) where J @ (j) is the concatenation of J and (j). The set of all the threads that originate from C5 is {Cr J'' is a nonempty tuple). We assume that terminating a concurrent thread terminates all the threads that originate from it.
Time in this paper is the wall time. We measure the time that passes from the initiation of a task until its termination. A task is a nonempty set of threads, denoted by {C, : E J}. Its time is
Twall ((C)) = max (Timepoint C2 finishes) - min (Timepoint CJ starts).
JEJ
JE3
When task consists of single thread, we omit the curly brackets, namely,
Twall [C.] := wall [[C)]] where =
Note that two threads, denoted by C) and C.J,, may run concurrently and overlap in time. Hence, it is possible that max (Tawll [Cy] Twall [C]+]]] S. Twan HCs, Cr.Hl < Twall [Cy] + Twall [Crr]. However, if C2 and Cy do overlap in time, then Twall [[C,, Cy. Z T wall [Cy] + T Tall [Cy]].
DISTRIBUTED SPECULATIVE INFERENCE
This section presents a theoretically sound orchestration framework for parallelizing SI (Leviathan al. 2023 Chen al.) 2023) Miao al.) 2024] Timor et al. (025) that essentially decoupled from the underlying computation of forward passes. Our method applies to any fixed number of processors (2 2), as shown later. Initially, we introduce a naive version our approach, which assumes access to sufficiently large number of processors, ensuring that threads never need to wait.
Before presenting our algorithm, we first discuss the limitations of existing SI methods. SI reduces target forward whenever draft token is accepted. With accurate and fast drafter, SI can significantly cut the number of target forwards, potentially speeding up the inference compared to non-SI. For instance, if on average one draft token is accepted per iteration, the number of target forwards drops to half since the average target forward generates two tokens. As long as the total drafting latency is less than the saved latency from reduced target forwards, SI offers a speedup over non-SI. The primary limitation of existing SI methods lies in their sequential nature. Each SI iteration requires target forward, and the next iteration only begins after the current one is completed. Therefore, SI with sufficiently slow or inaccurate drafters is slower than non-SI, even if reducing the number of target forwards. However, we observe that the verification of each iteration is not inherently sequential and could be parallelized.
Previous works on SI exemplified speedups using drafters that run within 1-5% of the time (compared to the target model), and iterations of 2-5 draft tokens (namely, 1 lookahead E 5). We can calculate the maximum speedup of our proposed method compared to by Amdahl's 's law, as follows. Assume the drafter is perfectly accurate. In that case, our method hides all the verification latency such that the overall end- and-to-end latency remains only the drafting latency. For drafters of 1-5% latency and lookahead [2,5], our proposed parallelization leads to theoretical speedup of 4x-50x compared to SI.
Figure and Table illlsstrate potential speedups of our proposed method compared SI and non-SI, given a drafter of 14% latency and lookahead = Larger lookahead values (as often used in practice) yield even larger theoretical speedups.
3

---

Published as a conference paper at ICLR 2025
Sh
GPU
GPU
DS
GF0
6PU
GPV
6PU
GPU
Figure 1: Illustration of the timeline for DSI, SI, and autoregressive inference (non-St). Blue and yellow mark the forward latency of the target and drafter, respectively. In this example we have lookahead = I, namely, the drafter generates single token every yellow square. Non-SI and SI are both sequential: each their iterations ends with a target forward, and this target forward must be completed before the next iteration start. In DSI, target forwards as in and non-SI. While DSI works for any given number GPUs (2 2), here orchestrates eight GPUs.
necessarily blocking
Table I: The number of tokens that non-SI, SI, and DSI generate according Figure In the worst case, all the draft tokens (yellow Figure ) are rejected. In the best case, all the drafts are accepted. The number of tokens generated by DSI is greater than equal to the number of tokens generated by SI and non-SI all cases, at any time, in expectation.
f1
f2
t3
f
non-SI
4
Worst case
DSI
2
4
9
non-SI
Best case
SI
8
14
DSI
8
26
50
58
3.1 METHOD OVERVIEW
Consider the task of computing N output tokens autoregressively from a target model fm given prompt Z-go- We have set of faster drafter models, fı,... .....,,mm fm-1, that are all faster than fm (as defined Assumption 2) Our goal is to compute n; = [m (# for all E [N]. Appendix C
provides a detailed, step-by-step explanation of Algorithm [
Acceptance rate. Lines X and I0 Algorithm [ terminate any thread (and its descendants) that returns a token that does not match the token returned by the current verifier. We say that this draft token rejected. Given target, a drafter, and an input prompt, we define the acceptance rate to be the probability of accepting the draft token. To increase the acceptance rate, exact-matching (lines 10p with relaxed methods for rejecting drafts. For example, applying the procedures proposed in Leviathan al. (023);; Chen et al. (2023) Miao al. (2024); Timor et al. (2025) increases the acceptance rate while maintaining the distribution of the outputs of the target model (namely, lossless in expectation).
replace the strict
Speculation parallelism (SP). In essence, DSI offers a new type of task parallelism we name speculation parallelism (SP) that orchestrates instances of the target and drafters overlap in time. Speculation parallelism degree (SP degree) is the maximal number of target servers (namely, servers dedicated to computing the target model) used the same time. DSI parallelizes all of the non- sequential parts of SI. Unlike SI, where verifying the drafts of the current iteration is sequential such that the verification blocks the algorithm from drafting the next iteration, DSI runs verifications on additional threads to hide their latency. In DSI, verifications contribute to the overall latency only when they reject a token. Rejecting token in DSI triggers a synchronization mechanism terminating threads that received rejected token (line 8). ensuring the output tokens are all accepted. The portion of the inference that DSI parallelizes tends to the expected acceptance rate as the number of generated tokens grows to infinity. For example, in the simple acceptance rate, DSI effectively parallelizes 80% the work such that the expected speedup is 5x by Amdahl's law for generating N tokens.
where we have single drafter with 80%
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Distributed Speculative Inference (DSI) of N tokens
Require: A prompt #go, and m autoregressive models, fi, J2,-.., Sm-
1: v =.= =.
2: initiate threads C(.)..... ((m) such that CG.) generates the token x fn (mgo) for all [m]]
concurrently.
3: label thread C(m) as the current verifier.
4: ONCE any thread Cyeg) finishes to generate a token, namely, sampled Crecy [new]
fg (Csesslpromptt::
5: if []] + 1 < N then
6: initiate threads, CJæGa), C1ou.,) Cregs.. generate token concurrentyy and respectively from fI, f2...-.-7 ......-- f-.
Crecy the current verifier thread then terminate all threads CJaGr) (and their descendant threads) that sampled a different token than CJe()-
8:
j = arg min [ Cregr) Crew))
J'E.m)
10:
terminate all threads CJa(r (and their descendant threads), where j' >. label _) the current verifier. update v u 1.
11:
12:
13:
if CJerj m) has already finished then
go back to step 7 with 7ith 15: end if
14:
@ (m).
16: end if
else if the last entry of J + (J) equals (i.c., j = m) then
18:
return Crecy/retumm.
19:
end if
20:
end ONCE
Lookahead. While the abstract version of DSI described in Algorithm I takes advantage sufficiently large number of servers, in practice typically have fixed number of servers. We can deploy DSI on an arbitrary number of servers (2 2) by selecting a sufficiently large lockahead hyperparameter, as elaborated in Appendix D The lookahead is defined the number of draft tokens in every verification task sent to a target server. The lookahead in Algorithm [ is set to 1 for simplicity, but can be arbitrarily large. For example, verifying every five draft tokens (lookahead = 5) instead of one (lookahead = 1) is standard configuration of SI methods in practice. In DSI, larger lookahead values decrease the frequency at which verification tasks are sent to the target servers, hence decreasing the required SP degree. Given an SP degree, the lookahead must be sufficiently large to satisfy the following inequality, ensuring that verification tasks never wait to be processed by target server.
(target latency)
S SP
(lookahead) (drafter latency)
Smaller SP degrees or faster drafters require selecting larger lookahead values. For example, given a single drafter 5% latency and SP = 4, having lookahead is sufficient. Assuming single drafter that runs single processing unit, the maximum number of required processing units is 1 5-0.05 = If more than five processing units are available, we can seleet a smaller lookahead value, yielding verification tasks more frequently. In general, selecting the minimum lookahead value that satisfies Equation] D is the optimal choice, allowing DSI to detect rejections
(line 8 earlier. Using an SP degree such that SP drafter lntency
target latency
reaches the maximum expected
speedup, and any larger SP degree cannot speed up the inference because there will be more target servers than verification tasks that can be processed in parallel.
Resource contention. In practice, resource contention might occur when multiple threads compete for the same hardware resources, such as memory bandwidth, data transfer channels, or CPU cores used for orchestration. By selecting the minimal lockahead that ensures the required SP degree is supported by the available resources (namely, satisfying Equation D. we can guarantee that the algorithm runs efficiently in practice, without significant resource contention, because the verification requests are sent to the target server at different times, and the responses are expected in staggered timings.
5

---

Published as a conference paper at ICLR 2025
way to parallelize that differs from tensor parallelism (TP) and pipeline parallelism (PP). DSI can employ TP, PP, or a combination both TP and PP. To simplify, we say that MP is any such parallelism combination, including TP, PP, TP+PP. MP speeds up the computation of forwards, and SI reduces the number of target forwards. Their combination (SI+MP) reduces the number of target forwards and accelerates each target forward. DSI further reduces the number of target forwards contributing to latency because DSI hides the latency of verification tasks by computing them in parallel. Below we compare SP and MP, providing simple example demonstrating that SP outperforms MP given the same computing budget.
Model parallelism (MP). DSI introduces
In DSI, target forwards contribute latency only if they reject draft. Below is a simple example of comparing MP and SP. Given drafter of 10% latency, we can set lookahead = 2 to allow DSI to run over single node of only 6 GPUs (5 for the target and for the drafter). Let the acceptance rate of the drafter. The expected number of target forwards that DSI eliminates by hiding is a², For example, for drafter with an 80% acceptance rate, only 36% of the target forwards contribute to latency (in general, 1 - a @ookaabed)). Under the same computing budget (MP=5), MP must accelerate the target forwards by 2.78x or more to become faster than DSI. However, MP is ineffective for certain hardware setups, model architectures and sizes, while DSI remains effective.
DSI could be naturally combined with MP to accelerate the underlying forwards, requiring no changes to the algorithm, because DSI offers an orchestration algorithm agnostic to the underlying computation of forwards. Since DSI can orchestrate multiple nodes, it unlocks setups with sufficiently large SP degrees so that there is no theoretical tradeoff between DSI and MP. Combining DSI with MP methods can possibly further accelerate the inference in both single- and multi-node setups. All the foundational concepts of such an implementation of DSI are covered in this paper.
KV cache. We can view DSI as an orchestration algorithm that constructs and verifies token trees on the fly. DSI is decoupled from the underlying computation of forwards, including KV cache management, both in theory and in practice. Each server maintains its own KV cache. The servers collaboratively process a token tree with shared prefixes. Synchronizations occur every draft rejection.
Efficient KV cache management of token trees has already been developed in SpecInfer, where tree paths can share common prefixes (Miao et al. (2024). Practitioners can apply SpecInfer's ' KV cache management as-is to achieve the expected speedups reported in this paper. While it might require some engineering effort to implement SpecInfer's s KV cache management, it is solved research problem and has been shown to add negligible latency.
3.2 THEORETICAL RESULTS
Next, we formally state that DSI (Algorithm M is strictly lossless, always returning the correct sequence of tokens T1..... IN, runs at least as fast as non-SI, and runs faster than both non-SI and SI in expectation. Before presenting the analysis. The proofs are provided in Appendix E
main theoretical results, outline the assumptions used in
Assumption We assume the existence of a (universal) constant 0 such that, for any input prompt ao and model index j E [m], we have:
Twall [computing fq (xco))= E c0 and Twall [sampling x ~ 83 (I< =
Assumption 2.
We assume that for all E [m 11. f3 is faster than f' (denoted fG 3 fmm in the following sense maX Twatl (computing f, (x<a)] S minggo Twatl [computing fm (2co)).
Assumption 3. We assume that Twoll ( = - Twall [CG...gn)]
The first assumption asserts that computing the output of amount of time, and sampling a token from the output probabilities takes a negligible amount time. The second assumption asserts that each drafter model runs faster than the target model, for any given input prompt. The third assumption asserts that computing x Ji....h takes the time of first computing xt' then x² 1. and so forth, up to x.. J.. with no delays.
of any model takes non-zero, bounded
The following theorem suggests that our method returns tokens from the same distributions those the target would generate without speculation, and is at least as fast as iteratively applying the target model itself.

---

Published as a conference paper at ICLR 2025
and 3 Algorithm I returns the same output and runs at least as fast as running the target model itself without speculative inference (SI).
Theorem 1. Under Assumptions 2
Theorem 2. Under Assumptions and
Algorithm 7
runs at least as fast as SI in expectation.
The advantage of Algorithm| 1 Ilies in its concurrency. The following proposition shows how DSI can accelerate the inference of given target model using drafter model that faster than the target model and returns the correct output with high probability.
Proposition Suppose we have a drafter model J1, iarget model f2 and a prompt T-çO- Assume that fi requires t time units to compute each of its outputs, and f2 requires t2 time units, where t2 > t1. Assume that given the prompt zgz = 2.00 + 11 + .... O Ii, the probability that fr returns the (correct) token 2i+1 is p. Then, the expected time it takes Algorithm too calculate the correct ousput is at most tip(N - -1) + 12((11 p) -1 -1++ + 1) time units, compared to the 22 time units required if we were compute f2 without speculative inference.
4 EMPIRICAL RESULTS
DSI can orchestrate any fixed number of processors (> 2) by selecting sufficiently large lookahead value. While the theoretical guarantees in section 3.2 hold for both single- and multi-node setups, our experiments confined to single-node scenarios with at most eight GPUs.
Configuring DSI for any given number of GPUs (2 2), target model, and drafter, requires calculating the SP degree by selecting the allocation of the available GPUs, then selecting the lockahead to be the minimal number satisfying Equation D In its simplest nontrivial setup, DSI orchestrates a single node with two GPUs, implementing target server one GPU and drafter server on the other. More advanced setups involve more GPUs, potentially on other nodes, or combine DSI with other parallelism techniques so that servers can utilize multiple underlying GPUs. For example, node with eight GPUs can run eight servers, each using one GPU, or four servers, each using two GPUs, employing model parallelism (MP) with tensor parallelism (TP) or pipeline parallelism (TP). To maximize the expected speedup, we select the minimal 1 lookahead value that satisfies Equation I so that verification tasks never wait to processed by target server. Some models require more than one GPU to avoid memory offloading to slower memories (like the host's CPU memory or hard disk). For example, given seven GPUs and a target model that requires two of the given GPUs to run without offloading (namely, MP > 2). the SP degree is at most three, assuming that the drafter can run on single GPU. With an SP degree of three, we will select the 1 lookahead to be the minimal number satisfying Equation [ If the drafter forwards take 5% of the target forwards, the ratio between their latencies is 20, hence, the minimum Lookahead value guaranteeing no waiting is 7.
Since our experiments focus on single-node setups, we implemented DSI as a multithreading system. We implemented DSI in Python, using Python's 's built-in thread pool manage operating system (OS) threads. These OS-level threads share CPU resources similarly to other real-world multi-threaded systems. This ensures that real-world thread management latencies, such as context switching, thread creation, and scheduling delays, are fully incurred in the experiments. For multi-node setups, implementing DSI as a multiprocessing system with Message Passing Interface (MPI) would be more appropriate than multithreading.
In both single- and multi-node setups, DSI can employ a thread pool design pattern, where verification tasks are sent to pool of servers computing the target model. The size of this target pool is, by definition, the SP degree. Our implementation based on a thread pool of targets and a single drafter server. In all our experiments, DSI has an SP degree of less than or equal to seven and one drafter server, where each server employs a single GPU.
Our implementation of DSI and the code for running the experiments are agnostic to the underly- ing hardware, and have been open-sourced, employing high software development standards with thousands of tests, ensuring complete reproducibility of the reported results and enabling further research.
Measuring the actual speedups of DSI for node with eight GPUs requires access such node. Due to budget constraints, instead of a node with eight GPUs, we only had access to one GPU. To evaluate DSI over node with eight GPUs without access to such hardware, we adjusted the DSI
7

---

Published as a conference paper at ICLR 2025
implementation accordingly. While DSI incurs all the real-world latencies typical of multithreading systems
(e.g.. latencies of managing OS threads), each call to compute the forward pass of an LM was replaced by wait command. The wait command blocks the thread for duration that matches the actual latency.
To ensure realistic wait times, conducted separate experiment estimate the Time to First Token (TTFT) and Time Per Output Token (TPOT) for each model and dataset. These TTFT and TPOT values were then used to set the wait times in the main experiment. To estimate the acceptance rate for each combination of (target, drafter, dataset), we performed another separate experiment and plugged in the approximated acceptance rate in the main experiment. Appendices F.1 and F.2 provide detailed explanations of the independent experiments to estimate the TTFT, TPOT. and acceptance rate.
The results of our main experiment, detailed in Table 2 affirm Theorems 1 and demonstrating that DSI outperforms SI (Leviathan al.) [023] Chen et al. 2023) practical settings across various models and well-known datasets. Overall, DSI consistently outperforms SI across all models and tasks. Table 2 also specifies the latency and acceptance rate estimates used each configuration.
Table 2: DSI speedups over SI for various off-the-shelf target/drafter pairs. We observe that DSI
outperforms consistently across all models and tasks.
Target
Drafter
Dataset
Target Lateney (ms))
Drafter Lateney (ms)
Drafter Lateney (%)
Acceptance Rate (%)
Speedup rs.
Starcoder-155
starooder-16an 68M
HumanEEval
20.6
6.8
32.3
93
1.92x
Starcoder-155
Starcoder-16at
MBPP
21.0
6.8
32,9
90
1.66x
Phi3-148
Phi3-48
Alpaca
33.4
67.4
Ph13-148
Phi.3-88
HumanFval
52.1
34.0
65.3
95
1.4xx
Phi3 14B
Phi3 4B
CNN-DM
52.4
34.6
66.0
93
1.39%
Phi3-14B
Phi3-4B
MBPP
52.2
34.3
65,8
94
1.37%
Vicuns-13s
Vicuna-
CNN-DM
37.7
2.5
6.5
63
147x
Vicuns-135
Viouna-58M
Alpaca
33.3
2.5
744
1.4x
Vicuna 7B
Vicana- 6BK
CNN-DM
29.4
2.5
844
67
1.29%
Vicuns-7R
Vicuna-68M
Alpaca
26.0
2.5
9.5
The reported speedup of DSI relative to SI ("Speedup DSI vs. SI" in Table 2) is the ratio between their estimated end-to-end latencies (wall time), including prefilling and decoding latency, but excluding tokenization latency. The end-to-end latency is estimated as follows: we generate 50 tokens using each target-drafter pair on each dataset, employing real-world forward latencies and acceptance rate values from independent experiments (elaborated below). Each combination of (target, drafter, dataset) is run on multiple lookahead values (specifically, 1, 5, and 10, because this range has been shown in previous works on SI to be effective). For the DSI run, further restrict the lockaheac ad values to ensure we only consider configurations for which DSI could have been deployed on a single node with up to eight GPUs, assuming the drafter runs single GPU. That is, we only use lookahead E {1, 5, 10) if this lookahead value satisfies Equation for SP=7.
In all our experiments, the models and datasets used as-is. We used four well-established datasets to estimate forward latencies and acceptance rates, spanning various tasks: text summarization using CNN Daily Mail (Hermann et al., 2015): instruction-folle following using Alpaca Taori et al. 2023); code generation using MBPP and Human Hval (Austin et al. [00I Chen et al.) 2021). Appendices F5 and F.6 provide complete description of the models, datasets, and examples of prompts.
downloaded from the Hugging Face Hub and
4.1 ABLATION VIA OFFLINE SIMULATION
The main experiment above (Table 2 can be viewed an "online" experiment because it employs thread pools and measures the overall wall time including real-world latencies of multithreading systems. This section presents a complementary "offline" experiment that simulates the inference algorithms by simply summing the forward latencies without thread pools, assuming zero multithread- ing latencies. The offline experiment is important for two reasons: (i) it decouples the implementation details of DSI from the theoretical analysis, and (ii) it allows us to explore a much larger space of configurations within constrained computational budget.
8

---

Published as a conference paper at ICLR 2025
Figure presents the results of this offline simulation, measuring the pairwise speedups (or slow- downs): SI compared to non-SI, DSI compared to SI, and DSI compared to non-SI. Since SI is slower than non-SI in some configurations,
have included Figure d an additional comparison that shows DSI speedups relative to the faster of the two algorithms-5 SI or non-SI-for any given configuration. Figure d) helps identify configurations where DSI achieves the highest speedup. It demonstrates that, unlike SI, our method introduces no slowdown compared to non-SI and consistently accelerates inference.
As shown in aa) to achieve a speedup with SI compared to non-SI, the acceptance rate of the drafter must at least match the latency the drafter model, which corresponds the non-pink region in the figure. This means that the SI algorithm cannot speed up the inference if the acceptance rate of the drafter not sufficiently high for given latency, corresponding to the pink region in the figure. Conversely, in Figure hi), we observe that DSI consistently speeds up inference time, regardless of the latency and acceptance rates of the drafter. This provides our method with much greater flexibility and robustness when selecting drafters for given target model. In Figure 2(c), observe that DSI faster than non-SI for all configurations for which non-SI faster than SI. Finally, to obtain comprehensive view of the inference speedup achieved by DSL, in Figure 2 d), compare the performance of DSI with the minimum runtime for any configuration between SI and non-SI.
The heatmaps represent millions of data points, where each point corresponds to a different configu- ration. Since offline simulations are insensitive to the real-world latencies of multithreading systems (e.g., context switching), offline simulation for particular configuration could be in parallel with other offline simulations. This approach of parallelizing the experiments- them sequentially-
than running feasible to scale up the number of configurations explored within constrained computational budget and provide comprehensive heatmaps of the expected speedups. In contrast, for online experiments, each of the millions of configurations represents distinct online run that necessitates a separate environment.
18
14
12
44 0.5 Dra Latercy
Grafter Lauandy
Drafter Latency
Drafeer Latency
(a) non-SI/SI
(b) SI/DSI
(c) non DIDSSI/DSI
min (Sl,0S),)
Figure 2: Expected pairwise speedups (or slowdowns) of DSI, SI, and non-SI. Each heatmap is labeled "X/Y" and plots the ratio between the run time of algorithm X and the run time of algorithm Y. The run time of each algorithm is computed by summing the latencies of all the forward passes and intentionally ignoring additional real-world latencies of multithreading systems like context switching, allowing to decouple the implementation details from the theoretical analysis. (a): SI is slower than non-speculative inference (non-SI) when drafter either slow or inaccurate enough (pink marks slowdowns). (b, d): DSI is faster than speculative inference (SI) and non-speculative inference (non-SI) for all configurations SI non-SI for all configurations. (d): DSI is up to 1.6x faster than the baseline algorithm, where the baseline the faster between SI and non-SI for each configuration.
non- non-zero zero acceptance rate. DSI is never slower than either
Appendix F provides additional implementation details about the experiments. We open-source the code for all the simulations.
RELATED WORK
Research SI has expanded the framework, including dynamically controlling the number of draft
tokens per iteration -a technique widely adopted in practice (Mamou et al. [2024] Liu et al. 2024a
Gante 2023) and exploring various other directions (Li et al. 2024 Cai et al.
2024 Sun et al.
2024b Liu et al. 2024b) Zhou et al. [024] Zafrir et al. 2024:] 1024:Narasinn Narasimhan et al.
2025 Bachmann
al. 2025 However, in prior works, computing target forward passes remains blocking operation,

---

Published as a conference paper at ICLR 2025
limiting the algorithm from processing tokens in later positions and leaving the fundamental limitation of SI as a sequential algorithm unaddressed.
PEARL is recen extension to SI, demonstrating that drafting in parallel verifying speeds up non-SI and vanilla SI by up to 4.43x and 1.5x, respectively (Liu et al. 2025). Their empirical results highlight the advantage of DSI in breaking the draft-then-
sequential nature of SI. However, PEARL suffers from a fundamental limitation: it remains a sequential algorithm because it can only process tokens of the next SI iteration, unlike DSI, which can process tokens of any future iteration. Their algorithm employs a heuristic (controlling whether to verify the first draft token of every iteration) and has no theoretical guarantees to speed up SI or non-SI. In fact, PEARL is slower than non-SI if the drafters are too slow or iaccurrte, unlike DSI. Since PEARL cannot orchestrate more than one instance of the target model and one instance of the drafter, it offers limited scalability to hardware setups, unlike DSI, which can orchestrate an arbitrary number of GPUs (2 2). As result, PEARL underutilizes the available hardware unless it runs with lookahead and is therefore strictly slower than DSI with smaller lookahead, in expectation.
target lntency drafter latency
1,
For other related work beyond SI, see Appendix A
6 DISCUSSION
This work proposes a method to reduce the run time of speculative inference algorithms by taking advantage of an arbitrary number of additional multiple processing units (c.g., GPUs). We have shown that despite the wide adoption of SI algorithms, they can end up slowing the inference of language models in various practical settings, when the drafters are insufficiently accurate fast. We showed that by taking advantage of at least one additional GPU, we can design a speculatively inference algorithm that provably reduces the inference time of both non-SI and SI algorithms. Our simulations affirm our theory, indicating significant speedups over both SI and non-SI for all possible configurations given a single node with up to eight GPUs. In essence, this work paves the way to additional SI algorithms that can orchestrate multiple processing units at the same time via speculation parallelism (SP).
We introduce distributed speculative inference (DSI) and show it is faster than SI and non-SI for all possible configurations by theoretical analysis and experiments. While the theoretical guarantees hold for both single- and multi-node setups, our experiments focus on single-node scenarios with up to eight GPUs with an SP degree < 7. Due to budget constraints, we adjusted our implementation of DSI to simulate an access to such a node rather than running a physical node with eight GPUs. Nevertheless, the empirical results are realistic because DSI is implemented as a multithreading system, incurring all real-world latencies of such systems (e.g., context switching), and all the real-world GPU-related latencies are based on independent experiments accessing a GPU.
Future Work. DSI's key strengths lie in its use of speculation parallelism reduce latency through parallelized verification, and its ability to achieve lossless speedups without modifying model architectures. Building that require multiple GPUs to avoid memory offloading (namely, MP degree > 1). While most state-of-the-as models can run on single GPU (via compression, see a trend towards even larger LMs. Since larger LMs are often slower, DSI's parallelism could be particularly effective. Although multi-node inference not yet a common setup, testing DSI in realistic multi-node environments, could unlock its potential despite communication latencies.
these, future work should focus on evaluating DSI with LMs
lower precision, etc.). we do
ACKNOWLEDGMENTS
We thank Intel Labs for funding this research.
This work was partially funded by the Israel Science Foundation (ISF grant 3698/21). Additional support was provided by a research grant to David Harel from Louis J. Lavigne and Nancy Rothman, the Carter Chapman Shreve Family Foundation, Dr. and Mrs. Donald Rivin, and the Estate of Smigel Trust. Michal Gordon-Kiwel Kiwkowitz's is contribution was conducted as part of her consulting position at the Weizmann Institute of Science. She is also affiliated with the School of Computer Science at
"Their work was released on arXiv and GitHub after ours and published concurrently ICLR.
10

---

Published as a conference paper at ICLR 2025
Holon Institute of Technology and has recently moved Indiana University Bloomington. Part of Tomer Galanti's ' contribution was conducted while he was at MIT.
REFERENCES
Gene M Amdahl. Validity of the single processor approach to achieving large scale computing capabilities. In Proceedings of the April 18-20, 1967, spring joint computer conference, pp.
483-485, 1967.
Jacob Andreas. Language models as agent models. Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5769-5779. Association for Computational Linguisties, 2022. doi: 10. 12m.::////2222..ndninng----
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint vrXiv:2108.07732 2021.
Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas K Kohler. Judge decoding: Faster speculative sampling requires going beyond model alignment. In The Thirteenth Interna- fional Conference on Learning Representations, 2025.
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Controlling computation versus quality for neural sequence models. arXiv preprint arXiv:2002.07/066 2020.
Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. A framework for the evaluation of code generation models, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, Bee Ka- mar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. /27/2, 2023.
F. Warren Burton. Speculative computation, parallelism, and functional programming. Transactions on Computers, 1-34(12):1190-111) 1985. doi: 10.1 10.1109T..1885.312221.
IEEE
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Forty-first International Conference on Machine Learning. 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Dean-Bapttitte Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01 00..000..33... 2023.
Mark Chen. Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss. William H. Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, 1lya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
11

---

Published as a conference paper at ICLR 2025
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359 2022.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 Q): 8-bit matrix multiplication for transformers scale, Advances in Neural Information Processing Systems, 35: 30318-30332, 2022.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. International Conference on Learning Representations, 2020.
Depth-adaptive transformer. In
Elias Frantar and Dan Alistarh, Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference Learning Representations, 2023.
Joao Gante. Assisted generation: direction toward low-latency text generation. Hugging Face Blog, 2023.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253) 2015.
Yen-Char Ten-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Conference Learning Representations, 2022.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research, (8(187):1-3) 2018.
Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp.
PMLR, 2023.
Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Urvashi Bhattacharyya, Wenhao Yu. Sasha Luccioni, Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan
12

---

Published as a conference paper at ICLR 2025
Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. Starcoder: may the source be with you! Transactions Machine Learning Research, 2023a. ISSN 2835-8856, URL http:ttts:/// hhtttps://openrevvt/ wtwpenrrwwwwwwweeeeeeeeeeeeeeeee Reproducibility Certification.
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective. In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023b.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In Forty-first International Conference on Machine Learning, 2024.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. In MLSys, 2024.
Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, Winston Hu, and Xiao Sun. PEARL: Parallel speculative decoding with adaptive draft length. In The Thirteenth International Conference on Learning Representations, 2025.
Xiaoxuan Liu, Cade Daniel, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Xiangxi Mo, Alvin Cheung. Zhijie Deng, Ion Stoica, and Hao Zhang. Optimizing speculative decoding for serving large language models using goodput. arXiv preprint arXiv:2406.1400 2024a.
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding. In Forty-first International Conference on Machine Learning, 2024b.
Xinyin Ma Gongfan Fang, and Xinchao Wang. LLM-pruner: On the structural pruning of large language models. Thirth- Conference on Neural Information Processing Systems, 2023.
Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Nadav Timor, Moshe Wasserblat, and Roy Schwartz. Dynamic speculation lookahead accelerates speculative decoding of large language models. In Proceedings of The 4th NeurlPS Efficient Natural Language and Speech Processing Workshop, volume 262 of Proceedings of Machine Learning Research, pp. 456-467. PMLR, 2024. Utthttps:://// http::/ss/////oeeeeeeeeeee ml p.ersspp/ssss. / v2.22/2222222 20122//mmu2220 4a html
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. Proceedings of the 29th ACM Interna- tional Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. pp. 932-949,220 2024.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19393. 2025.
Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar. Faster cascades via speculative decoding. In The Thirteenth International Conference on Learning Representations, 2025.
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-Im. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. Association for Computing Machinery, 2021.
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haim ing Bao. Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett- Shappioo, Christopher Berner, Lenny Bog- donoff, Oleg Boiko, Made laine Boyd, Anna-L Luisa Brakman, Greg Brockman, Tim Brooks, Miles
13

---

Published as a conference paper at ICLR 2025
Brundage, Kevin Button, Trevor Cai. Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Auty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Sim on Posada Fish- man, Juston Forte, Is abella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Lontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Jo hannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Hcewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Ma teusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski. Bianca Martin, Katie Mayer, An- drew Mayne. Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mchta, Jacob Menick, Luke Metz. Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David M'ely, Ashvin Nair, Reicchiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Ouyang Long, Cullen O' Keeff, Jakub W. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Paylov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alee Radford, Jack W. Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr. John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jor- dan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas A. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer'on on Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll L. Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng. Lilian Weng. Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qim ing Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report. arXiv preprint ar arXiv:2303.08774, 2023.
OpenAl, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman. Camillo Lugaresi. Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler. Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong. Elizabeth Prochl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang. Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song. Fred von Lohmann. Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian
14

---

Published as a conference paper at ICLR 2025
O'Connell, Ian Osband, Ignasi Clavera Gilaberte, llge Akkaya, llya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang. Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar. Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang. Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowd- hury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Míyara, Reimar Leike, Renny Hwang. Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojciech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai ol system card. 2024.
arXiv preprint ar arXiv:2412.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Samyam Rajbhandari. Conglong Li, Zhewei Yao. Minjia Zhang. Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. DeepSpeed-Mon Advancing mixture-of-expere of-expsrts inference and training to power next-generation Al scale. In Proceedings of the 39th International Conference on Machine Learning, volume 162 Proceedings of Machine Learning Research, pp- 18332-18346. PMLR, 17-23 Jul 2022.
David Rodgers. Improvements in multiprocessor system design. ACM SIGARCH Computer Architecture News, 13(3):225-233 1985.
Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En- Hsu Yen, Avner May. Tianqi Chen, and Beidi Chen. Magicdec: Breaking the latency-throughpe tradeoff for long context generation with speculative decoding. In The Thirteenth International Conference on Learning Representations, 2025.
Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. Consistent accelerated inference via confident adaptive transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4962-4979. Association for Computational Linguistics, 2021.
doi: 10.18653/v 1/2021 wttp:///...2.mmmaaiiaaa
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 15:17456-17472, 2022.
Benjamin Frederick Spector and Christopher Re. Accelerating LLM inference with staged speculative decoding. In Workshop on Efficient Systems for Foundation Models @ ICML2023. 2023.
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autore- gressive models. Advances in Neural Information Processing Systems, 31, 2018.
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024a.
15

---

Published as a conference paper at ICLR 2025
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024b.
Ziteng Sun, Uni Mendlovic, Yaniv Leviathan, Asaf Aharoni, Ahmad Beirami, Jae Hun Ro, and Ananda Theertha Suresh. Block verification accelerates speculative decoding. In The Thirteenth International Conference Learning Representutions, 2025.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-folowiin model. Stanford Center for Research on Foundation Models blog, 2023.
Nadav Timor, Jonathan Mamou, Daniel Korat, Moshe Berchansky, Oren Pereg, Gaurav Jain, Roy Schwartz, Moshe Wasserblat, and David Harel. Accelerating Ilm inference with lossless speculative decoding algorithms for heterogeneous vocabularies. arXiv preprint arXiv:2502.05522,, 2025.
URL https: / haarrg/bb////0
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024.
Mingxue Xu, Yao Lei Xu, and Danilo P. Mandic. Tensorgpt: Efficient compression of the embedding layer in Ilms based on the tensor-train decomposition, 2023.
Amy Yang, Jingyi Yang, Aya Ibrahim, Xinfeng Xie, Bangsheng Tang, Grigory Sizov, Jeremy Reizenstein, Jongsoo Park, and Jianyu Huang. Context parallelism for scalable million-token inference. arXiv preprint arXiv:2411.00 1,01783, 2024.
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Exploring post-training quantization in Ilms from comprehensive study to low rank compensation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 19377-19385, 2024.
Ofir Zafrir, Igor Margulis, Dorin Shteyman, and Guy Boudoukh. Fastdraft: How to train your draft. arXiv preprint arXiv:2411.11055 2024.
Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Re. The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry. In The Twelfth International Conference on Learning Representations, 2024a.
Michael Zhang, Aaryan Singhal, Benjamin Frederick Spector, Simran Arora, and Christopher Re. Low-rank linearization of large language models. In Workshop on Efficient Systems for Foundation Models II @ ICML2024, 2024b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh. Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation, In The Twelfth International Conference on Learning Repre- sentations, 2024.
16

---

Published as a conference paper at ICLR 2025
A OTHER RELATED WORK
Beyond SI, other recent efforts to reduce the inference latency of LMs can be classified into two main categories by their approach to hardware utilization. The first category focuses on acceler- ating the inference by using more computing power. It includes data parallelism (DP) and model parallelism (MP) methods of different types, such as pipeline parallelism (PP), tensor parallelism (TP) (araaanaan al. 2021). ) context parallelism (Li et al. 2023b; Yang al., 2024), and expert parallelism (Rajbhandari et al. 2022). Such partitioning over multiple processors can speed up hemory-boundee inference setups, for example, by avoiding memory offloading. They are often effective in increasing the inference throughput and supporting larger context lengths. However, the typical LM architectures necessitate autoregressive inference. Such inference is inherently sequential with only limited (if any) MP opportunities, depending on the LMM architecture. Since the portion of the inference that could be parallelized is small (if any) in typical LMs, the potential speedup of MP methods is limited, by Amdahl's ' law (Amdahl 1967; Rodgers 1985). Hence, while DP methods can increase the inference throughput, they remain inherently sequential. Furthermore, as the parallelisms above shard over more processors, the communication overhead increases, which can lead to diminishing returns, limiting the number of processors they can effectively utilize.
The second category focuses on making LMs use less computing resources or better utilize the same resources. This includes post-training compression through pruning (e.g., (Frantar Alistarh
2023, Sun et al., [2244]] Ma et al.
(203)), knowledge distillation (e.g.. Hinton et al.] (2015): Gu
ct al. (2024), quantization (c.g.. (Hbbra et al. 2018 Frantar et al. [023] Lin et al. 2024 Yao
al.
2024. Dettmers et al., ) low-rank factorization (e.g.. (Hsu et al. [022]] Xu et al. (023
early exiting (e.g.. (Schuster et al. 2022 Kim et al. 2023 Elbayad et al. 2020: Bapna et al. 2020
Schuster et al. 2021)), and alternative architectures (e.g.. (Cai et al. 2024 et al. 2024 Zhang
et al. 2024b Xiao et al. 2024| Gu & Dao 2024)). While these solutions are useful in practice, they often require modifications to the model architecture, changes the training procedure and
re-training of the models, without guaranteeing identical outputs. Despite reducing the inference time, these methods often have a significant drawback of degrading the output quality. There are also solutions that preserve the output quality, such as kernel optimizations (Dao et al. [022] Dao 2024). but they highly depend on the hardware and therefore are not always available or even feasible.
B EXTENDED PRELIMINARIES
Autoregressive language models (LMs) are deterministic, real-valued multivariate functions. An input to an LM is a sequence of vectors of dimension Revecab We call these vectors tokens, and the sequence prompt. LMs output a real-valued vector of dimension Nvocab, also known as the logits. Since prompts may vary in length, we simplify the notation of the forward pass as follows: f: 1huach
R Buac
Self-Attention LMs are LMs with a pre-defined context length nax (Vaswani et al.
2017). Hence, R Poaa For
represent the forward pass of such LMs in the following manner: f: R X.och
example, GPT-2 and GPT-3 are Transformers with noncchh 50257, and context lengths nax = 1024
and nax = 2048, respectively (Radford et al., 2019 Brown al. 2020). In this paper, all L.Ss Self-Attention ones with pre-trained (frozen) parameters.
We extend the prompt notation such that prompts have length i S nexx- Self-Attention LMs handle prompts of length / < 7lxx by starting the input sequence with prefix of Max tokens, followed by the given tokens. LMs ignore the prefix, either by zeroing (masking) the Attention parts corresponding to the prefix or by left- left-padding with dedicated tokens. In this paper, prompts of length I < Rexx are the non-masked, non non-padded suffix of the input sequence of length Rexx.
Generating the next token is the primary application of autoregressive LMs. This process consists of two steps: computing the forward pass of the LM and then selecting the next token based on the output. The selection can be deterministic or non-determiiistic. Non-deterministic selection procedures apply the softmax function after the forward pass of LMs and sample from the resulting probability vector:
softmax
R.aha -> (0, 1heee such that the entries sum 1.
17

---

Published as a conference paper at ICLR 2025
For convenience, we denote the output probability vector by (#g+): - (Zgg)) == softmax (f (zgg)) softmax (f (Ego @ 21 @ 0.z))
(3)
where a + b (a, b) is the concatenation of the vectors and and Igi Z50 @ 21 ... 2 I₁. For deterministic selection procedures, composing monotonic functions, such as softmax, is usually unnecessary. For example, the most likely next token is the arg max of both the logits and the output of the softmax. Still, for convenience, we assume that LMs always output probability vectors. The sampling process in equation iis either deterministic (i.e., 2(+1 is a token with maximal probability) or random (achieved by randomly selecting E141 from the distribution softmax (f (#zg:)))
C STEP-BY-STEP METHOD OVERVIEW
Consider the task of computing N output tokens autoregressively from a target model fm given prompt Z-go- We have set of faster drafter models, fi,...; Jm- that are all faster than fm (as defined in Assumption 2]. Our goal to compute I; = fm (Z:<=-) for all i E [N]. To achieve this, we initiate threads, Ca))...,, ((m) lineZin Algorithm D Each thread, denoted as (j1), is responsible for computing of' = fn (==o)) Once thread, C(y.). finishes computation, we instantiate m new threads, d( to calculate w//. = (x @ x²+) for all j2 E [m]]. In general, once we compute --111 -f1,:,de-11, we initiate m new threads, 111))- 1. to compute axp -. = -, (o @ x ① ....... @ - for all j. [m]] This captured in lines and 6
Once C(m) completes its computation and provides the correct value of the first output token zT" T1. we can verify which other threads, C(i.). have accurately computed ø1. Any thread C(j.) where
art' o is immediately terminated along with its descendant processes (line 8). For each j1 E [m] that correctly computed x²' 21, continue with computing مق1 fiz (z=o @ x²²) for all j2 E [m]. However, since all threads are computing the same set of tokens, we terminate all but the one corresponding to the smallest value of ji that satisfies xt = x1 (line 10p. In essence, C(m) serves as a verifier, identifying drafters that miscalculated the initial part of the autoregressive computation. Once we retain one valid j1, we relabel C(i..) as the new verifier thread. We know that since C(s))
returned the correct token x = 21 and 22 = fm (1) the output of C(ji.)) must be correct. When that thread finishes, among the remaining threads. (ji.z)) we terminate those that miscalculated T2 2 (line 8) and keep only the one with 22 2...2 x2 j.mm I2, whose index ja minimal (line 10). We continue this process until the output - is obtained from the last verifier thread Ch.....
The process of relabeling verifier threads and terminating irrelevant threads is Line 13 considers the case where the newly labeled thread may have already finished. so, in line 14 we return to line 7 with the new verifier thread.
outlined in lines 8 TO and
D LOOKAHEAD
We can deploy DSI on arbitrary number of servers by selecting sufficiently large 1 lockahead hyperparameter, as explained below. 6ine A Algorithm|] T invokes a new process to compute the target model fm immediately after generating any token (except for tokens of poisition N, namely, tokens corresponding to the last position). In particular, after generating a token from a drafter Jj (where j < m)) Such a "draft" token might be rejected in line 8 and is accepted otherwise, as discussed earlier. We can view this procedure as generating a draft token and sending it to "verification". Sending verification tasks of a single draft token only a private case of DSI, where lookahead = 1. For a sufficiently large SP degree, the number of such verification tasks that can run in parallel is unbounded. However, in practice, the number of available processors is given, hence the SP degree must be fixed. For example, given single drafter (that is, m 2) and single node with 8 GPUs, DSI must ensure that SP < 7, assuming that GPU is sufficient for computing the drafter. To control the SP degree, we introduce new hyperparameter, 1 Lookahead, which is the number of draft tokens in every verification task sent to a target server. For lookahead > 1, are adjusted follows: initiate 1 threads C) C)a(1))
lines and are generate lookahead tokens concurrently and respectively from fi, f2..... -.. and initiate 1 thread CJe(j)) generate token from fm. Here, we overload the notation for simplicity, allowing
Crocjmm.) to
18

---

Published as a conference paper at ICLR 2025
J @ (j) to be empty so that (i.)"" = (j"). This change reduces the number of invocations required. For any given models fi, f2. Jm and an SP degree, there exist a sufficiently large lookaheas value such that there is at least one available target server by the time that any verification task is sent, so that verification tasks never wait to be processed by a target server. Therefore, the 1 cookal head hyperparameter allows tuning DSI use an arbitrary maximal number of available processing units.
Theoretically, without scaling the lookahead such that lookahead dratter larency" the SP degree might grow to infinity. For example, if the time it takes to compute a forward pass of the drafter goes to 0 or the time it takes to compute a forward pass of the target model goes to infinity.
target lateney
E PROOFS
Theorem 1.
Algorithm 7 returns the same oulput and runs at least as fast as running the target model itself without speculative inference (SI).
Under Assumptions 2 and 3
Proof. We begin by demonstrating the losslessness of the algorithm. We would like to prove that when v t k, there is a thread J2.- that is the only thread that is labeled as a verifier, and correctly computes the next token and that Js J' @ (m) for some sequence J' = - of length where 2! 2.i
F; for all i [k 1]. We will prove this by induction on the value of v. In addition, we note that if this pattern is appreciated by the algorithm, then it is clearly a lossless algorithm.
Base case (u = 1): Initially, when = 1, there is only one verifier, ((m), which runs the target model fm- Thus, when it finishes, it will return the correct token, ø1. Since the verifier relabeled only when the value of e changes (see lines 11 12). as long as = 1, the only thread labeled as verifier is C(m)
= k. there is only one thread CJ, labeled verifier, which returns the correct token Tg, and that Js = J' @ (m) for some (J,,..., of length ls - 1, where x. -h. = x; for all i E [k: 1].
Induction hypothesis: Assume that as long
Induction step: When is updated from le to k + 1, this change only occurs when the condition in line is met. This condition indicates that the single verifier thread C.., which is of length |Ja] = k. has finished computing its output token. By the induction hypothesis, this thread returns Th as its output. Since Im is slower than all drafter models f all threads Crees)) have already finished computing their outputs. Thus, when exccuting lines 8 10 and IT] the only threads that remain active are the descendants of Creaj") and the only thread serving as verifier is Cya
Since 2. - = x4 for all i & - 1 and wi fh...- = Tg. then Crea* m) simply computes the output of the target model fm on the correct sequence x'00 I1 Q ... 2 Zg. Hence, it correctly returns the (k + 1)th token I as desired.
Time: We notice that the algorithm terminates once has computed the output of OJy- By
Assumption 3 we have Twall [Algorithm [
= >' Twall [computing fa, (æ<:) and by Assump-
tion 2 we have Twail [computing fj. (#gg)
< T Twall [computing fm (Eg.)) Together we obtain
Twall [Algorithm
| < E Twall (computing fm (x<+)] which is the amount of time that it takes
to compute the output tokens without speculative inference.
O
Proposition Suppose we have a drafter model Jú, a target model f2 and a prompt F- Assume that Ji requires i time units to compute each of its outputs, and f2 requires l2 time units, where t2 > t1. Assume that given the prompt Zgz- = 2<0 + x1 + ... O Iú, the probability that fr returns the (correct) token 2++1 is p. Then, the expected time it takes Algorithm [ to calculate the correct output is at most tpp(N - 1) +2((1- (N - 1)++ + 1) time units, compared to the 22N time units required if were to compute f2 without speculative inference.
Proof. To understand how it works, let jı E {1,2) be the smallest index such that zei = and for all i [V - 1]. recursively define j; 2} to be the smallest index such that z!! wiii..dhh Z; We also fix jN = 2. In addition, let ig = 0 and i, be the rth index in [N] such that ji, = 2. We notice that it takes L (i1 - -1)+ +/2 time units to compute the value of o,
dimnddy This is because we first compute =t.
19

---

Published as a conference paper at ICLR 2025
then 1.1, continuing up to =; and finally zi .......22 Each of the first (i1 - 1) tokens takes time units, while the final token takes t2 time units. After (i1 - 1) + time units, we will have computed of. 22 2z 1.1,2 and so on, up to 2 ".!.....? Since fi consistently generates accurate tokens up index once we observe that of matches x). we know that x22 = 22 and can then verify that that"2 _..1 = 22 is also correct. Once we verify that 2² = I2, we can verify z²2 1.1.2 and continue this pattern to verify 22 1.1.1 and so forth. We note that calculating all of these tokens up to the calculation of o;; .........2 take at most t (i1 - 1) +t2 time units. Thus, we can verify that " 2........2 = 24, with at most f (11 - 1) + tz time units. By the same argument as above, it takes (r (t - 1 - 1) + -22) time units to compute the value of znN...J (and to verify its correctnes)). We notice that Q = E (i,
i 11-1, once
the number of indices E IN -1 such that j: = 1. Since [[Q] = p(N - 1), we have E (4, i--1) - 1) + 12) = Lip(N - 1) + 12((1 p) 1) 1).
L
Theorem 2. Under Assumptions
aadd Algorithm 7 runs at least as fast as SI in expectation.
Proof. Suppose we have a drafter model fı, a target model f2, and a prompt &go- Assume that fi requires ( time units compute each of its outputs, and f2 requires l2 time units, where 4 < l2- Assume that given the prompt Igi = I<o @ x1 @-.. Ir the probability that fi returns the (correct) token 2++1 is p. Consider generating N with lookahead = k. At time = 0, SI starts generating draft tokens, by the definition of SI. At time- = kt,, SI completes generating the first &s draft tokens zi. 22 1.1 ..... 26 I.....1 At time = kt1 + t2, 2ig 1...... Let A1, A2, variables sampled as follows. A; = 1 with probability p and A, = 0 otherwise, for all E [k 1]. Let min A; = 0} 1. Note that is distributed as the number of accepted drafts among the first k drafts of SI (or DSI). SI completes generating the first n + 1 tokens at time = kll + l2 SI. The first iteration of SI cannot output tokens at positions > ++1 1. by the definition of SI. The carliest time at which SI can complete generating r by the end of its second iteration. Hence, SI completes generating Tx++2 at time 2 2(kt) + 12). ta
tokens from f2 using the SI (or DSI) algorithm
SI completes verifying the first &o tokens 2/ 22.
Ak+1 be indicator
for any n E {0, 1.................. k). ... by the definition
Consider DSI with the same f1, f2, and lookahead over least kt,
servers. We show that DSI
can complete generating F time I 2(kt) t2), and, in expectation, time 2(kt) + t2). By the definition of DSI, DSI never preempt the current verifier. At time = kt1, DSI invokes concurrently (i) the verifying of the batch containing the first l tokens x|, 1111 ....,,k 2 I.... that are by the definition of DSI. We use coupling argument to align the two algorithms over the indicator variables A, for all If n h. then both SI and DSI complete generating the (k + 1)th first token F4+1 at time kt, 12. At that time, DSI either invokes a new current verifier thread or labels an existing thread as the current verifier (depending on 3t and the lookahead). Hence, DSI completes generating 2&+2 at time < kt1 + 26g. exactly when the current verifier thread completes its verification. DSI is faster than SI for all ti, t2, since kt1 + 2t < 2(kt) + tz). Otherwise, both algorithms accept the first n + 1 tokens at time < kt, + tz. At that time, the proof repeats for N - (n + 1).
not yet verified, and (ii) the drafting of #k+1 Tk+2 .....,
1.....1 1.....
,.. 1.....1
F EXPERIMENTS DETAILS
F.I TTFT AND TPOT
This section elaborates on our separate experiment that estimates the expected TTFT and TPOT.
To ensure the wait waiting times in our simulations are realistic, we distinguish between time to first token (TTFT) and time per output token (TPOT). We estimated the TTFT and TPOT latencies for each combination of model and dataset independently
single NVIDIA A100 80GB GPU.
For each combination (d, f) of a dataset d and a corresponding target or drafter model f. we estimate the average latencies of f in the following manner. First, we select 50 prompts from d uniformly at random, and for each prompt, generate 20 tokens using f. measuring the latency for each token in milliseconds. Following prior work, we distinguish between Time to First Token (TTFT) generation and Time Per Output Token (TPOT) generation (of all subsequent 19 tokens). Finally, we calculate the average TTFTs and TPOT's over all prompts per model-dataset pair, to estimate the expected latency
20

---

Published as a conference paper at ICLR 2025
of single forward pass. In all our experiments that measure the latency of computing forwards (i.e., TTFT and TPOT), models run with an MP degree of one and without memory offloading. We used a single NVIDIA A100 80GB GPU and the measured model was fully loaded to the GPU memory.
Our main experiment (Table 2 and an ablation experiment ($4.I) both use the estimated TTFT and TPOT follows: generating the first token adds wait of TTFT while generating each subsequent token adds a wait of TPOT. Since TTFT is usually significantly longer than TPOT (which dominates the overall sequence generation time), all latency figures in Table estimated TPOT latency of the target model and the drafter are shown in "Target Latency (ms)" and "Drafter Latency (ms)", respectively. We also report the ratio between the target and drafter latencies and present it in percentages (" ("Drafter Latency (%)"). The effective prefillig-decoddin latencies ratio of every pair of model and a dataset provided in Table
TPOT, for brevity. The
Table 3: The ratio between the estimated time to first token (TTFT) and time per output token (TPOT) for various off- fff-the-shelf models and datasets.
Model
TIFITTPT Ratio
Imsys/vicuna- 13b-v1.3
cnn. dailymail dailymail
5.36
double double7/vicuns-6680
1:04
Imsys/vicuns 133bv 3.3
danielkorau/slpaa
1.15
domble7/vicume 68m
dunielkoratdalpaca cun. duilymuil
1.05
Imsys/vicuna- v1.3
4.53
double7/vicuna 68m
can. dailymail
Imsys/vicuna- 1.3
donielkoratáalpaca
1.19
kuuble7vvicuna-66mm
danielkoratdalpaca
1.06
strrede
openti/opensi humaneval
1.35
bigcode/tiny starender.py
openaifopenai Jurnnnvall
1.19
higrode/starender
mhpp
1.54
bigcode/tiny sttrrcdder.pp
mhpp
1.20
microsof/Phi..3 Ph...hh..mmiiiii 3-medim- 128k-instmctt
openaifopenai humaneval
microsoit/Phi-3-min 1aal-118kknnnssc
openai/openai haammioopraiihummaaaall humancval
1.23
MaaarosottPhi-3-mddddi 1288-instruct
mbpp
1.43
microsol/Phii 3-min- 128 instruct
mbpp
1.27
microsoft/Phi.3. 3-meiiii-mmmmmmm.....mm 288 instruct
enn daillymil
4.77
microsoft/Phi--mmiin 1--mmii--28kkinneee i8k--instruct
can.dailyaiil
3.88
F.2 ACCEPTANCE RATE
This section elaborates our separate experiment that estimates the expected acceptance rate.
To ensure our evaluation is realistic, we used real-world acceptance rates, calculated as follows. For any given combination of (target, drafter, dataset), we estimate their acceptance rate independently. For each input prompt from the dataset, we generate tokens from both the drafter and the target model. We then consider the lengths of the longest sequences of exact token matches between the target and the drafter. Below is simplified example where tokens are counted as English words. If the target generates "We can only see short distance ahead, but we can see plenty there that needs to be done. [...I" and the drafter generates "We can only see short distance ahead, we done. [...I",, then the longest sequence of Zim1 n, where
exact matches is 8 tokens long. The expected number of accepted drafts is ñ. == M n; is the number of accepted draft tokens for the ith prompt. The acceptance rate is then calculated I+ In general, the reported acceptance
from geometric distribution, (acceptance rate) := 1 1 rate is guaranteed to converge to the expected acceptance rate as N oo (Appendix [.2.1]) In this experiment, the reported acceptance rate calculated based on generating 256 tokens for each prompt.
This experiment suggests that off-the-shelf model "families" like StarCoder (Li et al., 2023a) or Vicuna (Zheng et al. 2023) can form good pairs of target and drafter because their acceptance rates are relatively higher, as reported in Table These families consist models of different sizes that were trained similarly and similar datasets. We observe that even relatively small drafters demonstrate good alignment with larger models from the same family. For example, St Srrooder- - 168M (drafter) and St rooder-11 5B (target) yield an acceptance rate of 93%.
21

---

Published as a conference paper at ICLR 2025
F.2.1 ACCEPTANCE RATE FROM GEOMETRIC DISTRIBUTION
Both our main experiment (Table 2p and ablation ($4.1) measure the total latency of Non-SI, SI, and DSL. The total latency is dominated by the target and drafter forward passes. The numbers of required target and drafter forwards depend on the acceptance rate. To count the number of forwards, we can simulate the algorithms by sampling from the models, execute the verification procedure (for SI and DSI), and count the number of required forwards. Since sampling from models inherently stochastic, the total latency T is a random variable, and we focus on its expectation, E[T]).
There exists an acceptance rate p such that the expected latency satisfies B = =T. This acceptance rate can be estimated in two ways. The first method directly computes the likelihood that token is accepted by the verifier. Alternatively, p can be estimated by fitting a geometric distribution to the acceptance process. Specifically, this involves calculating the average number of tokens accepted per iteration and extracting p as the parameter of the geometric distribution.
Both approaches are consistent. As the number of iterations approaches infinity, the estimated acceptance rate converges to the true empirical acceptance rate p. Consequently, the estimated total latency based on p converges the actual total latency. This guarantees that the reported latency and the computed latency are equal in expectation.
Prior works in the field have developed theories and methods based on the simplifying assumption that token acceptance is independently and identically distributed (i.i.d.); for example, see Leviathan al. (2023). In this paper, estimating the acceptance rate from the fitted geometric distribution is based on the same assumption. The results in Mamou et al. (2024) reveal that such an assumption actually realistic. Figure 4 in MMmou et al. (024) shows that the number of accepted tokens per iteration follows a geometric distribution.
F.3 ABLATION VIA OFFLINE SIMULATION
This section provides more implementation details about the ablation analysis, involving a comple- mentary "offline" experiment ($4.1).
We simulate non-SI over all possible configurations of (drafter latency, acceptance rate) which are the cartesian product {0.01, 0.02, .... 1} x {0, 0.00,,002, .... 1} respectively. We simu- late SI over all possible configurations of (drafter latency, acceptance rate, lockahead) where lcokahead E (1.2...... 200). However, given an arbitrary number of available servers, it is possible to tune the 1 lookhhead hyperparameter such that DSI orchestrates only available servers. To ensure that DSI could be deployed on single node with up to eight GPUs, we simulate DSI over all configurations of SI that satisfy Equation [ for SP = 7, assuming that the drafter runs on a single GPU. For each combination of (drafter latency, acceptance rate, lookahead), we run the algorithm (SI or DSI) for five repeats and average the results. Since the lookahead is tunable parameter, our experiment assumes that it will be optimized by the user so that SI is op- timized for each configuration. To let SI select its optimal lookahead, the expected latency for each (drafter latency, acceptance rate) configuration is the minimal average latency among all the lookahead values.
To calculate the speedup of algorithm X over algorithm Y per (drafter latency, acceptance rate). we divide the latency of Y by the latency of X. The speedups are not smooth for drafter laten- cies < 20% due to the discretization of the lockahead hyperparameter. For example, the speedups for Lookahead
are smooth for both SI and DSI seen Appendix F.7
As in online experiments, every forward pass in offline experiments is replaced with adding the realistic wait time the total latency. Therefore, the latency of non-SI in offline simulations is simply the target forward latency (i.e., the average time that it takes to compute single forward pass of the target model, which measured in a separate experiment that is described in the previous section) times the number of tokens to generate. For example, if the target forward latency is 30ms and the number of tokens to generate is 100, then the latency of non-SI is 3000ms. To estimate the expected latency of SI, we must introduce the lookahead hyperparameter and the acceptance rate of the drafter. Under the randomness of the acceptance rate, we count the number of target and drafter forward passes. Every draft token that is accepted is equivalent to generating another output token, hence reducing the number of remaining target forwards by one. For example, if the acceptance rate corresponds accepting 1.5 drafts per iteration, then the expected number of target
22

---

Published as a conference paper at ICLR 2025
forward passes is 100 40 because every target forward is expected yield 1.5 2.5 output tokens (i.e., accepted drafts plus an additional token). The number drafter forwards depends on the Lookahead hyperparameter. For example, if lookahead
5, then the number of drafter forwards is 40 - 5 = 200, because we have 5 drafter forwards for each target forward. The expected lateney of SI is then the sum of the expected latencies of the target and drafter forwards. For example, if the drafter forward latency is 6ms, then the expected latency of SI is 200 6 - 40 30 2400ms, which is 1.25x faster than non-SI. The simulation of DSI is similar the simulation of SI. In both, the only randomness is the acceptance rate of the drafter. However, since DSI can successfully hide the lateney of almost all the target forwards, naive summing of the forward pass latencies cannot estimate its total latency accurately. Instead, the DSI simulation is based on an analysis that sums target forward latencies only if they are not hidden by speculation parallelism.
F.4 SIMULATION OF SI
We open-sourced our implementation of DSI for the single-node setup and the code for all the experiments in this paper. Below is the code estimating an lower bound of the end-to-end latency of the SI algorithm. The end-to-end latency is the overall wall time, including the prefilling and decoding latency, and excluding any tokenization latency. It is an lower bound because we ignore any real-world latencies differ from the forward passes latencies. For example, we ignore all the latencies corresponding switching between the models, communication between the CPU and the GPU, etc.
def si target latency: float, drafter latency: float lookahead: int N: int) float:
total_cost: float =
total-toks:
while votal_toks <:N:
num_accepted:
get_numaaccepted)
total.toks += num_accepted +
total-cost += lookahead drafter.latency + larget
return total
F.5 MODELS
For all models, we retrieve model weights from Hugging Face provide the URLs for each model used:
For clarity and reproducibility, we
Vicuna-133 http:///nngiinnteeeeeees///////a tributed under Non-Commercial
dis-
Vicuna-77 7B: htteas///uuggignneee.cc/////////
dis-
tributed under Non-Commerrial License
Vicuna (ecunaa66MMhht0
https///h//aa////oooooooooo////a disttrrrbbuu
distributed
under the Apache License 2.0.
Starcoder-15B:
httpss//whhgiiieeee..coooo///oo dis-
tributed under the Responsible Al License
Starcoder-163M.
https://oggiinneee..///ooooo/o/.
starcoder_py also distributed under the Responsible Al License
Phi3-14B:
distributed under the MIT license
Phi3-4B:
https://huga: ctt.a/cco..mccoooooooofo....
www.iii1kkk-isttreeee distributed under the MIT license
F.6
DATASETS AND PROMPTS
We use standard datasets from Hugging Face and standard prompts from the state-of-the-arr,
F.
MBPP
MBPP dataset consists crowd-soureed Python programming problems and is distributed under the cc-by-4.0 License
23

---

Published as a conference paper at ICLR 2025
Concerning the prompt, followed (Ben Allal et al., 2022 Fried et al. 2023 and included the description of the programming task and a single test to verify solution, in order to help the model catch the signature of the function (see Figure
(text] }
Figure 3: MBPP Prompt
F.6.2 HUMANEVAL
HumanEval dataset includes programming problems and distributed under the MIT License Prompt contains only prompt field from the dataset.
F.6.3
CNN-DM
CNN-DM contains news articles and is distributed under the Apache License 2.0
We included the article field in the prompt as Figure
n Summariaz rize:
(article)
Summary:
Figure 4: CNN-DM Prompt
F.6.4 ALPACA
Alpaca dataset contains instructions and demonstrations. It is distributed under the cc-by-nc-4.0 License
We follow Taori al. (2023) to define the prompts. For samples with non-empty input field, the prompt as in Figure while for samples with empty input field, we use the prompt as Figure 6
"""Below is an instruction that describes task, paired with an input that provides further context. Write response that appropriately completes the request.
4## Instruction:
(instruction)
### Input: (input.)
## Response:
Figure 5: Alpaca prompt for samples with non-empty input field.
24

---

Published as a conference paper at ICLR 2025
"""Below is an instruction that describes task. Write response that appropriately completes the request.
### Instruction: (instruction)
### Response:
Figure 6: Alpaca prompt for samples with empty input field.
F.7 SPEEDUPS FOR =
Figure illustrates the performance comparison between Speculative Inference (SI), non- Speculative Inference (non-SI), and Dynamic Speculative Inference (DSI) for static lookahead of lookahead = 5. The figure consists of three heatmaps, each representing pairwise comparison of these algorithms across different drafter speeds and accuracies.
In Figure 7a), compare SI to non-SL. The pink regions indicate scenarios where SI is slower than non-SI, which occurs when the drafter is either too slow or inaccurate. This highlights the limitations of SI in certain conditions.
Figures b) and (c) demonstrate that DSI consistently outperforms both SI and non-SI across all drafter configurations. This empirical evidence supports theoretical findings that DSI is faster than both SI and non-SI in expectation, regardless of the drafter's speed or accuracy.
These results underscore the robustness and efficiency of DSI compared to existing inference methods, particularly in scenarios where SI might falter due to suboptimal drafter performance.
9
Acceptance Rate
0.8
0.8
8
Arcoptance Rate
0.8
0.6
0.6
6
0.6
6
0.4
3
0.2 04
0.8
1.0
0.2
06
0.8 1.0
0.8 Drafter Latency
0.4 0.6
1.0
Drafter Latency
Drafter Latency
(a) SI/non-SI
(b) S1/DSI
(c) non-SI/DSI
Figure 7: Each heatmap (labeled "X/Y") plots the ratio between the run time of algorithm X and the run time of algorithm Y. is run with 1 look Lhoka 5. (a): slower than speculaaee inference (non-Sl) when the drafter is either slow or inaccurate enough (pink marks slowdowns). DSI never slower than either SI non-SI. (b, c): DSI is always faster than speculative inference (SI) and non-speculative inference (non-SI) algorithms for various drafters.
25


Reviews:
[{"review_id": "PdWaFF4HCH", "reviewer": "Reviewer_U6hk", "summary": "The paper introduces a novel approach called Distributed Speculative Inference (DSI) for accelerating LLM inference. The algorithm is inspired by Speculative Inference (SI) but uses multiple drafters (small) and overlaps their execution with verifications from the verifier (large) to obtain greater speedup than SI. The authors show theoretically that DSI is faster than SI and standard LLM inference in expectation and \n also show empirical speedups.", "questions": "1. I do not understand the rationale behind the formula for acceptance rate on Page 8, line 410. Please clarify.\n\n2. Why are you using an estimate of the acceptance rate instead of independently simulating acceptance/rejection for each drafter, target pair? For e.g. you could generate 'lookahead' tokens, verify them, go back to the drafter, and so on until the entire sequence is generated and then use the estimates of TTFT and TPOT to estimate the total latency. \n\n3. Please report the value of lookahead used for each row of Table 2, and clarify why that value was chosen.\n\n4. Why are real world latencies of multithreading and context switching ignored (Page 10, lines 487-488)? These contribute to the overhead of DSI, and I feel that they should be reported and studied for a fair comparison of with SI and non-SI.\n\n5. Will it be possible to show any simulation results with multiple drafters? I feel that one of the strengths of DSI is that it can work with multiple drafters and I would like to see how using multiple drafters compares with using a single drafter.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. DSI appears to be a novel approach and effectively leverages the strengths of SI but also seeks to address its weaknesses like the drafter being blocked by verifier until the generated tokens are verified and also generalizes SI to work with multiple drafters in parallel.\n\n2. The paper provides empirical evidence demonstrating that DSI outperforms both traditional speculative inference (SI) and non-SI methods", "weaknesses": "1. The approach requires more compute and memory (multiple KV cache) than SI with the amount of extra compute and memory required increasing with increase in number and size of the drafters, and with increase in input context length. Therefore, I believe that the increase in cost/resources also needs to also be considered in addition to the reduction in latency to make a fair assessment.\n\n2. Evaluation is performed using simulations and not real implementations due to which it is not entirely clear if the gains will hold up in real settings. However, I understand that it is challenging to procure the amount of compute for inference with multiple LLMs and so I am willing to let this pass if my questions below on the evaluation are adequately answered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EgXg6ysw4p", "reviewer": "Reviewer_kPPx", "summary": "The paper introduces Distributed Speculative Inference (DSI), an inference algorithm that accelerates large language model (LLM) inference more effectively than traditional Speculative Inference (SI) methods or standard autoregressive techniques. Unlike SI, DSI achieves speed improvements even when these drafters are slow or less precise. Speculation Parallelism (SP) enables target and drafter models to overlap in execution, reducing latency and introducing a new balance between computational resources and response time", "questions": "* What kind of communications are involved in distributed speculative parallelism and how does that affect the latency?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Paper is well written upto method. \n* Proposed Speculation parallelism effectively solves the blocking operation of speculative inference into non-blocking. \n* Method is backed by proof.", "weaknesses": "* Evaluations are very weak. Can we get the experimental result by varying lookahead values on different machines?\n* Actual improvement measurement seems missing. \n* Evaluating distributed evaluation on a single machine doesn’t make sense.. Authors really need to find way to run the evaluation on multi-GPU machines. Distribution involves communication latency as well, so in practice there are many things involved but these practical concerns are not discussed in the paper. \n* Evaluation writing is also very confusing..", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "APNZDOmP3d", "reviewer": "Reviewer_4xNe", "summary": "The paper introduces the parallel verification for accelerating speculative decoding. Experiments show speedups compared to the original speculative decoding.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n\n- The idea is clear and seems to work well through evaluation.", "weaknesses": "- The largest target model is capped at 13B.\n\n  - How does the model work on larger models?\n\n- The method need multiple GPUs to host several target model. This can cause deployment issues and energy issues.\n\n  - How does the data communication defect the approach? The KV-cache also seems to move between GPUs. How about synchronization issues?\n  - The analysis of the energy consumption compared to the original speculative decoding method seems to missing. More justification may be needed given the energy inefficiency of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fkIuvL1guw", "reviewer": "Reviewer_LmXh", "summary": "This work proposes distributed speculative inference (DSI), or speculation parallelism in particular,\nwhich accelerates speculative inference (SI) by leveraging more hardware resources (e.g. GPUs) to hide the latency of the verification steps in SI via parallelism and careful synchronization across multiple threads.\nThrough rigorous analysis,\nit is shown that, while SI can be slower than standard autoregressive inference (non-SI) in certain scenarios, \nDSI is provably faster in expectation than both of them.\nWith a proof-of-concept implementation of DSI and careful design of numerical simulations,\nempirical results suggest that DSI achieves 1.29-1.92x speedup over SI in various settings.", "questions": "- Do you think DSI with a batch size larger than 1 is possible, at least in theory?\nBatch inference is already quite a challenge for SI, and things seem to get much more complicated with the proposed speculation parallelism.\n\n\n- Some minor issues about writing:\n  - The paragraph between Lines 422 - 487 is a bit long; might be better to divide it into several paragraphs\n  - Line 128 mentions \"implementing (3) above\", while Eq. (3) is actually located in the appendix \n  - Line 303, $f_1$ and $f_2$: should they be $f_j$ and $f_m$ instead?\n  - Line 931, \"upper bound\": should it be \"lower bound\", since some components of latency in reality are neglected in the estimate?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed speculation parallelism is novel and interesting, and might inspire future work on speculative inference or more general LLM inference acceleration.\n\n- The rigor in both theoretical analysis and numerical validation (via careful design of simulations) is appreciated.\n\n- The source code for the experiments is provided.", "weaknesses": "My main concern is that the practical value of this work is not yet evident enough.\nIf I understand correctly, the implementation of DSI in the current work is more or less a proof-of-concept, rather than one that is ready for application.\nMore specifically, only the outline of the multi-thread system is implemented, while the LLM part (e.g. forward passes of the drafter or target model) is not actually implemented but rather replaced by `WAIT` commands with delays that mimic the actual latencies of real LLM forward passes, for the purpose of simulations.\nOne might argue that, for a full implementation of DSI, we simply need to plug the LLM forward passes into the multi-thread system.\nBut this does not seem straightforward to me, since LLM inference is a complicated process by itself; \ncombining it with the implemented multi-thread system can be challenging, as these two might not be completely decoupled.\n\n\n\nOne particular aspect I can think of is the compatibility with KV caches, which have been supported by both non-SI and SI, and enabled by default in most cases (e.g. when huggingface transformers is used for profiling the latency of standard LLM inference in the `ExperimentLatency` class within the source code of this work).\nWith multiple target LLMs (or verifiers) on different GPUs, each at its own pace, \nmanaging and synchronizing their KV caches seems like a non-trivial task and requires quite some engineering work,\nwhich has not been implemented in this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)