Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference

Authors: Nadav Timor, Jonathan Mamou, Daniel Korat, Moshe Berchansky, Oren Pereg, Moshe Wasserblat, Tomer Galanti, Michal Gordon-Kiwkowitz, David Harel
Primary area: generative models
Keywords: inference algorithms for generative models, llm inference, speculative decoding

Abstract:
This paper introduces *distributed speculative inference (DSI)*, a novel inference algorithm that is provably faster than speculative inference (SI) [leviathan2023, chen2023, miao2024, sun2025, timor2025] and standard autoregressive inference (non-SI). Like other SI algorithms, DSI operates on frozen language models (LMs), requiring no training or architectural modifications, and it preserves the target distribution. Prior studies on SI have demonstrated empirical speedups over non-SI—but rely on sufficiently fast and accurate drafters, which are often unavailable in practice. We identify a gap where SI can be slower than non-SI if drafters are too slow or inaccurate. We close this gap by proving that DSI is faster than both SI and non-SI—given any drafters. DSI is therefore not only faster than SI, but also unlocks the acceleration of LMs for which SI fails. DSI leverages *speculation parallelism (SP)*, a novel type of task parallelism, to orchestrate target and drafter instances that overlap in time, establishing a new foundational tradeoff between computational resources and latency. Our simulations show that DSI is 1.29-1.92x faster than SI in single-node setups for various off-the-shelf LMs and tasks. We open-source all our code.

Decision: accept (poster)
Reviewer scores: 6, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730648698886

thread_type:
official_review

<thread_object>
creator: reviewer_lmxh

<content>
summary:
This work proposes distributed speculative inference (DSI), or speculation parallelism in particular,
which accelerates speculative inference (SI) by leveraging more hardware resources (e.g. GPUs) to hide the latency of the verification steps in SI via parallelism and careful synchronization across multiple threads.
Through rigorous analysis,
it is shown that, while SI can be slower than standard autoregressive inference (non-SI) in certain scenarios, 
DSI is provably faster in expectation than both of them.
With a proof-of-concept implementation of DSI and careful design of numerical simulations,
empirical results suggest that DSI achieves 1.29-1.92x speedup over SI in various settings.

strengths:
- The proposed speculation parallelism is novel and interesting, and might inspire future work on speculative inference or more general LLM inference acceleration.

- The rigor in both theoretical analysis and numerical validation (via careful design of simulations) is appreciated.

- The source code for the experiments is provided.

weaknesses:
My main concern is that the practical value of this work is not yet evident enough.
If I understand correctly, the implementation of DSI in the current work is more or less a proof-of-concept, rather than one that is ready for application.
More specifically, only the outline of the multi-thread system is implemented, while the LLM part (e.g. forward passes of the drafter or target model) is not actually implemented but rather replaced by `WAIT` commands with delays that mimic the actual latencies of real LLM forward passes, for the purpose of simulations.
One might argue that, for a full implementation of DSI, we simply need to plug the LLM forward passes into the multi-thread system.
But this does not seem straightforward to me, since LLM inference is a complicated process by itself; 
combining it with the implemented multi-thread system can be challenging, as these two might not be completely decoupled.



One particular aspect I can think of is the compatibility with KV caches, which have been supported by both non-SI and SI, and enabled by default in most cases (e.g. when huggingface transformers is used for profiling the latency of standard LLM inference in the `ExperimentLatency` class within the source code of this work).
With multiple target LLMs (or verifiers) on different GPUs, each at its own pace, 
managing and synchronizing their KV caches seems like a non-trivial task and requires quite some engineering work,
which has not been implemented in this work.

questions:
- Do you think DSI with a batch size larger than 1 is possible, at least in theory?
Batch inference is already quite a challenge for SI, and things seem to get much more complicated with the proposed speculation parallelism.


- Some minor issues about writing:
  - The paragraph between Lines 422 - 487 is a bit long; might be better to divide it into several paragraphs
  - Line 128 mentions "implementing (3) above", while Eq. (3) is actually located in the appendix 
  - Line 303, $f_1$ and $f_2$: should they be $f_j$ and $f_m$ instead?
  - Line 931, "upper bound": should it be "lower bound", since some components of latency in reality are neglected in the estimate?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696889584

thread_type:
official_review

<thread_object>
creator: reviewer_4xne

<content>
summary:
The paper introduces the parallel verification for accelerating speculative decoding. Experiments show speedups compared to the original speculative decoding.

strengths:
- The paper is well-written and easy to follow.

- The idea is clear and seems to work well through evaluation.

weaknesses:
- The largest target model is capped at 13B.

  - How does the model work on larger models?

- The method need multiple GPUs to host several target model. This can cause deployment issues and energy issues.

  - How does the data communication defect the approach? The KV-cache also seems to move between GPUs. How about synchronization issues?
  - The analysis of the energy consumption compared to the original speculative decoding method seems to missing. More justification may be needed given the energy inefficiency of the method.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730965847071

thread_type:
official_review

<thread_object>
creator: reviewer_kppx

<content>
summary:
The paper introduces Distributed Speculative Inference (DSI), an inference algorithm that accelerates large language model (LLM) inference more effectively than traditional Speculative Inference (SI) methods or standard autoregressive techniques. Unlike SI, DSI achieves speed improvements even when these drafters are slow or less precise. Speculation Parallelism (SP) enables target and drafter models to overlap in execution, reducing latency and introducing a new balance between computational resources and response time

strengths:
* Paper is well written upto method. 
* Proposed Speculation parallelism effectively solves the blocking operation of speculative inference into non-blocking. 
* Method is backed by proof.

weaknesses:
* Evaluations are very weak. Can we get the experimental result by varying lookahead values on different machines?
* Actual improvement measurement seems missing. 
* Evaluating distributed evaluation on a single machine doesn’t make sense.. Authors really need to find way to run the evaluation on multi-GPU machines. Distribution involves communication latency as well, so in practice there are many things involved but these practical concerns are not discussed in the paper. 
* Evaluation writing is also very confusing..

questions:
* What kind of communications are involved in distributed speculative parallelism and how does that affect the latency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731143254764

thread_type:
official_review

<thread_object>
creator: reviewer_u6hk

<content>
summary:
The paper introduces a novel approach called Distributed Speculative Inference (DSI) for accelerating LLM inference. The algorithm is inspired by Speculative Inference (SI) but uses multiple drafters (small) and overlaps their execution with verifications from the verifier (large) to obtain greater speedup than SI. The authors show theoretically that DSI is faster than SI and standard LLM inference in expectation and 
 also show empirical speedups.

strengths:
1. DSI appears to be a novel approach and effectively leverages the strengths of SI but also seeks to address its weaknesses like the drafter being blocked by verifier until the generated tokens are verified and also generalizes SI to work with multiple drafters in parallel.

2. The paper provides empirical evidence demonstrating that DSI outperforms both traditional speculative inference (SI) and non-SI methods

weaknesses:
1. The approach requires more compute and memory (multiple KV cache) than SI with the amount of extra compute and memory required increasing with increase in number and size of the drafters, and with increase in input context length. Therefore, I believe that the increase in cost/resources also needs to also be considered in addition to the reduction in latency to make a fair assessment.

2. Evaluation is performed using simulations and not real implementations due to which it is not entirely clear if the gains will hold up in real settings. However, I understand that it is challenging to procure the amount of compute for inference with multiple LLMs and so I am willing to let this pass if my questions below on the evaluation are adequately answered.

questions:
1. I do not understand the rationale behind the formula for acceptance rate on Page 8, line 410. Please clarify.

2. Why are you using an estimate of the acceptance rate instead of independently simulating acceptance/rejection for each drafter, target pair? For e.g. you could generate 'lookahead' tokens, verify them, go back to the drafter, and so on until the entire sequence is generated and then use the estimates of TTFT and TPOT to estimate the total latency. 

3. Please report the value of lookahead used for each row of Table 2, and clarify why that value was chosen.

4. Why are real world latencies of multithreading and context switching ignored (Page 10, lines 487-488)? These contribute to the overhead of DSI, and I feel that they should be reported and studied for a fair comparison of with SI and non-SI.

5. Will it be possible to show any simulation results with multiple drafters? I feel that one of the strengths of DSI is that it can work with multiple drafters and I would like to see how using multiple drafters compares with using a single drafter.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732393442553

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
**General Response to All Reviewers**

We sincerely thank all the reviewers for their insightful and constructive feedback. We are encouraged by the positive recognition of our work's novelty, theoretical rigor, and potential impact on accelerating LLM inference.

**Key Strengths Highlighted:**

- **Novelty and Contribution:**
  - *Reviewer U6hk:* "DSI appears to be a novel approach and effectively leverages the strengths of SI but also seeks to address its weaknesses..."
  - *Reviewer kPPx:* Our "proposed Speculation Parallelism effectively solves the blocking operation of speculative inference into non-blocking."
  - *Reviewer LmXh:* "The proposed speculation parallelism is novel and interesting, and might inspire future work on speculative inference or more general LLM inference acceleration."
  - *Reviewer 4xNe:* "The idea is clear and seems to work well through evaluation."

- **Theoretical Rigor:**
  - *Reviewer kPPx:* Noted that our "method is backed by proof."
  - *Reviewer LmXh:* Appreciated "the rigor in both theoretical analysis and numerical validation (via careful design of simulations)."

- **Empirical Validation:**
  - *Reviewer U6hk:* Observed that "the paper provides empirical evidence demonstrating that DSI outperforms both traditional speculative inference (SI) and non-SI methods."
  - *Reviewer 4xNe:* Mentioned that "experiments show speedups compared to the original speculative decoding."

- **Clarity and Presentation:**
  - *Reviewer kPPx:* Stated that the "paper is well written up to the method section."
  - *Reviewer 4xNe:* Found the paper "well-written and easy to follow."
  - *Reviewer LmXh:* Commented that "the paper is well-written and easy to follow."

- **Potential for Future Research:**
  - *Reviewer LmXh:* Mentioned that our work "might inspire future work on speculative inference or more general LLM inference acceleration."

**Revisions Made:**

- **Enhanced Evaluation Section:** We have improved the clarity of the evaluation section, providing more detailed explanations of our methodology and results to address concerns about the evaluations' strength and clarity, as suggested by *Reviewers kPPx* and *LmXh*.

- **Clarified Implementation Details:** In response to *Reviewers U6hk* and *LmXh*, we have provided additional explanations regarding the practical implementation of DSI, including KV cache management.

- **Addressed Resource and Deployment Concerns:** We have discussed the resource requirements and potential deployment challenges highlighted by *Reviewers U6hk* and *4xNe*, including the use of multiple GPUs and (energy) efficiency considerations.

- **Corrected Minor Issues:** We have fixed all identified typos and improved the writing for better flow and accuracy, as pointed out by *Reviewer LmXh*.

We believe these revisions address the concerns raised and enhance the overall quality of our work. We appreciate the reviewers' insights, which have significantly contributed to improving our paper.

Thank you once again for your time and consideration.
</content>
</thread_object>

</thread>
<thread>
creation date:
1740881774558

thread_type:
public_comment

<thread_object>
creator: ~nadav_timor1

<content>
comment:
We sincerely thank the reviewers and the area chair for their thoughtful feedback and support. We’ve made updates to the camera-ready version to better address the raised concerns, including expanded explanations of the acceptance rate estimation method and KV cache management implementation.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work introduces a new method for parallel inference in large language models that improves speed, theoretical rigor, and clarity. Most reviewers agreed on its novelty, strong analysis, and clear writing, though some raised concerns about implementation complexity and energy use. After the authors explained their approach and provided more details, opinions shifted towards a more positive stance. While challenges remain in scaling and real-world deployment, the method’s potential speedups are promising. Overall, the paper’s novelty, in particular the theoretical analysis, and the overall contributions and responses justify acceptance. Minor improvments and further practical research are encouraged.