Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models
Authors: Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Nadav Timor, Moshe Wasserblat, Roy Schwartz


Reviewer paper summaries: This paper proposes an approach to dynamically decide the number of draft tokens at each generation step given the likelihood of the next draft token being accepted.
Summary of strengths: 1.⁠ ⁠The method is well motivated by the fact that static draft length won't work in all cases, especially when the optimal acceptance tokens at each step are with high variance. 

2.⁠ ⁠The proposed method leverages a classifier to dynamically determine a score and compares it with a threshold to decide whether the drafting continues. Although it seems like a simplified version of SpecDec++ [1] which treats the speculative process as a Markov Decision Process, this simple design beats heuristic-based methods in terms of latency and speedup.

3.⁠ ⁠Experiments are relatively complete given the length limit of a short paper.
Summary of weaknesses: 1.⁠ ⁠This methodology only explores training-free speculative decoding pipeline. Methods like EAGLE, Medusa and Hydra train a drafter that produces a fixed number of draft tokens, which will heavily lower the variance of the optimal acceptance tokens at each step.

2.⁠ ⁠Definitely not worth rejection, but the proposed method is less exciting when employing tree attention as the verification method, as the tree verification nearly takes identical latency when the number of draft tokens grows. And that's why concurrent works [2, 3] focus on adjusting the optimal tree structure based on the drafter's confidence, instead of adjusting candidate length in this work.

[1] Huang K, Guo X, Wang M. SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths[J]. arXiv preprint arXiv:2405.19715, 2024.

[2] Wang J, Su Y, Li J, et al. OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure[J]. arXiv preprint arXiv:2406.17276, 2024.

[3] Li Y, Wei F, Zhang C, et al. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees[J]. arXiv preprint arXiv:2406.16858, 2024.
Comments, suggestions and typos: See above.
Overall assessment scores: 3