Draft Model Knows When to Stop: A Self-Verification Length Policy for
  Speculative Decoding
Authors: name: Ziyin Zhang; name: Jiahao Xu; name: Tian Liang; name: Xingyu Chen; name: Zhiwei He; name: Rui Wang; name: Zhaopeng Tu
Published: 2024-11-27 15:53:17

Abstract: Speculative Decoding (SD) has become an important technique in accelerating
the inference speed of large language models. Conventional SD methods employ a
fixed draft length, which ignores the token generation difficulty across tasks.
Consequently, in this paper, we address such an issue and introduce SVIP - a
difficulty-aware dynamic draft length policy for speculative decoding systems.
Based on a theoretical lower bound of draft token acceptance rate and its
inference-time approximation, SVIP adaptively determines the lengths of draft
sequences based on the entropy of each draft token distribution. Experimental
results on mainstream SD benchmarks and frameworks demonstrate the superior
performance of SVIP, achieving up to 20\% walltime speedup on SpecBench over
baseline SD methods and 60\% speedup on MT-Bench for long-form generation of up
to 8K tokens. Moreover, SVIP is totally training-free and compatible with any
existing SD methods that generate draft tokens autoregressively. Experimental
results also show that SVIP yields consistent walltime improvement on top of
GliDe & CaPE and EAGLE-2.