FR-Spec: Accelerating Large-Vocabulary Language Models via
  Frequency-Ranked Speculative Sampling
Authors: name: Weilin Zhao; name: Tengyu Pan; name: Xu Han; name: Yudi Zhang; name: Ao Sun; name: Yuxiang Huang; name: Kaihuo Zhang; name: Weilun Zhao; name: Yuxuan Li; name: Jianyong Wang; name: Zhiyuan Liu; name: Maosong Sun
Published: 2025-02-20 18:58:10

Abstract: Speculative sampling has emerged as an important technique for accelerating
the auto-regressive generation process of large language models (LLMs) by
utilizing a draft-then-verify mechanism to produce multiple tokens per forward
pass. While state-of-the-art speculative sampling methods use only a single
layer and a language modeling (LM) head as the draft model to achieve
impressive layer compression, their efficiency gains are substantially reduced
for large-vocabulary LLMs, such as Llama-3-8B with a vocabulary of 128k tokens.
To address this, we present FR-Spec, a frequency-ranked speculative sampling
framework that optimizes draft candidate selection through vocabulary space
compression. By constraining the draft search to a frequency-prioritized token
subset, our method reduces LM Head computation overhead by 75% while ensuring
the equivalence of the final output distribution. Experiments across multiple
datasets demonstrate an average of 1.12times speedup over the
state-of-the-art speculative sampling method EAGLE-2.