Direct Multi-Token Decoding
Authors: name: Xuan Luo; name: Weizhi Wang; name: Xifeng Yan
Published: 2025-10-13 21:42:37

Abstract: Decoder-only transformers have become the standard architecture for large
language models (LLMs) due to their strong performance. Recent studies suggest
that, in pre-trained LLMs, early, middle, and late layers may serve distinct
roles: Early layers focus on understanding the input context, middle layers
handle task-specific processing, and late layers convert abstract
representations into output tokens. We hypothesize that once representations
have been processed by the early and middle layers, the resulting hidden states
may encapsulate sufficient information to support the generation of multiple
tokens using only the late layers, eliminating the need to repeatedly traverse
the early and middle layers. We refer to this inference paradigm as Direct
Multi-Token Decoding (DMTD). Unlike speculative decoding, our method introduces
no additional parameters, auxiliary routines, or post-generation verification.
Despite being trained on a limited dataset, a fine-tuned DMTD Qwen3-4B model
has already demonstrated promising results, achieving up to a 2x speedup with
only minor performance loss. Moreover, as shown in our scaling analysis, its
performance is expected to further improve with larger training datasets.