Title: ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMs
Primary area: foundation or frontier models, including LLMs
Keywords: Parallel Decoding;Intrinsic Parallelism Mining;Large Language Models (LLMs) ;Hybrid Decoding Engine

Abstract: The increasing scale and complexity of large language models (LLMs) pose significant inference
latency challenges, primarily due to their autoregressive decoding paradigm characterized by the
sequential nature of next-token prediction. By re-examining the outputs of autoregressive models, we
observed that some segments exhibit parallelizable structures, which we term intrinsic parallelism.
Decoding each parallelizable branch simultaneously ($\textit{i.e}$. parallel decoding) can significantly improve
the overall inference speed of LLMs. In this paper, we propose an $\textbf{A}$daptive $\textbf{S}$erial-$\textbf{P}$arallel $\textbf{D}$ecoding ($\textbf{ASPD}$), which addresses two core challenges: automated construction of parallelizable
data and efficient parallel decoding mechanism. More specifically, we introduce a non-invasive
pipeline that automatically extracts and validates parallelizable structures from the responses of autoregressive
models. To empower efficient adaptive serial-parallel decoding, we implement a $\textbf{Hybrid Decoding Engine}$ which enables seamless transitions between serial and parallel decoding modes while
maintaining a reusable KV cache, maximizing computational efficiency. Extensive evaluations across
General Tasks, Retrieval-Augmented Generation and Mathematical Reasoning demonstrate that $\textbf{ASPD}$
achieves unprecedented performance in both effectiveness and efficiency. Notably, on Vicuna Bench,
our method achieves up to 3.10x speedup (1.82x on average) while maintaining response quality
within 1\% difference compared to autoregressive models, realizing significant acceleration without
compromising generation quality.
The source code is available for review at an anonymous repository: https://anonymous.4open.science/r/ASPD.

TL;DR: ASPD extracts intrinsic parallelism and utilizes a Hybrid Decoding Engine to achieve up to 3.19× speedup on Vicuna Bench with <1% quality loss.