Continuous Speculative Decoding for Autoregressive Image Generation
Authors: name: Zili Wang; name: Robert Zhang; name: Kun Ding; name: Qi Yang; name: Fei Li; name: Shiming Xiang
Published: 2024-11-18 09:19:15

Abstract: Continuous-valued Autoregressive (AR) image generation models have
demonstrated notable superiority over their discrete-token counterparts,
showcasing considerable reconstruction quality and higher generation fidelity.
However, the computational demands of the autoregressive framework result in
significant inference overhead. While speculative decoding has proven effective
in accelerating Large Language Models (LLMs), their adaptation to
continuous-valued visual autoregressive models remains unexplored. This work
generalizes the speculative decoding algorithm from discrete tokens to
continuous space. By analyzing the intrinsic properties of output distribution,
we establish a tailored acceptance criterion for the diffusion distributions
prevalent in such models. To overcome the inconsistency that occurred in
speculative decoding output distributions, we introduce denoising trajectory
alignment and token pre-filling methods. Additionally, we identify the
hard-to-sample distribution in the rejection phase. To mitigate this issue, we
propose a meticulous acceptance-rejection sampling method with a proper upper
bound, thereby circumventing complex integration. Experimental results show
that our continuous speculative decoding achieves a remarkable 2.33times
speed-up on off-the-shelf models while maintaining the output distribution.
Codes will be available at https://github.com/MarkXCloud/CSpD