Accelerating Auto-regressive Text-to-Image Generation with Training-free
  Speculative Jacobi Decoding
Authors: name: Yao Teng; name: Han Shi; name: Xian Liu; name: Xuefei Ning; name: Guohao Dai; name: Yu Wang; name: Zhenguo Li; name: Xihui Liu
Published: 2024-10-02 16:05:27

Abstract: The current large auto-regressive models can generate high-quality,
high-resolution images, but these models require hundreds or even thousands of
steps of next-token prediction during inference, resulting in substantial time
consumption. In existing studies, Jacobi decoding, an iterative parallel
decoding algorithm, has been used to accelerate the auto-regressive generation
and can be executed without training. However, the Jacobi decoding relies on a
deterministic criterion to determine the convergence of iterations. Thus, it
works for greedy decoding but is incompatible with sampling-based decoding
which is crucial for visual quality and diversity in the current
auto-regressive text-to-image generation. In this paper, we propose a
training-free probabilistic parallel decoding algorithm, Speculative Jacobi
Decoding (SJD), to accelerate auto-regressive text-to-image generation. By
introducing a probabilistic convergence criterion, our SJD accelerates the
inference of auto-regressive text-to-image generation while maintaining the
randomness in sampling-based token decoding and allowing the model to generate
diverse images. Specifically, SJD facilitates the model to predict multiple
tokens at each step and accepts tokens based on the probabilistic criterion,
enabling the model to generate images with fewer steps than the conventional
next-token-prediction paradigm. We also investigate the token initialization
strategies that leverage the spatial locality of visual data to further improve
the acceleration ratio under specific scenarios. We conduct experiments for our
proposed SJD on multiple auto-regressive text-to-image generation models,
showing the effectiveness of model acceleration without sacrificing the visual
quality.