Efficient Generative Modeling with Residual Vector Quantization-Based
  Tokens
Authors: name: Jaehyeon Kim; name: Taehong Moon; name: Keon Lee; name: Jaewoong Cho
Published: 2024-12-13 15:31:17

Abstract: We explore the use of Residual Vector Quantization (RVQ) for high-fidelity
generation in vector-quantized generative models. This quantization technique
maintains higher data fidelity by employing more in-depth tokens. However,
increasing the token number in generative models leads to slower inference
speeds. To this end, we introduce ResGen, an efficient RVQ-based discrete
diffusion model that generates high-fidelity samples without compromising
sampling speed. Our key idea is a direct prediction of vector embedding of
collective tokens rather than individual ones. Moreover, we demonstrate that
our proposed token masking and multi-token prediction method can be formulated
within a principled probabilistic framework using a discrete diffusion process
and variational inference. We validate the efficacy and generalizability of the
proposed method on two challenging tasks across different modalities:
conditional image generation} on ImageNet 256x256 and zero-shot text-to-speech
synthesis. Experimental results demonstrate that ResGen outperforms
autoregressive counterparts in both tasks, delivering superior performance
without compromising sampling speed. Furthermore, as we scale the depth of RVQ,
our generative models exhibit enhanced generation fidelity or faster sampling
speeds compared to similarly sized baseline models. The project page can be
found at https://resgen-genai.github.io