Efficient Generative Modeling with Residual Vector Quantization-Based Tokens
Authors: Jaehyeon Kim
Taehong Moon
Keon Lee
Jaewoong Cho
Venue: ICML 2025 poster (2025)
Keywords: generative models
discrete diffusion
residual vector quantization
transformer
text-to-speech
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: We introduce ResGen, an efficient Residual Vector Quantization (RVQ)-based generative model for high-fidelity generation with fast sampling. RVQ improves data fidelity by increasing the number of quantization steps, referred to as depth, but deeper quantization typically increases inference steps in generative models. To address this, ResGen directly predicts the vector embedding of collective tokens rather than individual ones, ensuring that inference steps remain independent of RVQ depth. Additionally, we formulate token masking and multi-token prediction within a probabilistic framework using discrete diffusion and variational inference. We validate the efficacy and generalizability of the proposed method on two challenging tasks across different modalities: conditional image generation on ImageNet 256×256 and zero-shot text-to-speech synthesis. Experimental results demonstrate that ResGen outperforms autoregressive counterparts in both tasks, delivering superior performance without compromising sampling speed. Furthermore, as we scale the depth of RVQ, our generative models exhibit enhanced generation fidelity or faster sampling speeds compared to similarly sized baseline models.

Efficient Generative Modeling with Residual Vector Quantizatin-Baased Tokens
Jachyeon Kim 11
Tachong Moon *2
Keon Lee² Jaewoong Cho
Abstract
et al., 2023; Shen et al., 2024; Rubenstein et al., 2023). While these models have demonstrated remarkable success, particularly with the effective scaling with both data and model sizes (Kaplan et al., 2020; Peebles & Xie, 2023), challenges remain when aiming for high-fidelity genera- tion, especially in terms of balancing generation quality with computational efficiency. The demand for more de- tailed, high-resolution outputs such as images (Kang et al., 2023; He et al., 2023), videos (Bar-Tal et al., 2024) and audio (Evans et al., 2024; Copet et al., 2024) has led to the exploration of new approaches that can handle long input sequences and complex data structure effectively (Saharia et al., 2022; Ding al., 2023).
We introduce ResGen, an efficient Residual Vec- tor Quantization (RVQ)-based generative model for high-fidelity generation with fast sampling. RVQ improves data fidelity by increasing the num- ber of quantization steps, referred as depth, but deeper quantization typically increases inference steps in generative models. To address this, Res- Gen directly predicts the vector embedding of collective tokens rather than individual ones, en- suring that inference steps remain independent of RVQ depth. Additionally, we formulate to- ken masking and multi-token prediction within a probabilistic framework using diserete diffusion and variational inference. We validate the effi- cacy and generalizability of the proposed method on two challenging tasks across different modali- ties: conditional image generation on ImageNet 256x256 and zero-shot text-to- to-eeecc synthesis. Experimental results demonstrate that ResGen outperforms autoregressive counterparts in both tasks, delivering superior performance without compromising sampling speed. Furthermore, as we scale the depth of RVQ, our generative models exhibit enhanced generation fidelity or faster sam- pling speeds compared to similarly sized baseline models.
One promising approach to address these challenges is Residual Vector Quantization (RVQ) (Lee et al., 2022), which improves data reconstruction quality without increas- ing sequence length. RVQ extends Vector Quantized Vari- ational Autoencoders (VQ-VAES VAEs) (Van Den Oord et al., 2017) by iteratively applying vector quantization to the residuals of previous quantizations. This process results in token sequences that are shorter in length but deeper in hierarchy, effectively compressing data while maintaining high reconstruction fidelity. However, despite the advan- tages of RVQ in data compression, generative modeling on RVQ-based token sequences introduces new challenges. The hierarchical depth of these token sequences complicates the modeling process, particularly for autoregressive mod- els whose sampling steps typically scale with the product of sequence length and depth (Lee et al., 2022). Although non-autoreggresivee approaches have been explored along either sequence length or depth (Borsos et al., 2023; Copet et al., 2024; Kim et al., 2024), existing methods do not ef- fectively eliminate the sampling complexity associated with both dimensions simultaneously.
1. Introduction
Recent advancements in deep generative models have shown significant success in high-quality, realistic data genera- tion across multiple domains, including language model- ing (Achiam et al., 2023; Touvron et al., 2023; Reid al., 2024), image generation (Rombach et al., 2022; Saharia et al., 2022; Betker et al., 2023), and audio synthesis (Wang
In this paper, we present ResGen, efficient baseb generative modeling designed to achieve high-fidelity sam- ple quality without compromising sampling speed. Our key idea lies in the direct prediction of vector embeddings of col- lective tokens rather than predicting each token individually. By predicting cumulative embeddings, the model captures correlations among consecutive tokens across depths. This approach allows to decouple inference steps from both
'Equal contribution. *Work done KRAFTON. 'NVIDIA *KRAFTON, Correspondence to: Jaewoong Cho <jw- tho@kraftooncom..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Efficient Generative Modeling with Residual Vector Quantization-Base Tokens
sequence length and depth, resulting in a model that gen- erates high-fidelity samples efficiently. Additionally, we extend our approach involving token masking strategy and multi-token prediction mechanism within principled probabilistic framework using a discrete diffusion process and variational inference.
where 0 denotes the model parameters and the summation over includes only those positions where the token is masked, denoted by my = 0. The masking process in- volves selecting a number of tokens to mask, determined by a masking schedule n = [n(r) I.]. Here, r indicates the current time step in the unmasking process, normalized to range from zero to one, and 7(.) is a pre-defined masking scheduling function that monotonically decreases from one to zero as increases. During training, r is sampled from a uniform distribution.
We validate the efficacy and generalizability of ResGen across two real-world generative tasks: conditional image generation on ImageNet 256x256 and zero-shot text-to- speech synthesis. Experimental results demonstrate superior performance over autoregressive counterparts in these tasks. Furthermore, as we scale the depth of RVQ, ResGen ex- hibits enhanced sampling quality or faster speeds compared to similar-sized baseline generative models. We also analyze model characteristics under varying hyperparameters, such as sampling steps, and examine their impact on generation quality in our ablation study.
In the decoding phase, the model employs an iterative pre- diction process to progressively fill in the masked sequence. At each iteration, the masking ratio r is updated to linearly increase from zero to one. Starting with an entirely masked token sequence, the model predicts the masked tokens, and subset of these predicted tokens is selected to be unmasked based on confidence scores calculated through prediction probabilities. The number of tokens to unmask at each iteration is determined by the masking schedule.
The rest of the paper is organized as follows. In Section 2, we provide the background for our study to establish the foundational understanding necessary for the subsequent discussion of our method. In Section 3, we introduce the ResGen framework, detailing the formulation of masked token prediction as discrete diffusion process and the de- coupling of generation iteration from token sequence length and depth. We also compare our approach with previous methods, highlighting the advantages of our strategy. In Sec- tion& 4, we provide the context for our study with relevant prior work. In Section 5, we present experimental results that validate the performance of ResGen, along with an abla- tion study on model performance with different RVQ depths and sampling steps. Finally, in Section 6 and Appendix D, we summarize the key findings of the paper, discuss limita- tions and directions for future research, respectively.
Residual Vector Quantization. Residual Vector Quan- tization (RVQ) (Lee al., 2022) has been employed as more precise alternative to VQ for quantizing latent vectors in VAE. While previous VQ-VAEs quantize an input by replacing each encoded vector with the nearest embedding from a codebook, RVQ iteratively applies vector quantiza- tion to the residuals of previous quantizations.
Formally, let the output of the encoder in a VQ-VAE the position i be h, The residual vector quantizer maps it to sequence of quantized tokens x E NLXD, where D is the total depth of the RVQ process:
2. Background
Fij arg min [1haj-1-e(us e( vE(l....,V)
Masked Token Modeling. Masked token modeling, in- troduced in prior work (Chang et al., 2022), is generative framework that operates on token sequences derived from the quantized encoder outputs of a Vector Quantized Varia- tional AutoEncoder (VQ- VAE) ( (Van Den Oord et al., 2017). The core idea involves randomly masking a subset of in- put tokens and training the model to predict these masked tokens using a cross-entropy loss.
hag = haj- (eug-1-((r.jt))) e(ma);)) for allll j E [1, D).
where e(v; j) is the v-th vector embedding from the code- book at depth j. and V is the number of embeddings per depth. Here, Iı, represents the selected embedding index for the i-th token at depth j. and his denotes the residual vector after the j-th quantization step.
Let (Z1,...., )L) be a token sequence in N and m (m1..... m.) m,) be a corresponding binary mask, where m; E (0, 13. By definition m, = 0 indicates that token x; is masked. We construct a masked token sequence æ @ m by element-wise multiplying x and m. The training objective then formulated as:
The final reconstructed vector is obtained by summing the embeddings across all depths, z; = > e(aa,ji)) This iterative quantization process enables RVQ to produce quantized output that closely approximates the original en- coder output by increasing the depth 1 of quantization steps. As a result, RVQ effectively captures the most significant features in the lower quantization layers, while finer details are progressively captured in higher layers.
Liank Cmr, m. 0)
Σ
log po (x. m),
ve(i m.=
2

---

Efficient Generative Modeling with Residual Vector Quantization-Ban Based Tokens
Forward masking process
18)
= 15)
(n 10)
(n
Depti
Ad
Length
Reverse unmasking process
Cumulative embedding vectors
Token Masking
LVO
ResGen
Figure 1. An overview of the forward masking and reverse unmasking processes shown at the top, with detailed depiction of the reverse unmasking process below. In the top figure. forward masking proceeds from right to left, incrementally masking more tokens, while reverse unmasking progresses from left right, iteratively revealing the masked tokens. White boxes denote masked tokens and colored boxes represent tokens that have been uncovered. The bottom figure illustrates the reverse unmasking process in detail. Starting from masked residual vector quantization (RVQ) tokens, our method first predicts cumulative RVQ embeddings. These embeddings are then quantized and partially masked again. Through series of iterations, each round predicts the values of the masked tokens and replaces them until the entire token sequence is filled.
3. Method
Given token sequence from RVQ, x E NLXD, with se- quence length L and depth D, we apply a binary mask m E {0, 11 where each mij indicates whether the token Fi,y is masked (m4,5 0) or not (mz.j 1). The total number of tokens mask is determined by masking [r(r) L D1. Here, indicates the current time step in the unmasking process, normalized to range from zero to one, and 7(-) is pre-defined masking schedul- ing function that monotonically decreases from one to zero as r increases. During training, r is sampled from a uniform distribution.
In this section, we introduce our method, ResGen, which iteratively fills tokens in a coarse-to-fine fine manner to achieve efficient and high-fidelity generative modeling with Resid- ual Vector Quantization (RVQ). We structure our discussion into three main parts:
schedule,
We present token masking strategy tailored for RVQ tokens and describe how we model masked token pre- diction by predicting sum of residual vector embed- dings to decouple the generation iterations from the length and depth token sequences.
To distribute the masked tokens across the L positions, the number of tokens to mask at each position i, denoted by A; is sampled without replacement from a multinomial distribution with equal probability across all positions, en- suring that Σ l = n. At each position é, k tokens are masked starting from the highest depth j = D and moving towards lower depths. This ensures that finer de- tails captured at higher depths are masked before coarser information at lower depths, illustrated in Figure 1.
We show that our proposed token masking and multi- token prediction method can be formulated within probabilistic framework using discrete diffusion pro- cess and variational inference.
We detail the training and sampling techniques of Res- Gen, focusing on the implementation of the mixture of Gaussians for latent embedding estimation and en- hanced sampling strategies based on model confidence scores.
Multi- Tokti-Token Prediction Masked Tokens. We describe the training and decoding phases of our multi-token predic- tion strategy, which efficiently predicts masked tokens by focusing on predicting the aggregated vector embeddings of collective tokens rather than the individual tokens
3.1. Masking and Prediction Task Design for RVQ Tokens
Token Masking for RVQ Tokens. Our masking strategy progressively masks tokens starting from the highest quanti- zation layers, capitalizing on the hicrarchical nature of RVQ where tokens at greater depths capture finer details.
Training: Given the input sequence x and the correspond- ing mask m, the model predicts the sum of masked embed-
dings z such that 2, = Σ, e(x..j; @ (1 - -m;,) rather
3

---

Efficient Generative Modeling with Residual Vector Quantization-Bast Based Tokens
than the target tokens directly, where e(u;j) denotes the u-th vector embedding from the RVQ codebook depth j. The training objective is to maximize the log-likelihood of the sum of masked embeddings:
number of masked tokens at each step.
The masking at each step t is governed by discrete ran- dom process that determines how many tokens to mask. Let wi(+)) denote the number of tokens to be newly masked at the next step t + 1 for the i-th position in a token se- quence. The total number of tokens to be masked at step + is (2(+1) = Σ/=1 k((+1), where and D are the length and the depth of the token sequence. The proba- bilistic mechanism is that at each step i, we sample the vector ((++1) = (4((+1) ;... ky a((+)) from a multivariate hy- pergeometric distribution, which corresponds to drawing ((++)) tokens without replacement from the pool of cur- rently unmasked tokens. Formally, if at step t there remain LD - Σi= n((r) unmasked tokens in ax(!), then drawing (t+11) tokens to mask can be modeled as:
Lmask (a,, m; m; --
Σ
log (2 @ m))
REtT) 23 m
(1)
where represents the model parameters and the summation over includes only those positions where at least one token masked, denoted by Σ, m < D. To model the distri- bution ps. employ mixture of Gaussian distributions. We modify the training objective to encourage the mixture component usage of the mixture of Gaussian distributions, which is described in Appendix A.2.
This method avoids imposing conditional independence of tokens along the depth, which could harm model perfor- mance. Instead, it relies on the key idea that accurately predicting the vector embedding z; is more critical than predicting the individual tokens F; as the decoder of VQ-VAE operates on vector embeddings.
L
11
(k((++ (1)
1=1
L.D )
Once we have sampled (((++1,, we construct a (1+1) from (x()) by masking out the newly selected tokens. Specifically, let e denote the masked token. Then, resulting masked tokens at each sequence position i are defined as:
the decoding phase, the model employs an iterative prediction process to progressively fill in the masked sequence. At each iteration, the masking ratio is updated to linearly increase from zero to one. Starting with an entirely masked token sequence, the model progressively fills in the sequence in a coarse-to-fine manner. At each step, the model predicts the cumulative masked token embedding z;. These predicted vectors are then quantized into tokens via RVQ quantization. A subset of these predicted tokens is randomly selected to be unmasked, where the number of tokens to unmask each step is determined by the masking schedule. Although the quantization step at each sampling iteration involves sequential operations to reconstruct tokens from embeddings, it adds negligible overhead compared to the model forward pass.
Sampling:
(t) A
i D D- K(T))
(1+1)
otherwise
An attractive property of this forward diffusion process is that we can write closed-form expressions for both the marginal distributions and conditional distributions at in- termediate steps. Since the forward process is defined by in- cremental masking without replacement, we can directly in- tegrate over all intermediate steps. This yields the marginal distribution of (t) given (");
H
(xx(1) (1)
x
(0))
We summarize the training and sampling algorithms for ResGen in Algorithm 1 and 2 of Appendix.
p(())))))))
LD
which expresses the probability of having Σt= (T( tokens masked in each segment i, given that a total of Σt= n(()
3.2. Formulation within Probabilistic Framework
tokens have been masked overall up to step t.
We now cast our masked token prediction procedure into probabilistic framework based on a discrete diffusion model and variational inference. This perspective allows us to view our method as a likelihood-based generative process and provides a theoretical foundation for its design.
Similarly, we can write the conditional distribution of (!) given ae(++1 and (0):
E ((7) 11 1=1
A R(1-1)
(()(1) (++1)) (1(+))).000))
- m(r))
Consider the token masking process described in Section 3.1. We can inter- pret this process as the forward diffusion step of discrete diffusion model on token sequences. The idea is to gradu- ally transform a fully unmasked token sequence (0) into a fully masked sequence (T) by progressively increasing the
Forward Discrete Diffusion Process.
ni0+13
reflecting the probability of having arrived at (9) from x (++1)) by considering how many tokens were masked in the last step. In this sense, the forward and backward processes are fully characterized by the combinatorial structure of drawing tokens without replacement.
4

---

Efficient Generative Modeling with Residual Vector Quantization-Basee Tokens
The reverse process aims reconstruct the original tokens from partially masked sequences. Given the model's probability to reconstruct the original tokens pe (x:(0 | xx(++)) the probability to reverse a diffusion step po (a:(0 | (t((+))) is defined as:
Reverse Discrete Diffusion Process.
sequences of tokens (Austin et al., 2021). This typically in- volves iteratively refining corrupted tokens progressively unmasking masked tokens. A prominent strategy within this framework is masked generative modeling, also known as masked diffusion or masked token modeling, where the corruption process specifically involves masking portions of the token sequence, and the model learns to predict the content of these masked positions (Gu et al.. 2022; Chang et al., 2022). Models such VQ-Diffusion (Gu al., 2022) and, conceptually, MaskGIT (Chang et al., 2022) exemplify this masked diffusion approach for generating flat token sequences, leading to improved sampling elfi- ciency over autoregressive models. GIVT (Tschannen et al., 2024) introduces a method that replaces coftmax-based to- ken prediction with ofuuur--of-GGusianss diction in masked token prediction, progressively filling masked positions with predicted vectors. Build upon prin- ciples similar to MaskGIT, MAGVIT-v2 (Yu et al., 2024) and MaskBit (Weber et al.) have demonstrated strong per- formance by incorporating innovations such as improved quantization schemes, notably Lookup-Free Quantization (LFQ) (Yu et al., 2024), which represents each token as a col- lection of bits. MaskBit further advances this by grouping the bits of each token and leveraging the partially unmasked bits to predict the masked portions.
((( (2) (++1), x (0) (x (((+))). (0
Σ
This formulation lets us compute the variational lower bound of the data log-likelihod:
log po po (2) 2 -Bag fr + Σ L + fu).
f
121
where fT= DKL (a(a!!?) 1 (0) p((x(T)))
vector pre-
C. DKL (o(z(() ((+11), (00+)))((00))) (0)) (o( | (x()))))))
and =o= log ())
Here, Lr is the prior loss, which becomes zero since (T) is fully masked, Le are the diffusion losses at each step 6, and Co is the reconstruction loss. By combining the diffusion and the reconstruction losses while placing equal emphasis on predicting the original tokens at each step, we can derive a simplified loss function:
Csimple (x: (0)))) log po (z
x (1)).
Separate from the discrete diffusion approaches defined above, recent works like VAR (Tian et al., 2024), MAR (Li et al., 2024) and HART (Tang et al., 2024) propose alter- native paradigms to token- autoregressive modeling. VAR introduces coarse-to-ine next-scale prediction mech- anism, effectively capturing hierarchical structures in im- ages; for a detailed discussion of the distinctions between VAR's 's hierarchical modeling and ResGen's 's approach, partic- ularly concerning representation, generation, and adaptabil- ity, please see Appendix C. MAR eliminates the reliance on discrete tokens by modeling probabilities in continuous- valued space using a diffusion-based approach, simplifying the pipeline while maintaining strong performance. HART proposes hybrid approach that decomposes the latent space into discrete tokens, modeled autoregressively, and contin- uous residuals, modeled with a diffusion process. This strategy is designed to reduce the number of sampling steps compared to fully continuous methods like MAR.
Latent Modeling with Variational Inference. To ef- ficiently handle dependencies across token depths, we adopt a multi-token prediction approach inspired by CLaM- TTS (Kim et al., 2024). Instead of predicting tokens indi- vidually. we predict their cumulative vector embedding x. This approach aligns naturally with the RVQ dequantization process and decouples the generation time complexity from the token depth.
We use variational inference to derive the upper bound of the negative log- likeliihoood - llgg po (x) x x²))::
po (1)
log p( x (0) |2, ()) - log
g+
g(x x(0), (t)
Assuming p(xx(0) |z, x: (1) corresponds to RVQ quantization and q(= (0))) ( to RVQ dequantization of the masked tokens, we focus on the remaining term:
However, these methods primarily deal with flat token se- quences and do not consider the hierarchical depth inherent in RVQ. RQ- Trannssffrre (Lee et al., 2022) was the first to demonstrate generative modeling on RVQ tokens using an autoregressive model over the product of sequence length and depth, resulting in increased computational complex- ity. Vall-E (Wang et al., 2023) predicts the tokens at the first depth autoregressively and then predicts the remaining tokens at each depth in single forward pass sequentially. SoundStorm (Borsos et al., 2023) generates tokens using
Cmask (0)
(x)) x (0:0) = log pe (z|a lo (0),
which matches the prediction loss in Equation 1.
4. Related Work
In this work, we refer to discrete diffusion models (DDMs) as a class of generative models that learn to reverse a de-
fined corruption process applied discrete data, such as
5

---

Efficient Generative Modeling with Residual Vector Quantization-Bant Baasd Tokens
masked token prediction given semantic tokens, but still has sampling time complexity that increases linearly with the residual quantization depth. NaturalSpeech 2 (Shen et al., 2024) employs diffusion- generative modeling in the RVQ embedding space instead of token generation. CLaM-TTS (Kim et al., 2024) employs vector prediction for multi-token prediction but operates in an autoregressive manner along the sequence length.
Baselines and Training Configurations. In the vision domain, we compare our models with recent generative model families, including (1) autoregressive models: RQ- transformer (Lee et al., 2022), VAR (Tian et al., 2024), MAR (Li et al., 2024); and (2) non-autoregressive models: MaskGIT (Chang et al., 2022), DiT (Peebles & Xie, 2023). For the audio task, we benchmark the proposed model against state-ofthhe-ae TTS models, including (1) autore- gressive models: VALL-E (Wang al., 2023), SPEAR-TTS (Kharitonov et al., 2023), and CLaM-TTS (Kim et al., 2024); and (2) non-auloregressive models: YourTTS (Casanova et al., 2022), VoiceBox (Le et al., 2023), and DiTTo-TTY (Lee et al., 2025). The training configurations for our mod- els are detailed in Appendix A.I.
In contrast these approaches, our method offers more efficient solution for generative modeling with RVQ tokens. We propose a strategy that predicts the vector embedding of masked tokens, decoupling the sampling time complexity from both sequence length and token depth. By predicting cumulative vector embeddings rather than individual tokens, our method efficiently handles the hierarchical structure of tokens, offering enhanced sampling efficiency and high- fidelity generation.
5.2. Experimental Results
5.2.1. VISION TASK: ABLATION STUDIES
To validate the effectiveness of our method as generative model for RVQ tokens, we compare it with two baseline models, MaskGIT and TQ-Transformer, using an identical RVQ representation with depth 16. MaskGIT, originally de- signed to predict randomly masked tokens a single depth, is adapted to multi-depth setting by autoregressively gen- erating tokens depth by depth. conditioning each prediction on the previously generated depths.
5. Experiments
In this section, we demonstrate the effectiveness of our approach in both image generation and text-to-speech syn- thesis, highlighting its generalizability and efficiency.
5.1. Experimental Setting
Experiment Tasks. For the vision domain, we fo- cus on conditional image generation tasks on Ima- geNet (Krizhevsky et al., 2017) at resolution of 256 x 256. In the audio domain, we evaluate our model using two tasks inspired by Voicebox (Le et al., 2023): 1) continuation: given a text and a 3-second segment of ground truth speech, the goal is to generate seamless speech that continues in the same style as the provided segment: 2) cross-sentence given text, 3-second speech segment, and its transcript (which differs from the text), the objective is to generate speech that reads the text in the style of the provided segment.
We therefore investigate three generation strategies for & x 8 16 RVQ tokens: (i) fully autoregressive (RQ- Transformer); (ii) masked sequence + autoregressive depth (our MaskGIT variant); and (iii) fully masked (ResGen). To ensure a fair comparison, we configured each model with comparable parameter counts: ResGen (576 M). RQ- Transformer (626 M), and MaskGIT (580 M), and trained them for 2.8 M steps.
As summarized in the left Table 1, ResGen shows higher generation quality than both baselines while requiring fewer inference steps than Trannasfforrer The relatively lower performance of the depth-wise MaskGIT variant highlights the difficulty of extending single-depth masked models to depth RVQ tokens, reinforcing our design choice of predicting all jointly.
Evaluation Metries. For vision tasks, we employ the Fréchet Inception Distance (FID) (Heusel et al., 2017) for comparing it with other state-of-the-an image generative models. We set the sample size for FID calculation to 50K in all experiments. For audio tasks, we evaluate the models using the following objective metrics: Character Error Rate (CER), Word Error Rate (WER), and Speaker Similarity (SIM), as described in VALL-E (Wang et al., 2023) and CLaM-TTS (Kim et al., 2024). CER and WER measure the intelligibility and robustness. For SIM, we adopt SIM-o and SIM-r metrics from Voicebox (Le et al., 2023). SIM-o evaluates the similarity between the generated speech and the original target speech, while SIM-r assesses the similar- ity between the target speech and its reconstruction, which is obtained by processing the original speech through a pre-trained autoencoder and vocoder.
Beyond these baseline comparisons, further experiments, also detailed in the left Table 1, explore key design choices and the scalability of ResGen. First, we investigate an ar- chitectural variant of ResGen that predicts discrete tokens directly and in parallel across all depths at each step, using the loss function defined in Equation 2. This variant yields competitive results relative to the baselines but outper- formed by the final version of ResGen, highlighting the effectiveness of the cumulative embedding prediction strat- egy. Second, to assess model scalability, we increase the parameters of ResGen from its 576 M version to a 1 B pa-
6

---

Efficient Generative Modeling with Residual Vector Duantization-Basee Tokens
Table 1. Ablation study on ResGen (Ours). The left table compares image generation quality and efficiency among ResGen, RQ- transformer, and MaskGIT, evaluated using the same RVQ tokens. The right table reports efficiency of AR-ResGen using 2. 4, and 8 iterative refinement steps for depth prediction, compared with RQ- Transformer. The boldface indicates the best result. Wall-clock time results reflect the time required to generate single sample on an NVIDIA A100 GPU.
Model
FID (wlo CFG)
FID (w/ CFG)
Steps
Model
FID (wlo CFG)
FID (w/ CFG)
Wallclock Time
RQ-transformer
15.71
5.50
1024
RQ-taassfrrer
15.71
5.50
5.35s
MaskGIT
28.40
9.83
63
AR ResGen step)
27.45
6.47
1.30s
ResGen
8.77
2.43
AR Ree-Reseeen (2 step)
24.10
5.33
1.56s
Architectinal and Scaling Analysis
AR-ResGen (1 step)
23.75
5.30
2.00s
ResGen (Direct Token Predictiinn
12.79
2.91
63
AR ResGen step)
23.48
5.22
3.05s
ResGen (LB Parameters)
8.12
2.26
rameter model. This larger model demonstrates further FID improvements, confirming that ResGen scales effectively with increased capacity.
ory efficiency is evaluated based on the maximum batch size, which refers to the maximum number of latent representa- tions that generative model can process during inference on the same device, and generation speed is measured as the wall-clock time required to generate a single sample. Detailed sampling hyperparameters can be found in Table 4.
The right of Table I summarizes a stricter ablation de- signed to disentangle the efficiency gains introduced by our depth modeling strategy from those brought by the masked- generation paradigm. This ablation features AR-ResGen, variant where the spatial sequence is still generated au- toregressively. identical to er-Transformer. Crucially, AR- ResGen keeps the overall parameter budget comparable (625 M) and differs from the RQ- -Q-arssfformer only by replacing its depth transformer with MI.P to predict our cumulative embedding.
Generation quality. Among models with similar param- eter counts, and specifically excluding larger models such as VAR-d30 and MAR-H, ResGen-rvg1ó demonstrates highly competitive performance, which achieves an FID of 1.93 with classifier-free guidance (CFG). Despite its slightly higher FID score compared to MAR-L,, ResGen achieves comparable quality with much faster sampling speed, demonstrating its efficiency in balancing quality and resource usage. These results highlight the competitive generation capabilities of ResGen, which closely rivals the MAR series.
The data in the table shows that AR- AR-ResGen attains FID of 5.22 with 8 refinement iterations while reducing the de- coding time from 5.35 to 3.05 s (x 1.8 faster). Even with just two iterations it reaches an FID of 5.33s in 1.56s, yield- ing a x3.4 speed-up over RQ- Trannssforrrrr at comparable quality. These results isolate and confirm that our depth prediction strategy alone, independent of the sequence gen- eration framework, delivers substantial improvements in both the generation speed and the sample quality. These results demonstrate that ResGen sets new benchmark for both efficiency and quality in generative modeling.
Speed efficiency. As shown in Figure 2, ResGen-rvq16 ranks second only to VAR, making it practical choice for scenarios where speed is critical. Unlike certain mod- els that compromise speed for quality, our method main- tains a favorable balance, offering both rapid generation and competitive quality. Compared to the MAR series, which achieves marginally better FID scores, ResGen's 's superior speed positions it as an efficient solution for real-time or tightthroughput applications.
Finally, Appendix B.2 examines the effect of sampling hy- perparameters, specifically the number of steps and tem- perature scaling, generation quality, and Appendix C.3 investigates the sensitivity of ResGen's performance to al- ternative masking strategies.
Memory efficiency. ResGen consistently exhibits supe- rior efficiency, achieving the largest latent batch size. Specif- ically. ResGen-rvql6 supports a maximum latent batch size of 1,915, surpassing MAR-B's capacity of 1,738. This ef- ficiency enables large-scale generative tasks and mitigates computational bottlenecks in production pipelines.
5.2.2. VISION TASK: BENCHMARK COMPARISONS
Next, we assess the effectiveness of our generative modeling by comparing it to existing vision generative model families. We train two variants of ResGen, termed ResGen-rvqs and ResGen-rvq16, using 8-depth and 16-depth RVQ tokens, respectively, for up to 7 million steps. The results, shown in Table 2, compare these models across three key aspects: generation quality, memory efficiency, and generation speed. Generation quality is assessed using the FID metric, mem-
Comparison with recent masked generative models. To provide broader context for ResGen's performance, we also consider recent -tate-of-the-art masked generative mod- els like MAGVIT-v2 (Yu et al., 2024) and MaskBit (Weber et al.). As detailed in Table 8, these models achieve strong FID scores (MAGVIT-v2: 1.78, MaskBit: 1.62), which
7

---

Efficient Generative Modeling with Residual Vector Quantization-Bast Based Tokens
Table Comparison of generative models on class-conditional ImageNet resolution of 256x256.. Boldface denotes the hest result and underline indicates the second best; asterisk (*) marks scores reported in their original papers. For RQ- To-arssfforerrs FID scores presented 'FID (w/ CFG) column are based on results obtained with rejection sampling. 'Code length' refers the sequence length of the latent representations.
Model
Code length
Params
FID (w/o CFG) +
FID (w/ CFG) + Maximum batch size
MaskGIT
256
277M
6.18*
DiT-XL/2
256
675M
9.62*
2.27*
1159
VAR-d16
256
310M
12.18
3.30
247
VAR-d20
256
600M
8.60
2.57*
148
VAR-d24
256
1.0B
6.43
2.09*
102
VAR-d30
256
2.0B
5.31
1.92*
60
MAR-B
256
208M
3,48*
2.31
1738
MAR-L
256
479M
2.60*
1,78
1167
MAR-H
256
943M
2.35
1.55*
812
NQ-Trassformer
64
1.4B
8.71*
3,89*
1151
TQ-rransformer
64
3.8B
7.55
3,80
390
ResGen-rvq8
64
576M
6,56
2.71
1995
ResGen-rvq16
64
576M
6.04
1.93
1915
DIT
2000
MAR
Maximum batch size
RQ Tran
1500
ResGen-rvge
Res Gen
1250
MaskGIT
3
750
500
250
10--
10a
10a
10a
DIT.UR2
VAR-I16
VAR-120
WAR-224
UAR-1230
MAR
MAR-L
MARA
ROTran- 48 Ro RoTran-3 as
ResGen-vvzz ResGen-rvop
Wallclock Time (sec)
Figure 2. The left figure shows the trade-off between sampling speed and generation quality across various generative models. For ResGen, dotted lines indicate performance models, solid lines connect results corresponding to variations parameter size. Note that in ResGen-rvg8 and ResGen-rvg the number specifies the depth of RVQ. Both wall-clock time maximum batch size are measured on an NVIDIA A.00 GPU.
different sampling steps, highlighting step-dependent performance improvements. For other
are currently lower than that of our ResGen-rvq16 (1.93 with CFG). However, ResGen offers distinct architectural advantages, including effective handling of hierarchi- cal token dependencies and flexible resolution-depth seal- ing, which enable shorter code lengths. These properties contribute to its strong overall performance and adaptabil- ity across diverse settings, as discussed detail in Ap- pendix C.4.
that greater RVQ depth enhances both reconstruction and generation quality. These results demonstrate the scalability of our approach, showing that deeper RVQ quantization improves performance while maintaining efficiency.
For qualitative evaluation, Figure 5 presents diverse samples generated by the baseline models (VAR, MAR, and DiT) alongside those from ResGen. Additional randomly gener- ated samples from our model are shown in Figure 6. Finally, in Appendix C.2, we present detailed comparison between ResGen and the faster MAR variant.
Interestingly, ResGen-rvgl6, with deeper RVQ quantization, achieves better sample quality than ResGen-rvg8 the same number of sampling steps, with minimal increase in inference time. This aligns with Appendix A.4, which shows
8

---

Efficient Generative Modeling with Residual Vector Quantization-Bant Bassdd Tokens
Table Performances for the continuation task (top tahle) and the cross-sentence task (bottom table). The boldface indicates the hest result, the underline denotes the second best, and the asterisk denotes the score reported in the baseline paper.
Model
Params
WER +
CER 4
SIM-o 1
SIM-r t
Inference Steps +
Ground Truth
n/a
2.2*
0.61
0.754*
0.754*
n/a
YourTTS
7.57
3.06
0.3928
1
Vall-E
302M
3.8*
0.452*
0.508
Voicebox
364M
2.0*
0.593
0.616
64
CLaM-TTS
584M
2.36*
0.79
0.4767*
0.5128*
DITTo-en-L
508M
1.85
0.50
0.5596
0.5913
25
DiTTo-en-XL
740M
1.78*
0.48*
0.5773*
0.6075*
25
Melvae-ResGor
457M
1.94
0.53
0.5421
0.5701
25
Rvgvae-ResGie ResGen
625M
1.86
0.50
0.5853
0.5886
25
Model
WER L
CER +
SIM-o f
SIM-r 1
YouTTTS
7.92 (7.7%)
3.18
0.3755 (0.337*)
Vall-E
5.9
0.580*
SPEAR-TTS
1.92*
0.560*
Voicebox
1.9
0.662
0.681
CLaM-TTS
5.11*
2.87*
0,4951 *
0,5382
DiTTo-en-L
2.69
0.91
0.6050
0.6355
DiTTo-en-XL.
2.56*
0.89*
0.627*
0.6554
Melvae-ResGen
1.75
0.48
0.5597
0.6061
Rvgvae-ResGen
1.70
0.46
0.6037
0.6307
5.2.3. AUDIO TASK
maintain computational efficiency while delivering high- quality outputs. For qualitative comparison, present generated audio samples in the project page!.
In our Text-to-Speech (TTS) experiments, we compare our method to autoregressive models that generate RVQ tokens, using the same MelVAE module from CLaM-TTS (Kim et al., 2024). As shown in Table 3, our model achieves lower word and character error rates (WER and CER) as well as higher speaker similarity scores (SIM-o and SIM-r) than the baseline and requires fewer inference steps, demonstrating its efficiency in token generation. Notably, our method uses only 25 iterations, which is fewer than the RVQ depth of 32.
6. Conclusion
In this work, we propose ResGen, an efficient RVQ-based discrete diffusion model that generates high-fidelity sam- ples while maintaining fast sampling speeds. By directly predicting the vector embedding of collective tokens, method mitigates the trade-offs between RVQ depth and inference speed in RVQ-based generative models. We fur- ther demonstrate the effectiveness of token masking and multi-token prediction within a probabilistic framework, employing a discrete diffusion process and variational infer- ence. Experimental results on conditional image generation and zero-shot text-to-speech synthesis validate the strong performance of ResGen, which performs comparably to or exceeds baseline models in terms of fidelity and sampling speed. As we scale RVQ depth, our model exhibits im- provements in generation fidelity or efficiency, showing its scalability and generalizability across different modalities.
We further evaluate our model with deeper RVQ quan- tization, up to 72 levels, referred to as Segvac-ResGen and compare its performance against recent TTS models. While our results do not surpass otate-of-the-art methods such as Voicebox and DiTTo-TTS on every metric, the pro- posed approach demonstrates substantial advantages in com- putational efficiency and accuracy. Specifically, ResGen achieves the lowest WER and CER in the cross-sentence task, outperforming all baselines. In the continuation task, ResGen attains competitive WER and CER scores, ranking second only to DiT.o-cn-XL. Furthermore, the use of deeper RVQ depth enhances leads to improved reconstruc- tion quality and enhanced generation performance. Despite the increased depth, ResGen effectively models these tokens with only 25 inference steps, demonstrating its ability to
'https:// witttauuub...................b.. io/

---

Efficient Generative Modeling with Residual Vector Quantization-Bast BBssd Tokens
Acknowledgement
Zada, S., Ephrat, A., Hur, J., Liu, G., Raj, A., et al. Lu- miere: A space-time diffusion model for video generation. In SIGGRAPHI Asia 2024 Conference Papers, pp. 1-11, 2024.
The authors would like to express our gratitude to Kang- wook Lee for the valuable discussions. We also extend our thanks to Beomsoo Kim, Seungjun Chung, Dong Won Kim, and Gibum Seo for their essential support in data han- dling and verification. This research (paper) used datasets from 'The Open AI Dataset Project (AI-Hub, S. Korea)". All data information can be accessed through 'Al-Hub (www.ainub.or.kr".
Betker, J., Goh, G., Jing, L.,, Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., et al. Improving image generation with better captions. Computer Sci- ence. https://cdn. openai. com/papers/dall-e3., pdf. 2(3y: 8, 2023.
Borsos, Z., Sharifi, M., Vincent, D., Kharitonov, E., Zeghi- dour, N., and Tagliasacchi, M. Soundstorm: Efficient par- allel audio generation. arXiv preprint 2xXix23305006636 2023.
Impact Statement
Our work advances the field of generative modeling by intro- ducing memory -eforry-fffcieet approach for high-fidelity sample generation using Residual Vector Quantization (RVQ). The proposed model, ResGen, demonstrates significant improve- ments in generation quality and efficiency across challeng- ing modalities such as image generation and text-to-speech synthesis. As data resolution and size continue increase in real-world applications,
Casanova, E., Weber, J., Shulby, C. D., Junior, A. C., Gölge, E., and Ponti, M. A. Yourtts: Towards zero-shot multi- speaker tts and zero-shot voice conversion for every- one. In International Conference on Machine Learning (ICML), pp. 2709-2720. PMLR, 2022.
method provides a scalable and efficient solution, reducing the memory and computational burden typically associated with high-resolution generative tasks.
Chang, H., Zhang, H., Jiang, L.,, Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325, 2022.
The potential societal benefits of this research are substan- tial, particularly in areas where efficient and high-quality generation is critical, such as accessibility technologies, creative industries, and scientific simulations. For exam- ple, ResGen can facilitate resource-efficient generation of synthetic media, enabling small-scale researchers and or- ganizations to access and utilize state-of-th state-of-the-ar oh-the-rrt generative technologies that were previously out of reach.
Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and Défossez, A. Simple and controllable music generation. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Ding, Z., Zhang, M., Wu, J., and Tu, Z. Patched denoising diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Repre- sentations, 2023.
However, with all generative models, there are poten- tial misuse risks, such the creation of synthetic media for deceptive purposes or the unauthorized replication of copyrighted content. To mitigate these risks, we encour- age researchers and practitioners to adopt responsible use policies, including mechanisms to detect and authenticate synthetic content, and to promote transparency model development and deployment.
Evans, Z., Carr, C., Taylor, J., Hawley, S. H., and Pons, J. Fast timing-conditioned latent audio diffusion. In Forty-first International Conference Machine Learn- ing, 2024.
Gu, S.. Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., and Guo, B. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10696-10706, 2022.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774: 2023.
He, Y., Yang, S., Chen, H., Cun, X., Xia, M., Zhang, Y., Wang, X., He, R., Chen, Q., and Shan, Y. Scalecrafter: Tuning-free higher-resolution visual generation with dif- fusion models. In The Twelfth International Conference on Learning Representations, 2023.
Austin, J., Johnson, D. D., Ho. J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 14:11991117993 2021.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in neural information processing systems, 30, 2017.
Bar- Tal, O.,, Chefer, H., Tov, O., Herrmann, C., Paiss, R.,
10

---

Efficient Generative Modeling with Residual Vector Quantization-Bale Based Tokens
Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., and Park, T. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124- 10134, 2023.
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11980 11986, 2022.
Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: Vq-vae made simple. In The Twelfth International Conference on Learning Represen- lations, 2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint ar 20Xx::000100861 2020.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Kharitonov, E., Vincent, D., Borsos, Z.,, Marinier, R., Girgin, S., Pietquin, O., Sharifi, M., Tagliasacchi, M., and Zeghi- dour, N. Speak, Read and Prompt: High-Fidelity Text-to- Speech with Minimal Supervision. Transactions of the As- sociation for Computational Linguistics, 11:1 11:1703-1718, 12 2023. ISSN 2307-387X. doi: 10.1 10.1112/tacl.a.. a 000611
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05534 2024.
Kim, J., Kim, S., Kong, J., and Yoon, Glow-tts: gen- erative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Sys- tems, 33:8067-8077, 2020.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and patter recognition, pp. 10684-10695, 2022.
Kim, J., Lee, K., Chung, S., and Cho, J. Clam-tts: Improving neural codec language model for zero-shot text-to-speech. In The Twelfth International Conference on Learning Representations, 2024.
Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., Quitry, F. d. C., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., et al. Audiopalm: A large language model that can speak and listen. arXiv preprint SKIv:2306.12922 2023.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Le. M. Vyas, A., Shi, B., Karrer, B., Sari, L., Moritz, R., Williamson, M., Manohar, V., Adi, Y., Mahadeokar, J., and Hsu, W.N. Voicebox: Text-guided multilingual universal speech generation at scale. In Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 14005-14034. Curran Associates, Inc., 2023.
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K.,, Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35: 36479-36494, 2022.
Lee, D., Kim, C., Kim, S., Cho, M., and Han, W.S. Autore- gressive image generation using residual quantization. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp.
Shen, K., Ju, Z., Tan, X., Liu, E., Leng, Y., He, L., Qin, T., Bian, J., et al. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. In The Twelfth International Conference on Learning Representations, 2024.
2022. Lee, K., Kim, D. W., Kim, J., Chung, S., and Cho, J. DiTTo-TTS: Diffusion transformers for scalable text-to- speech without domain-specifi factors. In The Thirteenth International Conference on Learning Representations, 2025. URL https: / pptp//veeeeeeeeee///eeeeeeoeeeee n forum? id=h@vx9MBowC.
Siuzdak, H. Vocos: and fourier-based neural vocoders for high-quality au- dio synthesis. In The Twelfth International Confer- ence Learning Representutions, 2024. URL https: / //openreview.co net/ for 2tt.ummiidvV99nn2mm0m
Closing the gap between
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37:56424- 56445, 2024.
Tang, H., Wu, Y., Yang, S., Xie, E., Chen, J., Chen, J., Zhang, Z., Cai, H., Lu, Y., and Han, S. Hart: Efficient visual generation with hybrid autoregressive transformer. arXiv preprint arXiv:2410 10812, 2024.

---

Efficient Generative Modeling with Residual Vector Quantization-Base Tokens
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang, L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information 2024.
processing systems,
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint 1.M.i::2302..3377... 2023.
Tschannen, M., Eastwood, C., and Mentzer, F. Givt: Gen- erative infinite-vocabulary transformers. In European Conference Computer Vision, pp. 292-309, 2024.
Van Den Oord, A., Vinyals, O., et Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXhv:2301.02111, 2023.
Weber, M., Yu, L.., Yu, Q., Deng, X., Shen, X., Cremers, D., and Chen, L.-C. Maskbit: Embedding- image gener- ation via bit tokens. Transactions on Machine Learning Research.
Xue, L.., Barua, A., Constant, N., A1-Rfou, R., Narang, S., Kale, M., Roberts, A., and Raffel, C. Byt5; Towards token-free future with pre-trained byte-to-byte mod- els. Transactions of the Association for Computational Linguistics, 10:291-306, 2022.
Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Haupt- mann, A. G. et al. Language model beats diffusion- tokenizer key to visual generation. In ICLR, 2024.
12

---

Efficient Generative Modeling with Residual Vector Quantization-Ban Based Tokens
A. Training details
A.I. Configurations for Training
Vision models We train our method using an architecture similar to DiT (Peebles & Xie, 2023), adopting the XLarge version while modifying the adaptive layer normalization layers for conditioning by replacing their linear layers with bias parameters. As shown in Table I, all generative models are trained for 2.8M iterations under the same RVQ token setting. The model sizes To maintain fair and consistent evaluation conditions across models, RQ-Transformer is assessed in this experiment using classifier-fre free guidance (CFG), rather than the rejection sampling adopted in its original publication.
follows: ResGen (576M), AR-ResGen (625M), RQ-Transformer (626M), and MaskGIT (580M).
In Table 2, all variants of ResGen are trained with a batch size of 256 across 4 GPUs for 7M iterations. The masking scheduling function 7(-) is defined as 7(r) = (1 11-re =²)² and applied throughout all training.
To increase the depth of RVQ, we warm-start from the 4-depth RQ-VAE checkpoint (Lee et al., 2022), excluding the attention layers, and reduce the latent dimension from 256 to 64. Each VAE is further trained for an additional 1M steps, prior work. For the RVQ quantizer, we
both with and without adversarial training, following the same configuration employ the probabilistic RVQ method from Kim et al. (2024), which updates the RVQ codebook embeddings proportionally based on how closely the VAE latents match each embedding. The codebook size at each depth is set to 1024. The resulting token embeddings obtained from the RVQ quantizer are then fed into ResGen, where they are projected to match the hidden size via a linear layer.
Audio models For the Texe-to-Speech task, our model, based on the DiT XLarge architecture as in the vision task, is trained using the same configuration as in prior work (Kim et al., 2024), utilizing 4 GPUs for 310M iterations. The masking scheduling function 7()) is defined as >(r) = (1 - =)) and applied throughout all training. We employ 4 transformer layers to train a linear regression duration predictor for the text inputs, built on top of the pretrained text encoder, ByT5-Large (Xue et al., 2022). The duration predictor is trained minimize the L.2 loss between the mel-spectrogram and the expanded hidden representation of text from ByT5, with alignment achieved using the monotonic alignment search algorithm (Kim et al., 2020). The expanded text hidden representation is downsampled through a strided convolution layer to match the length of the RVQ token sequence and combined with the embeddings of RVQ tokens obtained from the RVQ quantizer. These combined representations are then projected to match the model's hidden size using a linear layer.
To increase the depth of RVQ from 32 in MelVAE 72, we train a separate autoencoder that directly processes waveforms. The 44 kHz waveforms are transformed using the Short-Time Fourier Transform (STFT) with a hop length of and a window size of 32. The real and imaginary parts are concatenated and then encoded through encoder composed of three blocks, each consisting of three ID CenvNeXt layers (Siuzdak, 2024; Liu et al., 2022) followed by a strided convolution layer with a stride of 8 and a kernel size of 8. This results in final latent representation of 512 dimensions with temporal resolution of 10.7 Hz, matching that of MeIVAF. The decoder mirrors the encoder symmetrically, reconstructing STFT parameters, which are then converted back to waveforms via inverse STFT. The autoencoder is trained with similar training configurations as DAC, including mel-spectrogram reconstruction loss and adversarial losses. The RVQ codebook size at each depth is set to 1024, and the entire autoencoder model consists of 185 million parameters.
A.2. Implementation Techniques for Training
Mixture of Gaussians Implementation. Our model utilizes a mixture of Gaussian distributions to represent the distribution over latent embeddings. Specifically, for each token position i, the model outputs the mixture probabilities x; = {-]"))___ 1==1,
the mean vectors for each mixture component {n! K 1= and additional scale and shift parameters for affine transformations a, E R and b, R where K is the number of mixture components and H is the embedding dimension. When projecting the hidden output (size O) to K mixture probabilities, it leads to a projection complexity dominated by 0(0 K H).
For our vision model (O = 1152, K= 1024, H = 64), this cost is comparable to standard softmax layer with "64K vocabulary (V = K * H). which is practical. For higher-dimensions. embeddings like in audio (H = 512), we use low-rank projection for the means, reducing the dominant computational term to O(0 x K * h + H A), again making the effective cost similar to a "64K vocabulary model (h- = 64). This technique, previously used in CLaM-TTS et al., 2024), significantly mitigates overhead.
13

---

Efficient Generative Modeling with Residual Vector Quantization-Banl Based Tokens
Training Objective Modification. From Equation 1, the log-likelihood of the target embedding 2, is formulated log (2 O m) = - -log a; + log En ai (2) (2 (*) I), where 2; = (z; /.)/ag. To further encourage the usage of every mixture component, we modify the objective by decomposing into a sum classification and regression losses. Similar to prior work (Kim al., 2024), applying Jensen's 's inequality, we have:
log 0; log Σ T N
< - log 04
Σ
g(v (b) log log I)+ + DRIla(u Hi, || mi),
regression loss
classilication loss
where g(v) E. M:) is an auxiliary distribution defined as g(v ži M:) x ((z;p (I). This choice of ensures that mixture components with mean vectors closer Σ; have higher probabilities, while all components retain non-zero probabilities. Consequently, every mixture component contributes the training process, promoting higher component usage and diversity in the model's predictions.
Low-rank Projection. Increasing the number of mixture components K leads to substantial growth in the output dimensionality of the model, as it scales with K x H. To accommodate high number of mixtures without incurring excessive computational costs, we adopt a low-rank projection approach following the methodology of the prior work (Kim et al., 2024).
In this approach, the model outputs low-rank mean vectors (p(0) ) which are then transformed using trainable parameters ( and s g( (r) M x( s(r), This decomposition allows for efficient computation of the squared distance
|| 112 by expanding it follows:
|| 14ll || (MA, + +)1?
1 2 + 1i (M M) + TT 2(M 2T) - 22T
27 M
(3)
where we omit / for simplicity. This low-rank projection enables the model to handle large number of mixture components without significant overhead, therehy enhancing both the scalability and performance of the generative process.
14

---

Efficient Generative Modeling with Residual Vector Quantization-Base Tokens
A.3. Pseudo-code for Training
Algorithm Training
1: procedure BinaryMask(n, L, D)
2: Sample kı: without replacement with total draws n.
3: for 1 to L do
4: m(,1:(D-k) f-1
5: mL(D=kc+1): <- 0
6: end for
7: return m
8: end procedure
9:
10: repeat
11: Pdata
12: r ~ Uniform(0.1
n - D]
14:
+ BinaryMask(n, L, D)
15:
z Ej (e( @ (1 m)
Take gradient descent step on:
17:
VoLmusk (r., m;; 0)
18:
until converged
A.4. Scalability of our generative modeling with RVQ depth
3.0
1.25
2.5
1.00
0.75
2.0
0.50
rvq8
rvq16
Figure 3. Effect of RVQ depth both the autoencoder's reconstruction quality and method's generation quality. We compare configurations, rvg8 and rvql6, corresponding to RVQ depths of 8 and 16, respectively. As the RVQ depth increases, the autoencoder achieves better reconstruction quality (lower rFID). Despite the increased number of tokens deeper depth, generative models show better generation quality (lower FID), underscoring its scalability with deeper quantization.
B. Sampling details
B.1. Sampling with Confidence Scores
Inspired by confidence-based sampling with a choice temperature, as proposed MaskGIT (Chang et al., 2022) and GIVT (Tschannen al., 2024), we unmask tokens based on the log probabilities computed for all masked tokens. These log probabilities are derived from the squared distance between token embeddings and the sampled latent z; each position i. The log probability log p(#i,) =;) is calculated as log p(z..)) 2,) x log (z Ei= e(x;,a; d); of) for all masked positions: and j. Here, o, denotes the standard deviation of latents at RVQ depth j pre-calculated during RVQ training. The log probabilities are cumulatively summed across depths, and the confidence score is obtained by adding
15

---

Efficient Generative Modeling with Residual Vector Quantization-Bast Tokens
Gumbel noise, scaled by the choice temperature, them. The choice temperature remains fixed throughout all inference steps in our settings. Tokens with higher confidence scores prioritized for unmasking and filled earlier in the iterative generation process. In particular, Table 4 lists the exact hyper-parameters used during sampling.
Table Sampling hyper-parme syper-parameters the experiments Figure 2. The efg schedule indicates linear scaling classifier-fres guidance weight from the start value the first step to the end value at the last step. Top-p is the nucleus-sampa threshold and is the temperature.
Variant
Steps
cfg schedule
Top-p
ResGen-rvgl6
28
0.02 + 2.4
0.94
28.0
48
0.02 + 2.4
0.96
28.0
64
0.02 + 2.2
0.98
28.0
ResGen-rvq8
28
0.02 + 2.4
0.94
28.0
48
0.02 - 2.4
0.96
28.0
64
0.02 - 2.2
0.98
28.0
B.2. Ablation Studies on Sampling
We conducted ablation experiments to analyze the characteristics of our sampling algorithm, focusing on hyperparameters such as sampling steps, top-p values, and temperature scale, and their impact on generation quality. As illustrated in Figure 4a, increasing the number of sampling steps improves generation quality in both scenarios: with classifier-fres guidance (CFG) and without This demonstrates that additional steps enable the model refine its outputs more effectively, resulting in higher-quality generations.
Figure 4b explores the impact of top-p values generation quality, highlighting different trends depending the of CFG. With CFG, higher top-p values promote greater diversity in sampling, resulting in improved generation quality. In contrast, without CFG, lower top-p values lead to reliance on the model's confident predictions, thereby enhancing generation quality. These findings suggest that adjusting the top-p value can be beneficial, particularly the absence of CFG.
Finally, Figure 4e highlights the influence of temperature on generation quality. A moderate temperature introduces controlled stochasticity during sampling, mitigating the monotonicity inherent in RVQ-token unmasking, where the order is guided by confidence scores. By adjusting the temperature, achieve balance between diversity and fidelity, optimizing overall generation performance.
4.14
10.410
2.64
7.140
2.44
7.000
3.34
2.34
6,707
2.21
-6.613
2.55
7.337
2.05
6.273
1.98
6,227
1.75
5,840
1.75
-5,840
18
28
64
100
0.98
0.94
0.90
0.88
16
48
(a) Step search
(b) Top-p search
(c) Temperature search
Figure 4. Configuration search results for sampling methods with (blue) and without (green) classifier-free guidance (CFG). (a) The effect of varying the number of sampling steps, (b) the impact of different top-p values, and (c) the influence of temperature scaling on confidence scores.
16

---

Efficient Generative Modeling with Residual Vector Quantization-Base Tokens
B.3. Pseudo-code for Sampling
Algorithm 2 Sampling
1: procedure BinaryUnmask(n, L, D, m)
2: Compute the number of masked tokens = (1=(1-m:))
3: Sample kILL from a multivariate hypergeometric distribution with maximum number of selection qi total draws
Σ, %
4: for to L do
5: mli, (D- qi -): -;)]]
6: end for
7: return m
8: end procedure
9:
10: Initialize a fully masked sequence a E N
11: Initialize mask E {0, 1 with zeros.
12: for = ,,...,'' do
2 ~~ p=po=zl ④ m)
14: Apply residual vector quantization for masked tokens:
15:
- RVVQ((
r+
+
17:
(n() x D1
18:
if using random sampling then
m SinaayUnmaskie L, D, m)
20: else if using confidence- based sampling then
21: Update m by selecting n tokens with the confidence-Bee sampling from Section B.
22: end if
23:
end for
24: Return
C. Additional Analysis
C.1. Elaborating Differences with VAR
While both ResGen and VAR leverage the hierarchical structure of residual vector quantization (RVQ), they differ in how this hierarchy is applied throughout the model, leading to key distinctions in representation, generation, and adaptability.
In terms of structure and resolution, VAR assigns each RVQ depth distinct spatial grid (e.g.. 1x1, 2x2, up to the original resolution), requiring a fixed quantization depths to refine a single latent grid whose length matches the VAE encoder's output sequence. This difference also induces the generative process. VAR generates tokens sequentially across depths, meaning that depth can only be generated after completing depth 1, although parallel sampling is possible within each depth. ResGen, on the other hand, predicts cumulative embeddings that aggregate information from multiple depths in single masked- meneed-gennration step. enabling fully parallel prediction along the sequence-length dimension.
hierarchy to be specified in advance. In contrast, ResGen applies all
Furthermore, ResGen offers greater flexibility for modeling data across diverse domains. Because VAR's 's design depends on predefined resolution schedule, adapting it to domains with variable or arbitrary output lengths-such as text-to-specch-can be non-trivial. In contrast, ResGen employs a length-agnostic tokenization scheme, enabling straightforward application to such domains without requiring changes to the model structure. Finally, in terms of resolution flexibility, increasing the quantization depth RVQ (e.g., 16) allows ResGen to represent information at lower effective spatial resolutions (e.g..
17

---

Efficient Generative Modeling with Residual Vector Quantization-Bale Based Tokens
8x8) without extending the sequence length. Achieving comparable resolution is less direct in VAR's depth-resolution coupled structure, and it would require redefining the entire depthhrrsoolutio mapping.
C.2. Further comparison with faster MAR
To ensure fair comparison with MAR and directly address the question of how ResGen performs compared to faster MAR variants, it is important to clarify a key difference in how the generation speed is reported. MAR evaluates efficiency using throughput (i.e., seconds per image averaged over large batch), whereas we report wall-clock time for generating single image on one A100 GPU. This discrepancy in measurement protocols largely explains the apparent difference in reported latency.
Beyond the reporting protocols, fundamental differences in the generative process also distinguish ResGen from MAR. MAR utillizes multi-token prediction followed by continuous diffusion steps, a process that allows for iterative refinement of continuous-valued tokens. In contrast, ResGen performs disjoint token unmasking each step without revising tokens decided in prior steps. This architectural divergence contributes to differing performance characteristics, particularly highlighted by the impact of classifier-free guidance (CFG). Furthermore, although MAR-B can perform well without CFG, achieving its best results, especially with guidance, often requires significantly more sampling steps (c.g., 100+ diffusion steps) compared to ResGen's typically more constrained number of effective steps. This presents a trade-off between MAR-B's parameter efficiency and ResGen's 's inference efficiency, particularly when aiming for high-quality results with guidance.
To evaluate whether ResGen still outperforms speed-optimized MAR variants under these varied conditions and considering these modeling differences, conducted additional experiments in controlled settings. The first set of experiments reduced the number of auto-regressive (AR) steps in MAR while keeping the diffusion step count fixed at 100. As shown in Table 5, even with as few as 16 steps the fastest configuration we tested MAR-B achieved an FID of 4.11, substantially worse than 1 (FID 1.93), even if this MAR-B configuration took roughly 5.6 seconds per image.
ResGen 16
Additionally, we reduced the number of diffusion steps in MAR (with AR steps fixed at 256). shown in Table 6, MAR-B with only 25 diffusion steps required approximately 22.5 seconds per image, yet still underperformed relative ResGen, with FID of 3.38 compared to 1.93. These results highlight that ResGen consistently achieves better sample quality significantly lower latency, even when compared to aggressively optimized MAR configurations.
Table 5. FID (L) for MAR-B/L B/L when varying AR steps with fixed diffusion steps.
Model
AR = 64
AR = 32
AR = 16
MAR-B
2.33
2.44
4.11 4.32
MAR-L
1.81
2.10
Table 6. FID (1) for MAR-B/L when varying diffusion steps with fixed AR steps.
Model
Diff= 100
Diff= = 50
Diff= = 25
MAR-B
2.31
2.39
3.38
MAR-L
1.78
1.83
2.22
C.3. Sensitivity to Masking Strategies
We investigated the sensitivity of ResGen to different masking schedules during training and sampling. Specifically, we trained ResGen-rvq16 models (400K iterations) using three distinct masking strategies: cosine, circle, and exponential. At sampling time, each model was evaluated both under its original training schedule and under different one, resulting in cross-evaluation of masking strategies. All evaluations were conducted with and without classifier-free guidance (CFG), using exponential moving average (EMA) checkpoints.
Interestingly, as shown in Table 7, the best performance (FID 9.53 at 64 steps with CFG) was achieved when training with the circle masking strategy but sampling with the exponential strategy. This suggests that the choice of masking schedule inference time can have a meaningful impact, even when differs from the training configuration. The exponential
18

---

Efficient Generative Modeling with Residual Vector Quantization-Banl Based Tokens
schedule, in particular, unmasks fewer tokens in the early stages and progressively increases the unmasking rate, creating coarse-to-fine decoding process. This appears benefit ResGen by allowing it to first establish stable global structure before refining finer details. Notably, exponential sampling yielded strong performance even when the model was not trained with it, highlighting its robustness and potential as generally effective inference schedule for masked generation with RVQ tokens.
Table 7. FID scores for ResGen-rvqló under different combinations of training and sampling masking strategics. Sampling 64, 48, and 28
steps, with and without classifier-free guidance (CFG).
CFG
Training
Sampling
64 steps
48 steps
28 steps
w/o
cosine
cosine
28.13
28.01
29.18
w/o cosine
exp
32.30
32.70
32.81
w/o circle circle
26.04
26.41
26.73
w/o circle exp
32.44
33.12
33.92
w/o
exp exp
41.12
41.67
41.87
w/
cosine cosine
15.46
15.69
17.27
w/
cosine exp
9.66
9.78
10.08
w/
circle circle
10.09
10.35
10.44
circle exp
9.53
9.72
9.98
w/ exp
exp
12.63
12.65
12.83
Table Performance ResGen-rvay lv compared against leading masked generative models, MAGVIT-v2 and MaskBiT, on image generation. Boldface denotes the best FID score: an asterisk (*) marks scores reported in their original papers.
Model
Code length
FID (w/ CFG) 4
Inference Steps
ResGen-rvq16
64
1.93
63
MAGVIT-v2
256
1.78*
64
MaskBiT
256
1.62*
64
C.4. Comparison with Recent Masked Generative Models
We analyze ResGen's performance against recent leading masked generative models, MAGVIT-v2 (Yu al., 2024) and MaskBit (Weber et al.). As shown in Table 8, these models achieve FID scores of 1.78 and 1.62, respectively. Our ResGen-rvq16 (with CFG) achieves an FID of 1.93 with a comparable number of sampling steps. While their FID scores are lower than 1.93 of our ResGen-rvq16, ResGen presents distinct advantages stemming from its architectural design and modeling approach:
Modeling Cross-Depth Token Dependencies. ResGen predicts cumulative embeddings. thereby explicitly modeling dependencies across the RVQ depth dimension. This contrasts with approaches like MAGVIT-v2 and MaskBit, which predict independent groups of bits from Lookup-Free Quantization (LFQ). Notably, MaskBit's own ablation study (Table 3b their work (Weber al.)) indicates that performance can degrade as the number of independent bit groups increases larger than two, suggesting potential challenges in scaling very high fidelity (which would necessitate more bits or groups). ResGen's strategy of directly modeling these correlations may offer enhanced scalability, particularly for deep quantization schemes. The benefit of this explicit correlation modeling is further supported by internal ablation of ResGen (see Section 5.2.1 and Table 1, left): variant predicting discrete tokens directly in parallel across all depths (instead of cumulative embeddings) demonstrates strong performance, but slightly underperforms compared to our final ResGen using cumulative embeddings, underscoring the efficacy of our chosen strategy.
Resolution and Depth Flexibility. ResGen's proficiency in handling deep RVQ (e.g.. 16-depth) facilitates the use of a lower spatial resolution for the token map (e.g., 8 x 8), while maintaining high reconstruction quality, compared to typical VQ-based methods that use 16 x 16 spatial tokens. The efficient RVQ depth handling in ResGen makes this trade-off viable, offering valuable flexibility in balancing representational capacity and memory footprint.
19

---

Efficient Generative Modeling with Residual Vector Quantization-Bast Bassd Tokens
These distinctions highlight ResGen's unique strengths in managing complex token dependencies and architectural flexibil- ity.
D. Limitations and Future Directions
The proposed method demonstrates favorable memory efficiency alongside competitive sampling speed and generation quality, One potential avenue for further improvement that is not explored in our work utilization key-value (KV) caching within the transformer architecture. By progressively filling tokens, positions that have been completely filled can reuse their precomputed KV values, thereby reducing redundant computations. This strategy significantly enhance the sampling speed and reduce overall computational overhead, making a promising direction for future research.
While our approach is intricately designed around Residual Vector Quantization (RVQ) tokens, recent developments in quantization methods suggest that Finite Scalar Quantization (FSQ) may offer additional benefits (Mentzer et al., 2024). Extending our approach to support FSQ, however, is not straightforward, as it involves distinct tokenization and embedding processes. Nevertheless, exploring this direction could lead to novel quantization strategies and improved generative performance.
Another key observation is that approach achieves high-quality generation with a relatively small number of iterations. We hypothesize that this efficiency, compared to conventional diffusion models, stems from the unmasking process rather than denoising process. Since predicting tokens based on completely unmasked tokens is likely easier than predicting them based on noisy inputs, the model benefits from the simpler prediction task. Despite this empirical success, our work lacks theoretical justification for why such low number of inference steps is sufficient. Providing a formal theoretical and analytic explanation for this phenomenon represents another promising future direction.
20

---

Efficient Generative Modeling with Residual Vector Quantization-Base Tokens
(a) VAR-d30 (FID=1.92)
(b) MAR-H (FID=1.55)
(c) DIT-XI/2 (FID=2.27)
(d) ResGen, ours (FID=1.95)
Figure S. Model comparison ImageNet 256x256 benchmark.
21

---

Efficient Generative Modeling with Residual Vector Quantization-Base Tokens
Figure 6, Randomly generated 256x256 samples by ResGen trained on ImageNet.
22


Reviews:
[{"review_id": "YiP0LC6uWC", "reviewer": "Reviewer_nBTE", "summary": "This paper introduces ResGen, a method that directly predicts vector embeddings for groups of tokens rather than individual tokens. This design reduces the number of inference steps, thereby improving latency. Token masking is employed during training, while multi-token prediction is utilized during inference. Experimental results on image generation and audio synthesis demonstrate the effectiveness of the proposed approach.\n## update after rebuttal\nThanks for the authors rebuttal. Most of my concerns have been addressed and I decide to increase my rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments, which help improve our work's clarity. We address each point below:\n\n\\\n**Q1: Rationale for Predicting Quantized Discrete Tokens vs. Continuous Embeddings**\n\nOur approach of predicting discrete RVQ tokens via cumulative embeddings offers significant inference efficiency advantages over directly predicting high-dimensional continuous embeddings.\n\nWhile direct continuous prediction (e.g., DiT) is valid, it often requires many inference steps. Our method targets discrete RVQ tokens (8x8x16) for efficiency, even though it uses continuous cumulative embeddings internally.\n\nGenerating masked tokens generally demands fewer inference steps than refining high-dimensional continuous vectors, as shown by MaskGiT achieving strong results faster than many continuous methods. Quantization, despite information loss, simplifies the target space, reducing the burden of precisely predicting large continuous vectors (e.g., 8x8x64 in our case).\n\nPredicting cumulative embeddings enables a few step iterative prediction of all 16 token depths. This contrasts with single-step embedding prediction (e.g., GIVT) or continuous diffusion models needing many refinement steps. Quantization is crucial for this multi-depth strategy's feasibility.\n\nIn essence, we leverage internal continuous representations for modeling flexibility but target discrete tokens for efficient inference, particularly suited for our multi-depth prediction approach.\n\n\\\n**Q2: MaskGiT Reimplementation Baseline in Ablation Study (Section 5.2.1)**\n\nThe MaskGiT variant in Table 1 uses 8x8x16 RVQ tokens specifically for a controlled comparison within our ablation study on multi-depth token generation, not to replicate the original MaskGiT's performance on its native 16x16x1 format. The experiments aimed to compare strategies for generating multi-depth RVQ tokens (8x8x16): (i) fully autoregressive (RQ-Transformer), (ii) masked sequence + autoregressive depth (our MaskGiT variant), and (iii) fully masked (ResGen). Fair comparison required all models to operate on the same 8x8x16 RVQ format. The relatively lower performance of the MaskGiT variant (predicting depth-by-depth) compared to ResGen highlights the challenge of extending single-depth masked models to multi-depth RVQ tokens and supports our approach of predicting all depths collectively.\n\nTo further strengthen this analysis and provide a more precise ablation, we conducted additional experiments:\n1.  As detailed in our response to Reviewer aQEq (W1), we implemented an \"AR-ResGen\" model which combines an autoregressive sequence model (like RQ-Transformer's) with ResGen's efficient iterative depth prediction using cumulative embeddings. This isolates the benefit of our depth modeling strategy.\n2.  We also implemented a variant of our method which uses the same masked generation framework introduced in our method section but predicts discrete tokens directly and in parallel across all depths at each step, *instead* of predicting continuous cumulative embeddings. This represents a more direct application of masked diffusion to RVQ depths. This variant achieved FID scores of **12.79 (w/o CFG)** and **2.91 (w/ CFG)**.\n\nThese additional experiments demonstrate that while variations of our method can perform well, our proposed final method using cumulative vector embedding prediction achieves the best overall performance within our ablation study, validating its effectiveness.\n\nWe will revise Section 5.2.1 to clarify the baseline's purpose (controlled comparison on 8x8x16 RVQ tokens), implementation (sequential depth prediction), its role within the ablation, and add the results of our new ablation experiments as requested.\n\n\n\\\n**Q3: Clarity of the Term 'Discrete Diffusion Model'**\n\nWe acknowledge the ambiguity regarding 'discrete diffusion model' and thank you for highlighting it.\n\nOur discussion of 'discrete diffusion models' in the related works primarily referred to models like VQ-Diffusion, GIVT, and conceptually MaskGIT, which learn to reverse a corruption process (like masking) on discrete data (tokens). This was mainly confined to the first paragraph of the related works.\n\nOther mentioned models (e.g., VAR, RQ-Transformer) were included for broader context in token-based modeling but were not necessarily classified by us as discrete diffusion models.\n\nWe also aimed to highlight 'masked diffusion' (corruption via masking only) as an effective subset of the discrete diffusion model, following D3PM and VQ-Diffusion.\n\nWe will revise the paper to provide a precise discrete diffusion model definition as used, clearly delineate this category from related models, explicitly refer masked diffusion, and ensure consistent terminology throughout.\n\n\\\nWe hope these clarifications and the planned revisions adequately address the reviewer's concerns. We appreciate the constructive feedback.", "claims_and_evidence": "See Strength and Weakness.", "methods_and_evaluation_criteria": "See Strength and Weakness.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "N/A.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Strength:\n\n- By eliminating the auto-regressive prediction of depth tokens, the proposed method improve the inference latency.\n\nWeakness and Questions:\n\n- **Method**: My main concern is that the proposed method predicts continuous embeddings rather than discrete tokens, subsequently quantizing these embeddings into discrete tokens. Given this approach, it's unclear why the method doesn't simply predict continuous embeddings directly, similar to diffusion methods. The quantization step inevitably introduces information loss, raising questions about the purpose and effectiveness of using discrete tokens.\n\n- **Experiment**: In Section 5.2.1, the paper presents a reimplementation of MaskGiT using RVQ quantization with a depth of 16. However, the implementation details are unclear. If the reimplementation yields significantly worse results compared to the original MaskGiT—as Table 1 suggests—then its utility as a baseline becomes questionable.\n\n- **Expression**: Throughout the paper, the definition of 'discrete diffusion model' is unclear. Specifically, methods such as MaskGIT, VAR, and RQ-VAE, which are listed in the related works, do not align with what I would consider discrete diffusion models.", "other_comments_or_suggestions": "Please kindly review the weaknesses I have outlined and provide clarifications for each point individually.", "questions_for_authors": "See Weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ad2DEtrKvb", "reviewer": "Reviewer_BBR2", "summary": "This paper introduces ResGen, an efficient generative model leveraging residual vector quantization (RVQ). While RVQ typically enhances image fidelity by increasing quantization depth, it also demands more inference steps during sampling. Instead of sequentially predicting tokens at each depth, ResGen proposes a novel approach: predicting the sum of masked tokens at each layer, known as cumulative tokens. These cumulative tokens are then re-quantized using RVQ. By effectively integrating MaskGit with RVQ, ResGen achieves impressive performance improvements.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the positive evaluation and constructive feedback.\n\nWe address the specific points raised below:\n\n**Essential References Not Discussed (HART):**\nThank you for bringing HART to our attention. We agree it is a relevant recent work and will incorporate a discussion into our Related Work section. HART proposes a hybrid approach, distinct from purely discrete tokenizers like ours (ResGen, VAR) or continuous ones (MAR, GIVT). It decomposes latents into discrete tokens (modeled autoregressively) and continuous residuals (modeled via diffusion). This hybrid strategy aims to reduce sampling steps compared to continuous methods like MAR.\n\n**Theoretical Claims (Sec 3.2, Lines 263-265):**\nThank you for the careful reading and request for clarification. There might be slight confusion in terminology depending on perspective (encoding vs. decoding of VQ). Let us clarify our notation:\n*   `z` represents the continuous cumulative embeddings corresponding to the target clean tokens `x^(0)`. Hence the term `q(z | x^(0), x^(t))` represents a form of dequantization or embedding lookup.\n*   `p(x^(0)|z, x^(t))` represents the inference of the true clean tokens `x^(0)` given this embeddings `z` and the current masked tokens `x^(t)`. This involves finding the discrete tokens `x^(0)` whose corresponding embeddings are `z`. This step effectively performs the **quantization** of the continuous embedding `z` back into the discrete RVQ code space.\n\nSo, `p(x^(0)|z, x^(t))` indeed relates to determining the discrete tokens `x^(0)` *from* the continuous embedding `z`, which involves the RVQ quantization mechanism applied to `z`. \n\n**Weakness 1: Potential for Further Inference Speed-up:**\nThank you for this suggestion. In the vision domain, reducing sampling steps to 18 (Appendix B.2) improves inference speed but slightly decreases generation quality (FID 3.94). However, experiments in the audio domain show that fewer steps (8 and 16 steps) still achieve comparable performance (see Table below).\n|Continuation|WER|CER|SIM-o|SIM-r|\n|-|-|-|-|-|\n| melvae-resgen-25step|1.94|0.53|0.5421|0.5701|\n| melvae-resgen-16step|1.92|0.53|0.5419|0.5705|\n| melvae-resgen-8step|1.92|0.54|0.5429|0.5710|\n| rvqvae-resgen-25step|1.86|0.50|0.5853|0.5886|\n| rvqvae-resgen-16step|1.87|0.52|0.5820|0.5864|\n| rvqvae-resgen-8step|1.86|0.51|0.5847|0.5886|\n\n|Cross|WER|CER|SIM-o|SIM-r|\n|-|-|-|-|-|\n|melvae-resgen-25step|1.75|0.48|0.5597|0.6061|\n|melvae-resgen-16step|1.92|0.53|0.5419|0.5705|\n|melvae-resgen-8step|1.93|0.54|0.5433|0.5708|\n|rvqvae-resgen-25step|1.70|0.46|0.6037|0.6307|\n|rvqvae-resgen-16step|1.75|0.46|0.6037|0.6302|\n|rvqvae-resgen-8step|1.95|0.50|0.5898|0.6108|\n\n\n**Weakness 2: Elaborating Differences with VAR:**\nThank you for this suggestion; a clearer comparison with VAR will indeed strengthen the paper. While both ResGen and VAR utilize the hierarchical nature of RVQ, they differ:\n*   **Structure & Resolution:** VAR assigns different spatial resolutions to different RVQ depths (e.g., 1x1, 2x2, ... up to original resolution), requiring a predefined hierarchy. RVQ uses all quantization depths to refine the representation at a *single* resolution which is identical to the length of output sequence of the VAE encoder.\n*   **Generative Process:** VAR generates tokens autoregressively across depths, although sampling within a depth can be parallel given the previous depth. ResGen predicts *cumulative embeddings* representing multiple depths simultaneously within a masked generation framework, allowing parallel prediction across the sequence length dimension.\n*   **Task Adaptability:** VAR's predefined resolution hierarchy might be less straightforward to adapt to tasks with arbitrary output lengths, such as text-to-speech, where ResGen is easily applied.\n*   **Resolution Flexibility:** RVQ allows us to achieve lower final spatial resolutions (e.g., 8x8) by increasing quantization depth (e.g., 16), offering flexibility in balancing sequence length and depth. Achieving similarly low spatial resolution might be less direct in VAR's depth-resolution coupled structure.\n\nWe will incorporate a more detailed discussion contrasting these aspects in the related work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Make sense", "theoretical_claims": "I check the Section 3.2. \nIn lines 263-265, Shouldn't p(x^(0)|z, x^(t)) corresponds to RVQ dequantization instead of quantization? It seems there might be a mix-up here. Could you clarify?", "experimental_designs_or_analyses": "Yes, no issues.", "relation_to_broader_scientific_literature": "All appendix", "essential_references_not_discussed": "HART: Efficient Visual Generation with Hybrid Autoregressive Transformer. [ICLR'25]", "other_strengths_and_weaknesses": "Strengths:\n1. ResGen effectively combines MaskGit and RVQ for efficient generative modeling.\n2. ResGen achieve significant performance improvements over MaskGit and efficiency improvement over RQ-Transformer.\n3. The author provides the theoretical justification for RVQ.\n\nWeaknesses:\n1. Will the inference step be further decreased for faster inference?\n2. The discussion on the differences between ResGen and VAR could be elaborated. Since VAR also considers the hierarchical depth in RVQ and formulates it as a scale, a more detailed comparison would better highlight ResGen’s advantages and solidify its contributions.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "li0PnYqxjK", "reviewer": "Reviewer_q3hF", "summary": "This paper introduces ResGen, an efficient generative modeling method that uses Residual Vector Quantization (RVQ) for high-fidelity data generation. Its key innovation lies in predicting collective token embeddings rather than individual tokens, which decouples inference complexity from quantization depth.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thorough assessment, positive feedback on our claims, methodology, and supplementary material, and for recognizing the conceptual innovation of ResGen. We appreciate the constructive suggestions for further improvement.\n\nRegarding the points raised:\n\n**Limited Dataset Diversity/Scalability (W1):** We appreciate the suggestion to evaluate ResGen on more diverse datasets and higher-resolution settings. Our current study focused on established benchmarks like ImageNet and standard TTS datasets to ensure the adaptability of this method across different domains. We agree that broader validation is important and plan to explore ResGen's scalability and generalization on more diverse tasks, including higher-resolution image generation, as a key direction for future work. But to provide additional context on model scalability, we conducted scaling experiments with ResGen-rvq16, increasing model parameters from 574M to 1B on ImageNet. Although these scaling experiments are ongoing (currently at 400K iterations, batch size 256 on 4 GPUs), intermediate results reveal FID improvements, as shown below:\n| FID (Exp Sampling) | 64 steps | 48 steps | 28 steps |\n| :----------------- | :------- | :------- | :------- |\n| **w/o CFG 574M**   | 32.44    | 33.12    | 33.92    |\n| **w/o CFG 1B**     | 26.58    | 26.74    | 28.39    |\n| **w/ CFG 574M**    | 9.53     | 9.72     | 9.98     |\n| **w/ CFG 1B**      | **8.07** | **8.10** | **8.82** |\nWe will include the fully converged results in the manuscript.\n\n**Analysis of MAR-B vs. ResGen-rvq16 Performance (W2):** Thank you for prompting a deeper analysis of the results in Table 2. While MAR-B has fewer parameters and achieves slightly better FID *without* classifier-free guidance (CFG), ResGen-rvq16 excels *with* CFG. This difference stems from their distinct generative mechanisms. MAR-B utilizes multi-token prediction followed by continuous diffusion steps, allowing iterative refinement and revision of previously generated tokens. In contrast, ResGen performs disjoint token unmasking at each step without revisiting prior decisions. This fundamental difference in modeling leads to varying performance characteristics, particularly highlighted by the impact of CFG. Furthermore, although MAR-B can perform well without CFG, achieving its best results often requires significantly more sampling steps (e.g., 100+ diffusion steps) compared to ResGen, impacting inference speed. Thus, there is a trade-off between MAR-B's potential parameter efficiency (in certain configurations) and ResGen's inference efficiency, especially when aiming for high-quality results with guidance. We will incorporate this more detailed comparative analysis into the revised manuscript.\n\n**Sensitivity to Masking Strategies (Q1):** This is an excellent question regarding the robustness of our approach. We investigated ResGen's sensitivity by training ResGen-rvq16 models (400K iterations) using three distinct masking schedules during training: circle, exponential, and cosine. We then evaluated these models under two sampling conditions: using the same or different  schedule during sampling than training for all models.\n\nOur results, summarized below (FID scores, lower is better), show interesting interactions:\n\n| CFG     | Training | Sampling | 64 steps | 48 steps | 28 steps |\n| :------ | :------- | :------- | :------- | :------- | :------- |\n| **w/o** | cosine   | cosine   | 28.13    | 28.01    | 29.18    |\n| **w/o** | cosine   | exp      | 32.30    | 32.70    | 32.81    |\n| **w/o** | circle   | circle   | 26.04    | 26.41    | 26.73    |\n| **w/o** | circle   | exp      | 32.44    | 33.12    | 33.92    |\n| **w/o** | exp      | exp      | 41.12    | 41.67    | 41.87    |\n| **w/**  | cosine   | cosine   | 15.46    | 15.69    | 17.27    |\n| **w/**  | cosine   | exp      | 9.66     | 9.78     | 10.08    |\n| **w/**  | circle   | circle   | 10.09    | 10.35    | 10.44    |\n| **w/**  | circle   | **exp**  | **9.53** | **9.72** | **9.98** |\n| **w/**  | exp      | exp      | 12.63    | 12.65    | 12.83    |\n\nNotably, the best performance (FID 9.53 at 64 steps with CFG) was achieved when training with the *circle* masking strategy but sampling using the *exponential* strategy. This suggests that the different training and sampling schedules impact the final quality. The exponential schedule unmasks fewer tokens in the crucial early stages and more tokens later. This coarse-to-fine unmasking during inference appears beneficial for ResGen, likely allowing the model to establish a more stable initial prediction before revealing finer details. The strong performance under exponential sampling, even when trained differently, indicates its effectiveness for inference with ResGen. We will include a detailed discussion of these findings in the revised manuscript.", "claims_and_evidence": "The claims made in the paper—namely, improved efficiency and generation fidelity of RVQ-based generative modeling—are generally well-supported by clear evidence presented through extensive experiments.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are well-suited to the tasks at hand.", "theoretical_claims": "The paper formulates a probabilistic framework. While the paper presents equations clearly, no explicit mathematical proofs requiring detailed checking are presented.", "experimental_designs_or_analyses": "The experimental design is sound and well-executed. The authors clearly outline training configurations, baseline comparisons, and evaluation metrics. Ablation studies further strengthen the robustness of their conclusions.", "relation_to_broader_scientific_literature": "The paper fits well within the broader context of generative modeling literature. It makes a significant advance in RVQ-based generative approaches by solving efficiency bottlenecks that arise from depth-dependent inference complexity.", "essential_references_not_discussed": "No significant omissions were noted.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- The method introduces a valuable conceptual innovation—predicting collective embeddings to decouple inference complexity from RVQ depth.\n- Comprehensive and clear supplementary material strengthens transparency and reproducibility.\n\n**Weaknesses:**\n\n- While extensive, evaluations are limited to specific datasets (ImageNet, standard TTS benchmarks). Performance on more diverse datasets or more challenging, higher-resolution settings could be further explored to validate scalability.\n- The paper would benefit from deeper analysis of the results in Table 2, particularly regarding the substantial performance gap between MAR-B and ResGen-rvq16, given that MAR-B has fewer parameters.", "other_comments_or_suggestions": "None", "questions_for_authors": "How sensitive is ResGen's performance to different masking strategies, and could alternative masking approaches significantly affect the results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1RLyia70K0", "reviewer": "Reviewer_Bd57", "summary": "This paper proposes ResGen, an efficient RVQ-based generative modeling for balancing quality and efficiency. It involves a masked token modeling strategy similar to MaskGiT, and a multi-token prediction pipeline inspired by CLaM-TTS, in a discrete diffusion process and variational inference. Experimental results demonstrate that ResGen outperforms autoregressive counterparts in both tasks, without compromising sampling speed.\n\n## update after rebuttal\nWe thank authors' detailed rebuttal. Most of my concerns have been solved. I would like to improve the rate.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their detailed and constructive feedback. We appreciate the recognition of ResGen's motivation and its potential to address computational costs in RVQ-based generation. We address the specific concerns and questions below:\n\n**Concerning the fair comparison with MAR and further comparison with faster MAR (Claim Question)**, there's a key difference in how speed is measured. MAR reports throughput (sec/image averaged over a large batch, e.g., 256), while we report wall-clock time for single image generation on one A100 GPU. This accounts for the apparent discrepancy (0.3s vs. >100s). To directly address whether ResGen outperforms faster MAR variants, we conducted additional experiments:\n\n1.  **Reducing MAR's AR Steps (Diffusion=100):** Even MAR-B at 16 AR steps (fastest tested, ~5.6s/image) yielded worse FID than ResGen-rvq16 (FID 4.11 vs. 1.93).\n    |MAR FID (Diffusion=100)|AR=64|AR=32|AR=16|\n    |:-|:-|:-|:-|\n    |MAR-B|2.33|2.44|4.11|\n    |MAR-L|1.81|2.10|4.32|\n\n2.  **Reducing MAR's Diffusion Steps (AR=256):** MAR-B at 25 diffusion steps (~22.5s/image) performed worse than ResGen-rvq16 (FID 3.38 vs. 1.93).\n    |MAR FID (AR=256)|Diff=100|Diff=50|Diff=25|\n    |:-|:-|:-|:-|\n    |MAR-B|2.31|2.39|3.38|\n    |MAR-L|1.78|1.83|2.22|\n\nThese results demonstrate ResGen's consistent advantage in both speed and quality, even against accelerated MAR configurations. We will add this comprehensive comparison to the revision.\n\n**To clarify the experimental environment (Potential Improvement 1, W2.2):** All ResGen evaluations used a single NVIDIA A100 GPU. Training details are in the appendix. For Table 1, we reported the best performance for each model based on a hyperparameter search. Detailed sampling configurations and hyperparameters will be explicitly included in the final manuscript. For Table 2, max batch size was the largest fitting on one A100 using checkpoints' best CFG settings. Inference speed is measured as the wall-clock time required to generate a single sample. We will explicitly state these details in the final manuscript.\n\n**Regarding RQ-Transformer guidance in Table 1 (W2.1):** Thank you for noting potential confusion. For the ablation in Table 1 comparing generation on 8x8x16 tokens, we used CFG for ResGen, RQ-Transformer, and MaskGiT to ensure a fair comparison under identical conditions. As the reviewer noted, the RQ-Transformer paper uses rejection sampling. Rejection sampling is applicable to any generative model, but was not used in Table 1. We will clarify it in the revision.\n\n**Concerning scaling to deeper RVQ depths (Additional Weakness 2):** For images, we used depths 8 and 16 due to excellent reconstruction fidelity (RFID 1.29 and 0.67, Appendix A.4). However, in audio (Table 3), we extensively evaluated deeper depths (32, 72) and observed consistent performance improvements, demonstrating stable training and benefits from increased depth in that domain. This suggests potential benefits for vision too, though not explored here due to already high fidelity at depth 16.\n\n**Regarding clarifying the benefit of discrete diffusion over autoregressive generation:** Please refer to the newly added experiments and analyses presented in our response to **Reviewer aQEq** (\"Regarding W1: Isolating the Efficiency Gains of Depth Modeling\"), which directly address this point.\n\n**Regarding generation scaling law (Q1):** We investigated scaling ResGen-rvq16 from 574M to 1B parameters on images (400K iterations, batch 256 on 4 GPUs). We observed consistent FID improvements across sampling steps:\n\n|FID (Exp Sampling)|64 steps|48 steps|28 steps|\n|:-|:-|:-|:-|\n|**w/o CFG 574M**|32.44|33.12|33.92|\n|**w/o CFG 1B**|26.58|26.74|28.39|\n|**w/ CFG 574M**|9.53|9.72|9.98|\n|**w/ CFG 1B**|**8.07**|**8.10**|**8.82**|\nThis indicates a positive scaling trend, which we will add to the manuscript.\n\n**Regarding the complexity and overhead of the Mixture of Gaussians (MoG) head (Additional Weakness 1, Q2)**, we acknowledge the need for clarity. The MoG head's computational cost is manageable. As detailed in Appendix A.2, parameter prediction involves projecting the hidden output (size `O`) to `K` mixture probabilities, `K` mean vectors (size `H`), and affine parameters. This leads to a projection complexity dominated by `O(O*K*H)`. For our vision model (`O=1152, K=1024, H=64`), this cost is comparable to a standard softmax layer with a ~64K vocabulary (`V = K*H`), which is practical. For higher-dimensional embeddings like in audio (`H=512`), we use low-rank projection for the means (`μ = M * μ_tilde + s`, `h << H`), reducing the dominant computational term to `O(O*K*h + H*h)`, again making the effective cost similar to a ~64K vocabulary model (`h=64`). This technique, previously used in CLaM-TTS, significantly mitigates overhead. We recognize these details were mainly in the Appendix and will clarify the MoG parameterization, its computational cost, and the low-rank optimization in the main paper.", "claims_and_evidence": "Claim: ResGen outperforms autoregressive counterparts in both tasks, without compromising sampling speed.\n\nQuestion: In Tab. 1, ResGen shows a slightly higher FID score compared to MAR-L, achieving comparable quality (1.93 of ResGen v.s. 1.78 of MAR-L) with a much faster sampling speed. However, MAR [autoregressive’24] by Li et al. of NeurIPS 2024 demonstrates the inference time of MAR-L about 0.3 sec/image, which is significantly faster than the wallclock time over 100 sec in this paper in Fig. 2. Moreover, MAR with lower steps is also sufficient to achieve a strong generation quality. Could ResGen achieve higher performance compared with a faster MAR with lower steps?\n\nPotential Improvement: It will be helpful to figure out the details of the experiment environment, especially the GPU type and AR steps used in MAR. Further evaluation of MAR with lower steps could also be beneficial for proving the necessity of using ResGen.", "methods_and_evaluation_criteria": "Methods: \nThe motivation of this method is to eliminate the problem of sampling complexity associated with sequence length and depth for efficient RVQ-based image generation. The proposed masking and prediction strategy makes sense for the problem. And the discrete diffusion process helps with high-fidelity generation.\n\nEvaluation Criteria: \nFor image generation,  the paper relies on standard generative modeling metrics like FID, and evaluates the efficiency using metrics such as inference time and batch size. For audio tasks, the paper follows VALL-E (Wang et al., 2023) and CLaM-TTS (Kim et al., 2024).\n\nPotential Improvement:\n1.\tBetter to figure out the details of the experiment environment, especially when evaluating the maximum batch size and inference time.", "theoretical_claims": "This paper do not have some theoretical claims.", "experimental_designs_or_analyses": "Strengths: \n1.\tThey run on both visual and audio tasks: ImageNet 256×256, and zero-shot TTS tasks.\n2.\tThey compare memory usage (max batch size) and speed to a variety of generative baselines.\n3.\tThey show ablations of top-p, number of steps, and temperature in the appendix.\n\nPotential Weaknesses: \n1.\tIn Tab. 1, RQ-Transformer uses rejection sampling rather than CFG, which might be misleading.\n2.\tIn Tab. 1 and Fig. 2, The speed comparisons and maximum batch size comparisons are primarily self-reported. Without consistent inference setups or more hardware details, we can’t be sure these speed gains generalize.\n3.\tComparing with the conventional RQ-transformer which is an autoregression architecture, it is not clear that the preliminary improvement is from the discrete diffusion process or the masking and prediction strategy. A comparison between using autoregressive generation and discrete diffusion generation could help clarify this, similar to MAR [autoregressive’24] by Li et al. of NeurIPS 2024.", "relation_to_broader_scientific_literature": "No more specific contributions to a broader scientific literature.", "essential_references_not_discussed": "Most of the references have been discussed.", "other_strengths_and_weaknesses": "Strengths:\n1.\tTake advantage of RVQ and address the computational cost problem, bringing residual codebook design back for fast and high-fidelity image generation.\n\nAdditional Weaknesses (detailed):\n1.\tComplex Implementation: The usage of a mixture of Gaussians for each token plus multi-depth unmasking is quite complicated. The authors do not deeply address computational overhead or memory usage for these mixture parameters, especially as code dimension grows.\n2.\tUnclear Large-Scale Scaling: Since the authors use depth 16, fewer than the RVQ depth of 32, it remains unclear if that scaling continues to yield benefits or stable training for the performance.", "other_comments_or_suggestions": "Codebook usage details: Detailing the usage of the codebook, especially at each depth, could help in understanding how the masking and prediction strategy and RVQ process work.", "questions_for_authors": "1.\tGeneration Scaling Law: How does ResGen perform when scaling up from 574M for image generation? Could deeper depth gain more from a larger model with more parameters?\n2.\tMixture-of-Gaussians Overhead: Since each token position has multiple mixture components, how big is the compute overhead compared to simpler single-Gaussian or discrete classification heads, especially for wide embeddings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LHOVjvggvT", "reviewer": "Reviewer_aQEq", "summary": "This paper proposes an efficient generative framework (ResGen) to model residual tokens that has an additional depth dimension in the code sequences. Specifically, ResGen adopts the masked generative framework to sample tokens, and leverages Gaussian mixtures to directly predict the sum of masked token embeddings. This method achieves strong results with improved efficiency on both class-conditional image generation and text-to-speech tasks.\n\n## update after rebuttal\nThe authors' response addressed my concerns regarding the efficiency of the proposed GMM head, in comparison with the AR head in RQ-Transformer. Therefore, I have adjusted my rating to borderline accept. However, I observed that the major performance gains in FID primarily come from the use of a MaskGIT-like generation framework, which is not an innovation of this work. I recommend that the authors clarify this point in their paper to ensure transparency and accuracy in reporting their contributions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive feedback. We address the specific weaknesses and questions raised below:\n\n**Regarding W1: Isolating the Efficiency Gains of Depth Modeling**\nWe agree that comparing ResGen (masked generation) directly with RQ-Transformer (autoregressive generation) makes it difficult to isolate the efficiency gains specifically from our depth modeling strategy versus the gains from the masked generation framework itself. To provide a stricter ablation, as suggested:\n*   **New Experiment:** We created an \"AR-ResGen\" variant. This model uses the *exact same spatial autoregressive transformer* as the RQ-Transformer baseline. The key difference is that it replaces RQ-Transformer's autoregressive depth transformer with our *cumulative embedding prediction MLP* (of similar size) to handle the depth dimension. This isolates the effect of the depth modeling approach while keeping the sequence modeling framework constant (autoregressive).\n*   **Results:** This AR-ResGen achieves better FID scores than the original RQ-Transformer, particularly with few iterative refinement steps for the depth prediction, demonstrating the efficiency of our cumulative embedding approach even within an AR framework. The results (FID) are:\n    | AR-ResGen Iterations | w/o CFG | w/ CFG |\n    | :------------------- | :------ | :----- |\n    | 1                    | 27.45   | 6.47   |\n    | 2                    | 24.10   | 5.33   |\n    | 4                    | 23.75   | 5.30   |\n    | 8                    | 23.48   | 5.22   |\n*   **Conclusion:** This targeted ablation confirms that our cumulative embedding prediction for depth modeling is inherently more efficient and effective than standard autoregressive depth handling, independent of the sequence generation strategy. We will include these findings in the revised manuscript.\n\n**Regarding W2: Comparison with MAGVIT-v2 and MaskBit**\nWe thank the reviewer for highlighting the missing comparisons with MAGVIT-v2 and MaskBit. We acknowledge their strong performance (FID 1.78 and 1.62) and will add these results to Table 1 and Figure 2 for context. Our current FID is 1.93 with similar steps. While their FID scores are currently lower, ResGen offers distinct advantages:\n*   **Theoretical Advantage (Modeling Correlations):** ResGen predicts cumulative embeddings to explicitly model dependencies across the RVQ depth dimension. This differs from methods like MAGVIT-v2/MaskBit using Lookup-Free Quantization (LFQ) and predicting independent bit groups. MaskBit's own ablation (Table 3b) shows performance degrading as independent groups increase, suggesting potential scaling limitations for higher fidelity (requiring more bits/groups). ResGen's approach of modeling correlations directly may offer better scalability for deep quantization or numerous discrete outputs. To show the benifit of modeling these correlation between tokens across depth, we implemented a variant of our method which uses the same masked generation framework introduced in our method section but predicts discrete tokens directly and in parallel across all depths at each step, instead of predicting continuous cumulative embeddings. This variant achieved FID scores of 12.79 (w/o CFG) and 2.91 (w/ CFG). While this performance surpasses the other comparison methods evaluated in our study, it is slightly worse than our final proposed model which predicts cumulative embeddings, thus supporting the efficacy of the cumulative embedding strategy to capture such token correlations.\n*   **Practical Advantage (Resolution/Depth Trade-off):** Our use of 16-depth RVQ enables a lower spatial resolution (8x8) than typical 16x16 VQ methods, while achieving high reconstruction quality measure in rFID. ResGen's efficient depth handling makes this viable, offering flexibility in memory usage and model design.\nWe will clarify these points and add the comparisons to the manuscript.\n\n**Regarding Q1: Inclusion of VAR-d30 in Figure 2**\nThank you for this suggestion. To provide a more complete comparison of performance and efficiency trade-offs, we agree that including VAR-d30 in the wallclock time comparison (Figure 2) would be beneficial. We will update Figure 2 to include VAR-d30 in the revised manuscript.\n\n**Regarding Q2: Exact Coordinates for Figure 2 (left)**\nCertainly. The exact coordinates (Wallclock Time [s], FID) for the data points in Figure 2 (left) are:\n*   DiT: [ (4.68, 2.27) ]\n*   VAR: [ (0.16, 3.60), (0.19, 2.95), (0.24, 2.33) ]\n*   MAR: [ (89.69, 2.31), (105.6, 1.78), (133.01, 1.55) ]\n*   RQTran: [ (3.61, 3.89), (3.73, 3.80) ]\n*   ResGen-rvq8: [ (0.84, 2.87), (1.43, 2.78), (1.86, 2.75) ]\n*   ResGen-rvq16: [ (0.98, 2.26), (1.67, 2.14), (2.28, 1.98) ]\n*   MaskGiT: [ (0.98, 6.18) ]\nSolid lines connect different models; dashed lines connect points for the same model with varying sampling steps. We will add this detailed information, likely in the figure caption or appendix, for clarity in the revision.", "claims_and_evidence": "The claims in this paper are supported by experimental results or prior studies.", "methods_and_evaluation_criteria": "I am not familiar with text-to-speech evaluation. The metrics used for class-conditional image generation are common and standard.", "theoretical_claims": "I did not fully check the details of the theoretical proofs in Section 3.2.", "experimental_designs_or_analyses": "The experimental designs are sound.", "relation_to_broader_scientific_literature": "The key contribution of this paper is to model residual tokens more efficiently.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths:**\n- The paper is well written and easy to follow.\n- The proposed method can be generally applied to both image generation and text-to-speech generation.\n\n**Weaknesses:**\n- Residual tokens have both length and depth dimensions. The focus of ResGen is to model **the depth dimension** more efficiently, i.e., by predicting embeddings of depth tokens collectively rather than individually. However, it is not clear to what extent does this method improve the efficiency over its baseline, i.e., a depth transformer in RQ-Transformer. Notably, ResGen adopts the masked generative framework to model **the length dimension**, which requires much fewer sampling steps than the autoregressive framework used in RQ-Transformer. This makes ResGen and RQ-Transformer less comparable. The authors are encouraged to provide a strict ablation on this. For example, using the the same autoregressive framework to model the length dimension.\n\n- Comparisons with recent masked generative models such as MAGVIT-v2 [1] and MaskBit [2] are missing in Table 1 and Figure 2. It seems ResGen does not show clear advantages in terms of both generation quality and sampling speed over these methods.\n\n[1] Yu, Lijun, et al. \"Language Model Beats Diffusion-Tokenizer is key to visual generation.\" The Twelfth International Conference on Learning Representations.\n\n[2] Weber, Mark, et al. \"MaskBit: Embedding-free Image Generation via Bit Tokens.\" Transactions on Machine Learning Research.", "other_comments_or_suggestions": "Please see weaknesses.", "questions_for_authors": "1. Why not include VAR-d30 in Figure 2 for wallclock time comparison?\n\n2. Could the authors provide the exact coordinates of the data points in Figure 2 (left one)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces an efficient RVQ-based generative model for fast sampling by predicting vector embeddings for groups of tokens. It received mixed reviews initially, where reviewers have concerns regarding motivations, differences with prior work, and missing partial comparisons with competing methods.

After the rebuttal, the authors addressed most of the concerns, and three reviewers increased their final ratings. By taking the overall contributions and additional results in the rebuttal into consideration, AC believes most significant concerns have been alleviated and this paper is above the acceptance threshold.
Decision: Accept (poster)