Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching
Authors: Enshu Liu
Xuefei Ning
Yu Wang
Zinan Lin
Venue: ICLR 2025 Poster (2025)
Keywords: image autoregressive models
parallel decoding
distillation
Primary area: generative models

Abstract: Autoregressive (AR) models have recently achieved state-of-the-art performance in text and image generation. However, their primary limitation is slow generation speed due to the token-by-token process. We ask an ambitious question: can a pre-trained AR model be adapted to generate outputs in just one or two steps? If successful, this would significantly advance the development and deployment of AR models. We notice that existing works that attempt to speed up AR generation by generating multiple tokens at once fundamentally cannot capture the output distribution due to the conditional dependencies between tokens, limiting their effectiveness for few-step generation. To overcome this, we propose Distilled Decoding (DD), which leverages flow matching to create a deterministic mapping from Gaussian distribution to the output distribution of the pre-trained AR model. We then train a network to distill this mapping, enabling few-step generation. The entire training process of DD does not need the training data of the original AR model (as opposed to some other methods), thus making DD more practical. We evaluate DD on state-of-the-art image AR models and present promising results. For VAR, which requires 10-step generation (680 tokens), DD enables one-step generation (6.3$\times$ speed-up), with an acceptable increase in FID from 4.19 to 9.96. Similarly, for LlamaGen, DD reduces generation from 256 steps to 1, achieving an 217.8$\times$ speed-up with a comparable FID increase from 4.11 to 11.35. In both cases, baseline methods completely fail with FID scores $>$100. As the first work to demonstrate the possibility of one-step generation for image AR models, DD challenges the prevailing notion that AR models are inherently slow, and opens up new opportunities for efficient AR generation. The code and the pre-trained models will be released at https://github.com/imagination-research/distilled-decoding. The project website is at https://imagination-research.github.io/distilled-decoding.

Published as a conference paper at ICLR 2025
DISTILLED DECODING 1: ONE-STEP SAMPLING OF IMAGE AUTO-REGRESSIVE MODELS WITH FLOW MATCHING
Enshu Liu;" Xuefei Ning, Yu Wang, Department of EE, Tsinghua University les230ma i 1s. s. t. nghua. edu. on com
Zinan Lin
Microsoft Research z i nan in@mi wwwsort......... com
foxdoraame@gmai yu-wangema1 1. t s nnghua edu. cn
ABSTRACT
Autoregressive (AR) models have recently achieved state-of-the-art performance in text and image generation. However, their primary limitation is slow genera- tion speed due to the token-by-token process. We ask ambitious question: can pre-trained AR model be udapted to generate outputs in just one or two steps? If successful, this would significantly advance the development and deployment AR models. We notice that existing works that attempt to speed up AR generation by generating multiple tokens at once fundamentally cannot capture the output distribution due to the conditional dependencies between tokens, limiting their ef- fectiveness for few-step generation. To overcome this, we propose Dist i ed Decoding (DD), which leverages flow matching to create deterministic map- ping from Gaussian distribution to the output distribution of the pre-trained AR model. We then train network to distill this mapping, enabling few-step genera- tion. The entire training process of DD does not need the training data of the orig- inal AR model (as opposed to some other methods), thus making DD more practi- cal. We evaluate DD on state-of-the-an image AR models and present promising results. For VAR, which requires 10-step generation (680 tokens). DD enables one- step generation (6.3 x speed-up), with an acceptable increase in FID from 4.19 to 9.96 on ImageNet-256. Similarly, for LlamaGen, DD reduces generation from 256 steps to 1, achieving an 217.8x speed-up with a comparable FID increase from 4.11 to 11.35 on ImageNe fail with FID scores > 1000. DD also excels text-to-imann imo-iaage generation, reduc- ing the generation from 256 steps to 2 for LlamaGen with minimal FID increase from 25.70 28.95. As the first work to demonstrate the possibility of one- step generation for image AR models, DD challenges the prevailing notion that AR models are inherently slow, and opens up new opportunities for efficient AR generation. The code and the pre-trained models will be released at https::/ /// github, com/ htt.:/aa............aaaaaateeeee project website is at httos: atttos///iooohatooooeeeeorror.. bistilled-decocinn
both cases, baseline methods completely
The .o/
INTRODUCTION
Autoregressive (AR) models (Van den Oord et al. 2016 Chen et al. 2018 Esser et al. 2021 Razavi
al.
2019 Lee et al. 2022; Yu al.] 2021] Chang al. [022] et al. [2023] 2024a; Touvron
et al.
2023a by Ouyang et al.
2022) are the foundation of sfate-of ffute-of-the-ape
the-ahh art (SOTA) models for text
generation (e.g., GPT (Brown
2020 Radford et al. 2019 Radford 2018 Achiam et al. 2023)) and
image generation (e.g., VAR (Tian et 2024). LlamaGin (Sun et 20244).
Despite their impressive performance, AR models suffer from slow generation speeds due their sequential generation process. More specifically, AR models formulate data (e.g., text, images) as a sequence of tokens and are trained to predict the conditional probability distribution one token given all previous tokens. This means that AR models can only generate data in a token-by-token
"Work mostly done during Enshu Liu's internship at Microsoft Research Projecct advisor: Zinan Lin

---

Published as conference paper at ICLR 2025
DD-Istep (218x)
DD-2step (117x)
DDD-42step (5.7x)
L
Figure 1: Qualitative comparisons between DD and vanilla LlamaGen Sun et al. (2024) on ImageNet x We show that the generated images of DD have small quality loss compared to the pre-trained AR model, while achieving x000 speedup. More examples are in App. G
"Activate Upholstered Fabric Armchair"
"Butterfly Women's T-Shirt"
"A pair shoes on the floor"
"A bowl fresh fruits"
Figure 2: Qualitative results of DD-2step on text-to-image task. The model is distilled from LlamaGen model with prompts from LAION-COCO dataset. The speedup around 93 x compared to the teacher model. More examples are in App. G
fashion, which is slow. For example, LlamaGien333M-2256-2 requires 256 steps (~5 seconds to generate one 256x 256 image.
In this paper, we ask the ambitious question:
Can a pre-trained AR model be adapted to generate data in a few (e.g., one two) steps?
successful, this would greatly benefit both the model developers (e.g.. reducing the testing and deployment cost) and the end users (e.g., reducing the latency).
Apparently, this problem challenging. While speeding up AR models by generating multiple to-
kens at time (Fig. 4 is extensively studied in literature (Ning et al. 2024a Liu et al., 2024 Jin
2024 Stern al.
2018: Kou al.] 2024; Gloeckle et al.) 2024] Cai et al., 2024, Santilli 2022 Li et al. 2024a), none of these works were able to generate the
al. 2023 Chang et al.
entire sample (i.e., all tokens) in one step. Indeed, we find that there is fundamental reason for this limitation (Sec. 3.1 Sampling multiple tokens in parallel (given previous tokens) would have to assume that these tokens are conditionally independent with each other, which is incorrect in prac- tice. As extreme, generating all tokens in one step (i.e., assuming that all tokens independent) would completely destroy the characteristics in data (Sec. (3.). This insight suggest that few-step AR generation requires a fundamentally different approach.
In this paper, we introduce Distilles bestilled Decoding (DD) (Fig. 4), a novel method for distilling pre-trained AR model for few-step sampling. In each AR generation step, we use flow matching (FM) (Liu et al. 2022 Lipman et al. 2022) to transform a random noisy token, sampled from an isotropic Gaussian distribution, into the generated token. FM ensures that the generated token's distribution aligns with that of the AR model. However, this generation process would be even slower than vanilla AR due the added FM overhead. The actual benefit is that, the mapping between noisy and generated token is determinstic. Therefore, we can train model that directly distills the mapping between the entire sequence of noisy tokens and the generated tokens. This enables one-step generation by inputting the sequence of noisy tokens into the distilled model. Moreover, this deep synergy between AR and FM allows for flexible use of additional generation steps improve data quality without changing the model (see Sec. 3.3h. Note that the training of DD does not need the training data of the original AR model. This makes DD more practical as training data is often not released especially for SOTA LLMs.
Mcasured NVIDIA A10--80G GPU.
2

---

Published as a conference paper at ICLR 2025
VAR
LlamaGen
10
Du
Di
Du- Pooreee VAX
DO the mntedd Liantties
Pormsie VAR
Peetroadd Liaerties
Llanadin dkp
waster"
10
10
20
40
60
100
1.0
140
10
10a
162
10*
Time Image/ms
Time per Imageltzs
Figure 3: Comparison of DD models, pre-trained models, and other acceleration methods for pre- trained models. DD achieves significant speedup compared to pre-trained models with comparable performance. In contrast, other methods' performance degrades quickly as inference time decreases.
token needs
1 group needs
step
Austoregretsivve Decoding
Parallel Decoding
Distilled Decoding (Ours)
Figure 4: High-level comparison between our Distilled Decoding (DD) and prior work. To
generate a sequence of tokens q; (a) the vanilla AR model generates loken-by-token thus being slow; (b) parallel decoding generates multiple tokens in parallel (Sec. [4I)) which fundamentally cannot match the generated distribution the original AR model with one-step generation (see Sec. 3.1: (c) our maps noise tokens E; from Gaussian distribution the whole sequence of generated tokens directly in one slep and is guaranteed that (in the optimal case) the distribution of generated tokens matches that of the original AR model.
As the first work in this series, we focus on image AR models. We validate the effectiveness of DD on the latest and SOTA image AR models: VAR (Tian al. 2024) and LlamaGen (Sun al., 2024). Our key contributions are:
We identify the fundamental limitations of existing methods that prevent them from achieving few-step sampling in AR models.
We introduce
distill pre-trained AR models for few-step sampling.
For the first time, we demonstrate the feasibility of 1-step sampling with SOTA image AR mod- els. On ImageNet, DD reduces the sampling of VAR from 10 steps to 1-step (6.3 x speed-up) with an acceptable increase in FID from 4.19 to 9.96; for LlamaGen, DD cuts sampling from 256 steps to I (217.8x speed-up) with comparable FID increase from 4. to 11.35. In both cases, baseline methods completely fail on step generation and achieve FID > 100. For texi-to- image generation with LlamaGen LAION-COCO dataset. DD is able reduce the sampling steps from 256 to 2 (92.9x x speed-up) with minimal FID increase from 25.70 to 28.95. DD also supports more steps for better image quality (Fig. 3). See Figs. I andd for visualization.
This work challenges the assumption that AR models must be slow. We hope it paves the way for efficient AR models and inspires research into one-step AR models in other areas, such as text generation, where the task is more challenging due to the higher number of steps.
2 PRELIMINARIES
In this section, we introduce the preliminaries required to understand DD.
2.1 AR MODELS
Given a random variable arranged in a sequence of n tokens (qi, 4.)) AR models learn the conditional distribution tokens given all previous ones: p(gila<s) p(g: (4-1 qi ... v. The data likelihood is given by p(2) = I p(q: (4ss). At generation. AR models sample tokens one by one using the learned conditional distribution for each token, which therefore slow.
2.2 IMAGE AR MODELS
Image tokenizer. To apply AR models to images, we need to represent continuous images sequence of discrete tokens. Early works of image AR models operate on quantized pixels (Van den Oord et al. 2016 Chen et al. 2018). Later works propose the vector quantization (VQ) method, uti- lizing a encoder E, a quantizer Q, and a decoder D to quantize and reconstruct images. Specifically, after [E, Q. D) is fully trained, the encoder will transform the original image E 3 x into
3

---

Published as a conference paper at ICLR 2025
more compact latent space: Z E(x), where Z = (21, 22, 2bxw } E RC consists of x w embeddings, each has a dimension of C. Then, the quantizer will look up the closest token qi in the codebook V (cI, C2. ... .vv) E RV'C for each embedding 2,. AR works on the quantized sequence Z = {qI, 42, ... -[hw Finally, one can use the decoder to reconstruct the image from Z2)))
)- A distance loss [ x)) is used in training for accurate recon- struction. The VQ scheme is used in many popular image AR models (Li et al. 2023 Chang et al. 2022 Tian et al. 2024] Sun et al.) 2024 Li et al. 2024a).
the quantized sequence:
The order of tokens. Another important design choice of image AR models is the order of to- kens. A typical and straightforward method of ordering tokens is following the raster order of the embeddings 2,, such as LlamaGen (Sun et al., (024). Random order is more general option (Chang et al. 2022 Li al. 2023, 2024a). The SOTA method VAR (Tian et al., (024) proposes a novel method where the tokens are arranged according to the resolution. Specifically, give the latent Z E Cxxxx VAR down-samples
a series of resolutions ((h), w1), (h2. W2)..... ... (hx: "x) )), where h4 < hj,22 w; Wj for < j. Then, VAR arranges the augmented tokens from these la- tents in the order of resolution. Tokens from the same resolution are sampled simultancously. This more natural and auman-aligned next scale prediction approach enables VAR to achieve excellent performance. However, still needs 10 steps to generate a 256 x266 image, making it inefficient.
2.3 FLOw MATCHING
Given two random distribution To(x), Ti (2) with I E R4, flow matching (FM) @Liu et al. 2022
Lipman et al. 2022) constructs probability path which connects the two distributions. Setting up a continuous timestep axis and putting the two distribution at = 0 and t = respectively, such probability paths can be viewed as an ordinary differential equation (ODE) trajectory. Specifically, flow matching defines velocity field (ax, () at any timestep l, given by: 0:p(zo 31, (xp x1, =))
= V(x, = Bzo. To 1 (2)
(1) x1 = T (Z1) and
x),
Öt where To,1 (20, æ1) is any joint distribution that satisfies both f I To, I z )drı = To( and (0x,, y, is a trivariate bijection function for x and y, which means that at any give timestep i, knowing any two of the so, y will determine the third one. (p(, y, t) should also satisfy the boundary conditions: pp(0, Z1, 0) = Io. (o(o 11. ø1.
The flow matching ODE has marginal preserving property. Given 20 ~ To. we can obtain ze by stepping along the ODE trajectory: V₁ = xo + J ()dr, It can be proved that the distribution of F4 is equal to the distribution of (2 21. (Liu et al./ 2022 Lipman et al. 2022): (xx)) x <vp(<70 (1, ()), where To, ~ To. (4ro, When we step to / = 1, we will get sample from another distribution æ1- In practice, T1 is usually set as the target distribution we want to sample from, while To is set as simple prior distribution, e.g., Gaussian distribution. Then flow matching enables a generative process from the prior to the target.
3 METHOD
3.1 TRAINING FEW-STEP AR MODELS IS NON-TRIVIAL
In this section, we explain why few-step generation is challenging for AR models and why existing approaches fundamentally cannot achieve it. To finish the sampling process of the whole token sequence (q1," Qn) in as few steps possible, each step should generate as many tokens as possible. Assume our goal is to generate a set of next tokens (qK+1, ... (m) based on a sub-
sequence (qı.-.., , 9 as prefix in one step, there are several straightforward ideas to solve this problem. We discuss each method and the reason why they do not work as below:
(1) Train a neural network with the prefix sequence as input and output the subsequent tokens in one run. Actually, the information in the prefix (4o. ... <4k) is not enough to deterministically specify the (gk+1.-. .. because there are many feasible possibilities for the subsequent tokens that can fit the prefix, leaving a enn-to-many mapping for the neural network to learn. For example, we can mask the face portion of a portrait and then select another face from many choices to replace it. In this case, the neural network can only learn an average of all possibilities and output blurry im- age tokens, like the one in MAE (He et al. 2022). At the stage where the prefix token is insufficient to constrain the subsequent tokens (e.g., at the beginning of the AR generation process), feasible choices are even much more, leading to poor generation quality. A further potential solution is to select the choice with the highest probability (202T)). However, such a method will destroy the distribution completely by collapsing into the most likely mode, which is impractical for image generation.
the output target of the network (e.g., Song et al.
4

---

Published as a conference paper at ICLR 2025
Prot, over the codebook
Gaussian distribution
Autoregressive Decoding
0.94
6. C5 62
Mixture of Dirac delta distriutiioss
Figure 5: AR flow matching. Given all previous tokens, the teacher AR model gives a probability vector for the next token, which defines mixture of Dirac delta distributions over all tokens in the codebook. We then construct a deterministic mapping between the Gaussian distribution and the Dirac delta distribution with flow matching. The next noise token 6 is sampled from the Gaussian distribution, and its corresponding token in the codebook becomes the next token qt-
(2) Model the distribution of all subsequent tokens. The neural network can be trained to output the probability of each subsequent token given the prefix as the input. This approach avoids the challenges caused by the one-to- one-to-many mapping and used in mask-based AR models (Chang et al., 2022 Li et al., 2023] 2024a) and VAR (Tian et al. (024). In this case, the sampling processes of each token are independent from each other, introducing a gap between the modeled distribution - p(g:lge,-.. [ga. . Q1) and the ground truth p(gm. - (9k..... .Q1). For image AR models, the gap is acceptable when m - is small. However, when the number of new tokens is large, the gap will increase exponentially, making the performance much worse. Consider the case where the model is trained to learn to sample all tokens in one run (which is our goal), In this case, the cross-entropy loss of every token is calculated separately. The final objective can be written
L 1 Σ
Pijklogōejk
(2)
where V is the codebook size, N is the dataset size, Pijk is one-hot vector along the third dimen- sion given any (). indicating the ground-trutt probability distribution, and Payk is the modeled distribution. In the following proposition, we give the optimal solution of pajk- Proposition 3.1. The optimal solution for Eq. is po-jk Ei=; Pish
This solution equals to occurrence frequency of the k-th token at the j-th position within the dataset. The proof can be found in App. V Based this proposition, consider toy case where the dataset only contains 2 data samples: D = {(0, 0), (1, 1)). It is easy to find the optimal one-step sam- pling distribution is uniform distribution among {00, 0), (1, 1), (0, 1), 0)3. which is incorrect. It indicates that the widely used method which predicts the next group of tokens is fundamentally impossible apply for few-step generation. We will see such experimental evidence in Sec. 5
Additionally, attempting to model the joint distribution by outputting the probability distribution over all possible next m- k tokens is also impractical because of the large number - of possible values, where V is the codebook size, which is typically several thousands or even tens of thousands.
3.2 CONSTRUCTING DETERMINISTIC TRAJECTORIES FROM GAUSSIAN TO DATA TIIROUGHI AR FLOW MATCHING
As discussed in section Sec. 3.1 training model to generate all the tokens in single run impos- sible due to the issue of one- to--to--mmn many mapping. Therefore, constructing one-to-one mapping is the key for training a model to generate more tokens simultaneously. The process illustrated in Fig.
Construct a mapping for single token. Consider the sampling process of a single token q; E RCC given (q), ... 46-1). Inspired by the knowledge distillation method for diffusion model (Luhman & Luhman 2021; [alimaan & Ho, 2022] Song et al. 2023) Song & Dhariwal, 2023 Kim et al.] 2023) which map noise to data, we propose to use noise token as additional information to deter- mine this single token. Specifically, we sample noise token from a prior distribution and map it to the generated token. We hope that (1) every noise token will be transferred to a deterministic data token in the codebook, and (2) the distribution of generated token equals to po (g: 1,,- (0) given by the pre-trained AR model. We propose to use flow matching as the desired mapping for the single token. We operate on the continuous embedding space E of each token given by the codebook V = (c1.-.. ).)= E VVC, Since the distribution of the next token is discrete probabil- ity distribution among all tokens in the codebook:
P(g: cy) Pj. where pj 2 0, DY Pj 1, it can also be viewed as a weighted sum of point distributions in the continuous embedding space: p(g:) Σj= p,5(q: - <)), where 5(-) is the Dirac function. Denoting ne the marginal distribu- set this weighted sum of point distributions the target distribution лi and
tion at timestep
apply a standard Gaussian distribution as the source distribution To. We further choose the linear interpolation used in Liu et al. (2022) as the perturb function: sp(20, 2), (1 1)zn + tzj. Then
5

---

Published as a conference paper at ICLR 2025
the velocity field in the flow matching framework is given kep)2
V(z, t) = Σj Py(c x)e (1
(3)
(a-tep)
(1-t) Pye (1-1)²
In this way. we construct mapping from a noise token embedding q E RC to the generated token embedding qi with the ODE dr (xx,t)d while keeping the distribution of qi unchanged. We denote this mapping as
9.
(4)
In practice, we use numerical solvers (Lu et al. 2022a b) to solve the ODE process, qi will not have an exact match in the codebook. To tackle that, we use the token closest to qi in the codebook.
Construct the whole trajectory along the AR generation process. After constructing mapping from (q).-.. (;) to (q), ... : 1; qi), we can extend such mapping to situations with arbi- trary length subsequent noise tokens (41. 9i-1: Eg. ... ,Em) via an iterative way. After gener- ating the current token from noise 6,, we can update the sequence with (q),-.. , 9i E++1, 6x,) and get the conditional probability p(g+11 19.................... .91). Then the same method can be applied to transfer the next noise token 4++1 to data token 4++1, until all the subsequent noise tokens are mapped to the corresponding generated data. This process imposes no constraints on the length of the prefix, so we can start from a pure noise sequence (eI, (E1, en). In this way, we construct an AR trajectory {x+)== from pure noise to the final data using flow matching, given as:
X; (Q1 (,, (6)
(5)
The recurrence relation is given by Eq.
3.3 DISTILLING AR ALONG THE TRAJECTORY
The AR trajectory transfer the pure noise sequence progressively to the final data sequence, thus is suitable for neural network to learn. To enable trade-off between sample quality and sample step, we train the model to predict the final data not only at the beginning of the trajectory but also at intermediate points. Based on this, we first clarify the notation, discuss the model parameterization and training loss, and finally introduce the overall workflow of distillation and sampling.
Notation. Suppose X = (II,,.. , .In) and Y = (y1....,,2))) .n.) are two arbitrary sequence. We denote X[: t] as a sub-sequence with the first tokens in X and X + 1 :] is the rest part: X|: t = (x1, .rt), X[t + 1: (#+41, ... .r, Additionally, we define the Concat operation as concatenating two sequences: Concat(X, Y) = (I1, In, U1, B.)).
Model parameterization. The model takes an intermediate value X, = (91.-.. 9t-1, Cn) of the trajectory and the position / as input, and output the final data Xn+1 E B corresponding to the X,- Suppose we have a neural network So and denote its output as fa(X) = (Jor,... Sen) )- Since the first t - tokens of X, is already correct tokens, they can be directly used the first tokens of the predicted results. We thus parameterize the model output as: FX,,,, f) Concat(X,[[: t - 1], fo (X.) :]) (q).-.. It.1.
fee...... fon)
(6)
Training loss. We train the model by minimizing the distance between its output and the target data sequence, which gives the following loss:
- = (X) X)++1)
(7)
Denote 41. } as the from which timestep / can be sampled, where 4 E [1, n] and 4 1, and X, = Concal (x,o: 1- 1]. [1 :)), where X,, is the final generated data by the pre-trained AR model corresponding to X,. The expectation of Eq. (7) taken with respect t Uniform ) / and X, ~ (0, f). X(-) # 0 is a step-wise weighted function. d(-,-) is any distance function that satisfies d(x.g) v) 2 and d(a, y) = O if and only x =
Overall distillation workflow (Fig, 6). Our overall workflow consists of two parts: generating the training set and training the model. First, we iteratively draw noises from standard Gaussian distri- bution and calculate the deterministic generated data through the method in Sec. [.2] to construct a dataset of noise-data pairs (see Alg. []. Then we train the model with Eq. 77) using the dataset (see
Alg. 2
In this case, the noise-data pair (X1, X,)) is directly drawn from the dataset.
Note that this is different from Python indexing.

---

Published as a conference paper at ICLR 2025
Training
Generation
8-step
+
1-step
X
X
2-step
X2 @
Legend: Teacher AR model
X
Our model
Figure 6: The training and generation workflow of DD. Given X1 with noise tokens cG, the whole tra- jectory X1,... .xs consists of data tokens 9i and noise tokens E, is uniquely determined (Sec. 3.2).
Assuming the timesteps are set to {41 = 1, l2 = 3). During training (Sec. (.3.. we train DD model to reconstruct Xs given X1 or X3 as input. The DD will then have the capability of jumping from X1 and X3 to any point in the later trajectory (c.g.. X1 to any of (X2.... ..,)) During generation (Sec. 3.3), we can either do 1-step (X1 + Xs) or 2-step generation (X1 - X3 + Xs). Addi- tionally, we can do generation with more steps by incorporating the teacher AR model in part of the generation process, such as 3-step generation X1 - X2 + X3- - Xs where X₂ + Xg utilizes the AR model and other steps use the DD model.
With one trained DD model using the above workflow, we can flexibly ad- just the trade-off between sample quality and the number of generation timesteps during inference (Alg. }). Specifically, any subset of the training timesteps {fk).... .12,) where tx, = 1 can be cho- as timestep path from noise data. We can choose only the first timestep {t1} to do one-step generation, or leverage more timesteps to improve sample quality. Utilizing the AR property of the trajectory, jumping from step tk, to the41 can be implemented by predicting all tokens at timestep la, and only keeping the ones before 1A,11"
Generation (Fig. 6).
utilize the pre-trained AR model to achieve more flexibility in the trade-off between sample quality and timestep. Specifically, we can use the pre-trained AR model to generate from any position along the sampling process until reaching the next trained timestep for few-step sampling. Alg. 4 is an example of 2 + l2g - t, step sampling with the pre-trained model inserted into the original two-step sampling process. We use the DD model on the first step, re- re-generate the last fop t. tokens with the pre- rre-trained AR model, and then apply the DD model for the second step.
Additionally,
Discussions. As a deep synergy between AR and flow matching, DD possesses interesting prop- erties that none of the flow matching (which is closely related to diffusion models) and AR have. Compared to AR, DD has the capability of adjusting the number of generation steps. Compared to flow matching (and diffusion models), DD uses AR to construct the trajectories, and therefore, DD has an easy way to jump back to any point in the trajectory by replacing the last few generated to- kens with the original noise tokens, which we utilize in the sampling procedure above. This property enables DD to keep the trajectory unchanged when sampling with multi-steps, in contrast to other intermediate-to- -onal distillation methods for diffusion models (Song et al., 2023 Song & Dhariwal 2023) which cannot maintain the trajectory during multi-step sampling.
Another point mention is that the DD does not require the training data of the original AR model, as opposed to some related work (Li et al., 2024b) [Song et al. (003). This makes DD more practical in cases where the training data of the pre-trained AR models not released.
4 RELATED WORK
4.1 DECREASING THE NUMBER OF SAMPLING STEPS OF AR MODELS
Image generation. (1) based image AR methods (Chang et al. 2022 Li et al. 2023 2024a)
adopt a random masking ratio when training, allowing them directly perform trade-off between performance and number of steps. Specifically, these models are trained to predict an arbitrary number of tokens given some other arbitrary number of tokens. By increasing the number of tokens generated in each step, they can sample faster at the cost of loss in generation quality. However, the fewest number of generation steps evaluated these works is only 8 and we show in Sec. 3.1 that they fundamentally cannot support very few steps (c.g., one or two steps). (2) For causal- transformer- -assed methods, how to reduce their number of steps still unknown.
Text generation (large language models). There are already many works focusing on speeding up the generation of L.LMs (Zhou et al., 2024h. (1) Speculative decoding first generates a draft of multiple following tokens and then applies the target LLM to verify these tokens in parallel. The draft tokens can either be generated (a) sequentially by another (small) AR model (Chen et al.
2023 Leviathan et al. 2023 Li et al. 2024b) or (b) in parallel by specially trained models or
7

---

Published as a conference paper at ICLR 2025
heads (Cai et al. 2024 Gloeckle al., 2024 Xia et al., 2023). Method (a) does not reduce the number of generation steps and therefore is irrelevant to our few-step generation goal. Method (b) fundamentally cannot match the distribution of the target AR model (Sec. B.I. As result, even when generating a small number of tokens at a time, they rely on a verification step to ensure correctness, let alone one-step generation. (2) In contrast token-level parallelism exploited in speculative decoding, content-based parallelism is another paradigm that exploit the weak relation between different parts of the generated content (Ning et al.) 2024a Liu al., 2024 Jin al. 2024).
These methods prompt or train the LLM to generate an answer outline, enabling the LLM to generate independent points in the outline parallel. However, these methods change the output distribution
of the original AR model. (3) Conceptually similar to our methods, CLLMs (Kou et al., 2024)
utilize Jacobi decoding trajectory (Song et al. [021] (aantllli et al. 2023) and train student model to conduct one-step sampling along the trajectory. However, CLLMMs only support greedy decoding, which deviates from the (non-deterministic) generated distribution of the original AR model.
In contrast to all these methods, DD supports one-step sampling while theoretically having the po- tential to match the output distribution of the original AR model.
4.2 DIFFUSION DISTILLATION
Similar to AR models, diffusion models (Sohl-Dickstein et al. 2015 Ho et al. 2020 Song et al., 2020) also suffer from a large number of sampling steps. Knowledge distillation is a promising approach to address this problem and has been widely studied. These methods leverage the trajec- tory constructed by the pre-trained diffusion model, and train neural networks to directly predict the value multiple steps ahead in the trajectory, rather than predicting just one step ahead as in the pre-trained model. One key difference between various knowledge distillation methods lies in how the starting and ending points of multi-step skipping are selected. The earliest method (Luhman & Luhman 202T) trains the model to skip the entire trajectory (i.e., predicting the final data directly from the intiaal Gaussian noise), enabling one-step sampling. PD (Salimans & Ho 2022 progres-
sively merges two adjacent steps into one, which makes the optimization task smoother and provides
a trade-off between steps and quality. CM (Song et al. 2023 [Song & Dhariwal 2023) skips from
any intermediate points along the trajectory to the final data. CTM (Kim et al., 2023) proposes a
more general framework which enables the transfer between any two points in the trajectory.
Unlike diffusson models, AR models do not come with deterministic trajectories, and therefore, doing distillation on AR is not straightforward. One of our key innovations is the construction of de- terministic trajectories out from a pre-trained AR (Sec. (.2), thereby making knowledge distillation for AR possible. In this paper, we only explored a simple knowledge distillation paradigm, and we hope that this work opens the door for more knowledge distillation paradigms for AR in the future.
See App. F for more discussions on related works (Li et al. 2024a
Yin et al. 2024
EXPERIMENTS
In this section, we use DD to accelerate pre-trained image AR models and demonstrate its effective- ness. More details are in App. Ablation studies are in App. E.2
5.1 SETUP
Training. For pre-trained AR models, we choose VAR TTin et al. 2024) and LlamaGen (Sun al. (024) for the following reasons: (1) Both methods are recently released, very popular, and have achieved excellent performance (c.g., LlamaGen claims to beat diffusion models on image
generation). (2) Their token sequences are defined very differently (Sec. 2.2): : VAR uses images of different resolutions to construct the sequence, while LlamaGen adopts traditional method where tokens are latent pixels and arranged in raster order. We want to test the universality of DD across different token sequence definitions. (3) Their number of generation steps vary significantly: VAR has 10 steps while LlamaGen uses 256 steps. We want to test whether DD is able to achieve few-step generation matter whether the AR model has small or large number of steps. (4) They both provide different sizes of pre-trained models for us to study how DD scales with model sizes.
Following these works, we choose ImageNet 256 x 225 as the main benchmark. We set the number of available timesteps as 2 and use [1,6). {1, 81} as (tk, tkp } for VAR, LlamaGen, respectively. Results show that DD can compress both models to or 2 steps with comparable quality.
We further evaluate DD on eoxt-to-image models to assess its capability in handling large-scall scale tasks. We select the 1 text-to-image model from LlamaGien as base model, which uses 256 steps
8

---

Published as a conference paper at ICLR 2025
Table 1; Generative performance on class-conditional ImageNet-256. "#Step" indicates the number of model inference to generate one image. "Time" is the wall-time of generating one image in the steady state. Results with are from
Tian et al. 22024). See Tab. 4 for results more model sizes.
Type
Model
FID
1st
Pret
Reet
#Para
#Step
Time
GAN!
StyleGan-X XL. (Sauer et al.
2022
2.30
265.1
0.78
0.53
166M
0.3
DifL.
ADM (Dhariwal Nichol
2021
10.94
101.0
0.69
0.63
554M
250
168
Difr.
LDM-4G IRombach el
2022
3.60
247.7
400M
250
Diff.
DIT-L/2 Pochles & Xie
2023
5.02
167.2
0.75
0.57
458M
250
31
Dift.
L-DiT-7B Peebles & Xie 2023
2.28
316.2
0.83
0.58
7.0B
250
245
Mask.
MaskGTT
Chang et al.
2022
6.18
182.1
0.80
0.51
227M
8
0.5
ARt
VQVAE-21 (Rzaaii et al 2019
31.11
45
0.36
0.57
13.5B
5120
AR!
VQGAN Esser ct al
2021
15.78
74.3
1.48
256
24
AR!
VITVQ Yu ct al. 2021
4.17
175.1
1.78
1024
>-2
ART
RQTran. 00 al
2022
7,55
134.0
3.8B
68
21
AR
VAR-d16 Tian ct al.
2024
4.19
230.2
0.84
0.48
310M
10
0.133
AR
LlumaGen-L Sun et al.
2024
4.11
283.5
0.85
0.48
343M
256
5.01
Baseline
VAR-skip-/
9.52
178.9
0.68
0.54
310M
0.113
Baseline
VAR-skip-2
40.09
56.8
0.46
0.50
310M
0.098
Baseline
VAR-onestep*
157.5
Bascline
LlamaGen-skip-100
19.14
80.39
0.42
0.43
343M
150
2,94
Buseline
LlamaGen-skip-156
80.72
12.13
0.17
0.20
343M
100
1.95
Baseline
JamaCeen-nnestee"*
220.2
Ours
VAR-dI6-DD
9,94
193.6
0.80
0.37
327M
0.021 (6.3x)
Ours
VAR-d16-DD
7.82
197.0
0.80
0.41
327M
2
0.036 (3.7x)
Ours
LJamaGen-IL-33
11.35
193.6
0.81
0.30
326M
0.023 (217.8x)
Ours
LlamaGon-L-30
7.58
237.5
0.84
0.37
326M
0.043 (116.5x)
for generation. For evaluation, we generate images based 5k unseen prompts from the LAION- COCO dataset and report the FID between the generated images and their corresponding real images. All other settings are consistent with those used for the ImageNet dataset.
We additionally use Alg. Sec. [.33 5.3 for better quality with more generation steps. We use series of staring timesteps for the pre-trained model to get smooth trade-off between generation cost and quality.
Generation. We apply Alg. 3 with and 2 steps as our main results in Sec. 5.2
Baselines. Since there is no existing method for decreasing the generation steps of visual AR with causal transformers (Sec. 4. we design two baselines: (1) Directly skipping the last several steps, denoted as skip-n where is the number of skipped steps. and (2) predicting the distribution of all tokens in one step with the optimal solution in Prop. 3.1] denoted as onestep We also compare with the base pre-trained AR models and other generative models for comprehensiveness.
5.2 MAIN RESULTS
The main results of DD are shown at Tab. 1 The key takeaways are:
DD enable few-step generation of AR models. By comparing the required generation step and time between the pre-trained models (VAR and LlamaGen) and our distilled models (VAR-DD and LlamaGen-DD), we can see that DD compresses the step and accelerates the pre-trained AR by very impressive ratio (6.3x on VAR and 217.8x on LlamaGen). Notably, DD decreases the generation step and time of LlamaGen by two orders.
Baselines do not work for few-step generation. From Tab. | we can see that DD steadily surpasses the two types of baselines. For skip baseline, we find that the performance rapidly declines as the number of skipped steps increases. The onestep* indeed does not work as expected in Sec. 3.1 due to the lack of correlation between tokens.
DD does not sacrifice quality too much. While achieving significant speedup, DD does not sacrifice the generation quality too much. For VAR, the FID increases of one-step and two-step generation are smaller than 6 and 4, respectively. For LlammGien which has more original steps, the FID increase around 3.5 for two-step generation. Note that as DD learns the mapping given by the pre- pre-trained AR, its performance is bounded by the pre-trained model. Such results have already outperformed many other popular methods like ADM (Dhariwal & Nichol 2021 with much faster generation speed.
DD scales well with different model sizes. We experiment with two different sizes of LlamaGen and three different sizes of VAR (Tab. 4). DD always achieves reasonable sample quality across different model sizes. In addition, for each model family, with larger model sizes, DD can achieve better FID. These results suggest that DD works and scales well with different model sizes.

---

Published as a conference paper at ICLR 2025
Table 2: Generation quality of involving the pre-trained AR model when sampling. The notation pre-trained-n-mm means that the pre-trained AR model is used to re-generate the n-th to 1-th tokens in the sequence generated by the first step of DD.
Type
Model
FID4
ISt
Pref
Reet
#Para
#Step
Time
AR
VAR (Tian et al. 2024
4.19
230.2
0.84
0.48
310M
10
0.133
AR
LlamaGinn (Sun et al. 2024
4.11
283.5
0.865
0.48
343M
256
5.01
Ours
VAR-pre-truinede
5.03
242.8
0.84
0.45
327M
6
0.090 (1.5x)
Ours
-rr-ie--
5,47
230.5
0.84
0.43
327M
4
0.062 x2.1x)
Ours
6,54
210.8
0.83
0.42
327M
3
0,045 (2.6x)
Ours
I JamaGien-pre-traineed -ra-------------1 1-81
5.71
238.6
0.83
0.43
326M
81
1.725 (2.9x)
Ours
6,20
233.8
0.83
0.41
326M
42
0.880 (5.7x)
Ours
LlamaGen-pre (wwp-re---ddd-61---
6.76
231.4
0.83
0.40
326M
0.447 (11.2x)
Table 3: Generation results of DD on text-to-image task.
Type
Model
FID
#Param
#Step
Time
AR
LlamaGen
25.70
775M
256
7.90
Ours
LlamaGen- -D
36.09
756M
0.052 (151.9x) x)
Ours
LlamaGen--DD
28.95
756M
2
0.085 (92.9x)
DD allows users to choose the desired trade-off between quality and speed. From Tab. we can see that generation with two steps has better quality than generation with one step, indicating that DD offers trade-off between quality and step, which is a property that AR models using causal transformer (such as VAR and LlamaGen) do not have.
5.3 GENERATION INVOLVING THE PRE-TRAINED MODEL
In this section, we test the method in Alg. 4 which utilizes both the distilled model from DD and the pre-trained AR model to provide more trade-off points between quality and step. As discussed in Sec. pre-trained-ty. -ttt- Results are shown at Tab. [ We can see this generation method offers a flexible and smooth trade-off between quality and step. For example, for VAR, by totally replacing the first step with the pre-trained model, DD reaches a FID of 5.03 which is very close to the pre-trained model, while achieving 1.5x speedup. For LlamaGen, DD achieves FID 6.76 with 11.2x speedup.
3.3 we use the pre-trained AR model to re-generate token from t, to &22 - 1, denoted as
5.4 TEXT-TO-IMAGE GENERATION
In this section, we report the results of DD on text-to-image generation in Tab. We can that DD achieves performance comparable to the pre-trained model (i.e., 25.70 v.s. 28.95) with only 2 sampling steps. Generated examples are demonstrated in Fig. 2 and App.
6 LIMITATIONS AND FUTURE WORK
Training few-step AR models without teachers. In this work, we distill pre-trained teacher AR models to support sew-step sampling. The sample quality is therefore bounded by the pre-trained AR model. It would be interesting to explore if it is possible to eliminate the need for a teacher, which offers more flexibility and potentially can lead to better sample quality. As discussed in Sec. [42) the trajectory construction in DD opens up the opportunity to apply other diffusion distillation ap- proaches to AR models. One potential direction is to apply the teacher-free consistency training approaches (Song al. 2023 Song & Dhariwal 2023) on AR models.
Applications on LLMs. Theoretically, DD can be directly applied to LLMs. However, different from visual AR models, the codebook of LLMs is much larger in both size and embedding dimen- sions. Additionally, the sequence length of LLMS may be much longer. Therefore, applying DD on LLMs introduces new challenges be solved.
Questioning the fundamental trade-off between inference cost and performance. It was re- cently believed that for AR models scaling up the inference compute and number of generation steps is important for getting better performance, as exemplified by various LLM prompting tech- niques (Wei et al. 2022 Nori et al. 2023 Ning et al. 2024b) Snell al. 2024 and the inference scaling law presented in OpenAl ol (OpenAl 2024). However, as our experiments demonstrated, it is possible to significantly reduce the inference compute and generation steps without losing too much fidelity, at least for current image AR models. This suggests that while the inference compute is important, current models might be wasting compute in the wrong place. It would be interesting to study (1) where the optimal trade-off between inference cost and performance is, (2) how much current models are away from that optimal trade-off, and (3) how to modify current models or design new models to reach that optimal trade-off
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
The authors would like to thank Sergey Yekhanin and Arturs Backurs of Microsoft Research for their support and suggestions of this work, Qian Chen for his help with experiments, and the anonymous reviewers for their valuable feedback and suggestions.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.000 08774,, 2023.
Tom B Brown,
Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason Lee, Deming Chen, and Tri Dao. Medusa: Simple IIm inference acceleration framework with multiple decoding heads. arXiv preprint ar arXiv:2401. 10774 2024.
Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325, 2022.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01311, 2023.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved au- toregressive generative model. In International conference machine learning, pp. 872. PMLR, 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 12873-12883, 2021.
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Syn- naeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404. 19737, 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Shuowei Jin, Yongji Wu, Haizhong Zheng, Qingzhao Zhang, Matthew Lentz, Z Morley Mao, Atul Prakash, Feng Qian, and Danyang Zhuo. Adaptive skeleton graph decoding. arXiv preprint arXiv:2402.1 12280 2024.
Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang I Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning proba- bility flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279. 2023.
Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Cllms: Consistency large language models. arXiv preprint urXiv:2403.00835, 2024.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-11533 2022.

---

Published as a conference paper at ICLR 2025
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, Pp. 19274-19288 PMLR, 2023.
Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang. Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2142-2152, 2023.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image
generation without vector quantization. arXiv preprint arXiv:2406. 1838, 2024a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint urXiv22401.15077 2024b.
Yaron Lipman, Ricky TQ Chen, Heli Ben- Hem-Hamu,, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXi::2210.02777, 2022.
Mingdao Liu, Aohan Zeng, Bowen Wang, Peng Zhang, Jie Tang, and Yuxiao Dong. Apar: Lims can do auto-parallel auto-regressive decoding. arXiv preprint arXiv:2401.06760 2024.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0.00 000033 2022.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffiusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX19:221..01095, 2022b.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv: 1.Xiv:2211..2.::: 003388 2021.
Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang. Huazhong Yang, and Yu Wang. Skeleton-of- thought: Prompting Ilms for efficient parallel generation. In The Twelfth International Conference on Learning Representations, 2024a.
Xuefei Ning. Zifu Wang, Shiyao Li, Zinan Lin, Peiran Yao, Tianyu Fu, Matthew B Blaschko, Guo- hao Dai, Huazhong Yang, and Yu Wang. Can Ilms learn by teaching? a preliminary study. arXiv preprint arXiv:2406. I /4629, 2024b.
Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. Can generalist foundation models outcompete ppecial-purposs tuning? case study in medicine. arXiv preprint arXiv:2311.16455 2023.
OpenAI, Sep
2024.
URL
https://opents ctw.i.cooomoo/ommmmmm.omo / ndex
Learning-ton to- reason with 11ms
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205, 2023.
Alec Radford. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high- high-fidelity images with vq-vae-2 Advances in neural information processing systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint AxXiv:222.000512 2022.
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Ric- cardo Marin, and Emanuele Rodolà. Accelerating transformer inference for translation via paral- lel decoding. arXiv preprint arXiv:2305. 10427, 2023.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally be more effective than scaling model parameters. arXiv preprint arXiv:2408.085 03314, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265, PMLR, 2015.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310. 14189, 2023.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020.
Yang Song, Chenlin Meng, Renjie Liao, and Stefano Ermon. Accelerating feedforward computation via parallel nonlinear equation solving. In International Conference on Machine Learning, pp. 9791-9800. PMLR, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autore- gressive models. Advances in Neural Information Processing Systems, 31, 2018.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06555, 2024.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.022 02905, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0m 092888, 2023b.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. Advances neural information processing systems, 29, 2016.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 248377 2022.
Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decod- ing: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the A Association for Computational Linguistics: EMNLP 2023, pp. 3909-3925, 2023.
13

---

Published as a conference paper at ICLR 2025
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2024.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint 2..0:221000062,, 2021.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. survey on efficient inference for large language models. arXiv preprint arXiv:2404.14220 2024.
A PROOF OF PROP. 3.1
Proof. Since all positions in the sequence are symmetrical, we can just consider one certain position The loss on this position is
N
V
C2 1
Pijklogpajk
(8)
N
i=1 k=
Due to the normalization constraints of probability, we have
Σ po5k =
(9)
Considering the Lagrange Multiplier Method, the Lagrange function given
1
F(peyi. .PoJV, A)
Pizzlogbejk
pajk 1)
(10)
N
The optimal poya and the corresponding À satisfy
0F
Σe Piyk
A= -,k=,,,,,...
(11)
Opez
Np0sk
OF
Σ Posk 1 =
(12)
0X
k=1
By solving the above equations, we can get the final solution
ΣN 1i=1 Σ k=
Piyk
A
(13)
N
N
ii Pijk
D0-3k
(14)
N
B ALGORITHM PSEUDOCODE
In this section, we provide the algorithm pseudocode of training data generation (Alg. training (Alg. 2 and sampling (Alg. Bnd Alg.
C MODEL ARCHITECTURE DESIGN
Since our method does not alter the AR property nature of the generation task, we can use the same architecture the pre- pre-trained model while slightly modify several modules as discussed below. Aside from these adjusted modules, all other modules inherit the weights from the teacher model for quicker convergence.
14

---

Published as a conference paper at ICLR 2025
Algorithm 1 Generate dataset
Algorithm 2 Training
Require:
Require: :Generated dataset with noise-data pairs D, the pro- trained AR model 8".
0 The pre-trained AR model
Generation Process
Hyper-parameter
Available (4,, .1,,),.,..). learning rate v/.
timesteps for training
1: D 4-2
for 1, 1,... do
Training Process
3: X1 f- (61, ;c cn) (0,,
1: 0 initialized from 0%
4:
for tort= = 1,... do
2: for i 1, Iter do
5:
@t FM (M, poo end for
Sample (X1, Xn) ~ D. (ta,..,ttt) ,t
6:
4: Xt + Concat(Xa|::
Xi(t
7:
X, f qu)
X, f t)
8:
DU ((X1, Xn)]
L d(Xn.. Xn)
9:
end for
f + VoL
10:
return D
end for 9: return 0
Algorithm 3 Sampling
Algorithm 4 Sampling with the pre- pre-trained AR model
Require: The distilled few-step model 0, sam- Require:
The distilled few-step model 0, the pre- pre-trained AR model 8º, sampling timesteps (ta, 1, tug J. start- ing timestep of pre-trained model fxy < to < txg.
pling timesteps (tay.".. .tay }.
Sampling Process
1: X1 < (EI, ,,n) NN(0,)) X X1
Sampling Process
2: for in (tk). tx do
I: X1 (EL.... Fn) NNO, X X1
3: X Concat(X 11, (t:))
2: X Concat X1: t2g Xi,A, :I)
4: X (X),
3: X Fo(X, (c,)
end for
4: X + Concot (x
[1, :))
6: return X
5: for in (lo, .lk,, - 1) do
6: Sample qi Poo (-1X: -
7: X(t] + Q
8: end for
9: X + Fx(X, 4x2)
10: return X
Two final heads for logits and embedding prediction. The task of the model is to output the token in the codebook (with shape of V x C) which corresponds to the input noise. There are two approaches to achieve this goal: we can view the problem as a classification problem among all tokens in the codebook, we can treat it regression task aimed at outputting the correct embedding. Therefore, we set two final heads after the transformer backbone. For each token, one outputs p E RV the predicted logits among all tokens in the codebook, and the other outputs c E IRC as the predicted embedding. For logits output, we use cross entropy as the distance function d in Eq. 77. while we apply LPIPS loss (Zhang al., 2018) for the embedding prediction. The overall objective is a weighted sum of the two losses. We empirically find that the predicted logits perform well when is small and much worse when is large, while the predicted embeddings perform the opposite. Thus we simply set a split point and use the predicted logits for small ts and the predicted embeddings for the rest.
Additional embeddings for noise and data tokens. Since data tokens and noise tokens are fed in the model simultaneously, and their distribution differs significantly, it is necessary for the net- work to distinguish between them. Therefore, before the transformer block, We add two different learnable embeddings to the data tokens and noise tokens, respectively. The two embeddings are randomly initialized.
Positional Embedding. Actually, the processed sequence of our model is one token longer than the pre- pre-trained AR model. It is because the pre-trained model needs 1 class label token and 1 generated token to obtain the whole sequence, while our model needs class label token and n noise token generate the whole sequence. Thus, for models who have positional embedding, increase its length by and randomly initialize this new part.
Attention Mask. The attention mask of the traitra AR model also can not be used directly due to the mismatch in length. In our case, we let every token see all previous tokens. For VAR, all tokens generated at the same step each other as well.
15

---

Published as a conference paper at ICLR 2025
Table 4: Generative performance on class-conditional ImageNet-256. "#Step" indicates the number of model inference to generate one image. "Time" is the wall-time of generating one image in the steady state.
Type
Model
FID4
1St
Pret
Rect
MPara
SSepp
Time
AR
VAR-d20
(Tan er al.
2024]
3.35
301.4
0.84
0.51
600M
10
0,184
AR
VAR-d24 Than er al.
2024
2.51
312.2
0.82
0.53
1.03B
10
0,251
AR
LlamaGen (Sun et al.
2024
5.42
193.5
0.83
0.44
I11M
256
2.93
Ours
VAR-d20-DD
9.55
197.2
0.78
0.38
635M
1
0.027 (6.9x)
Ours
VAR-d20-DD
7.33
204.5
0.82
0.40
635M
2
0.047 (3.9x)
Ours
VAR-d24-DD
8.92
202.8
0.78
0.39
1.09B
1
0.034 (7.4x)
Ours
VAR-d24-DD
6,95
222.5
0.83
0.43
1,09B
2
0.059 (4.3x)
Ours
LlamaGen-B-DD
15.50
135,4
0.76
0.26
98.3M
1
0.018 (162.7x)
Ours
LlamaGen-B-DD
11.17
154.8
0.80
0.31
98.3M
0.029 (101.0x) 2
D EXPERIMENTAL DETAILS
In this section, we introduce more details of the experiments in Sec.
Dataset Generation. In the dataset generation phase, we use Alg. D to construct the data-noise pairs.
For the ImageNet dataset, generated total of 1.2M data-noise pairs; for the pext-to-image task, we generated a total of 3M data-noise pairs. For the AR models, we set the classifier- free guidance scale to 2.0 for both VAR and LlamaGen, while other settings follow the default configuration of these two works. the flow matching process, we employ the perturb function from Rectflow (Liu al., p022p: (( 21, t) (1 t.)ro tæ We use the DPM-Solver (Lu et al. 2022a b) to efficiently solve the FM ODE. Specifically, after calculating the velocity model given by Eq. (3 we wrap the model function Vax, to get the noise prediction model e(x;, l): <(x,
/V (2,,
Then we can directly use the official implementation of DPM-Solver (see https:: /tttps://githb. com/ homm/LuChengTHH.. hppp:/ppmsssa/ dpp--soveesv solver) to conduct sampling with the noise prediction model and the Rectflow noise schedule. Our configuration includes 10 NFE multistep DPM-Solver++ with an order 3.
Training Configuration. We follow most of the training configuration for the pre-trained AR model. For VAR, we use batch size of 512, a base learning rate le-4 per 256 batch size, and an AdamW optimizer with 8 0.9, 82 0.95. For LlamaGen, all other settings are the same ex- cept for the learning rate, which is fixed at le-4 and doesn't 't vary with the batch size. We distill VAR model for 120 epochs and LlamaGen model for 70 epochs. We additionally moving average (EMA) with a rate of 0.9999. As discussed in App. [ our model has two types of prediction results, therefore two types of loss. We assign loss weight of 1.0 for the embedding loss (LPIPS) and a weight of 0.1 for the logits loss (cross entropy) to maintain similar loss scale for them. For the timestep-wise loss weight, use a uniform one with A(t)
exponential
E ADDITIONAL RESULTS
E. RESULTS WITH MORE MODEL SIZES
In this section, present results with VAR and LlamaGen models of more parameter sizes in Tab.
E.2 ABLATION STUDY
Effect of the training iteration. From Fig. we can see that the few-step FID decreases rapidly in the first few epochs, demonstrating the high adaptability of the pre-trained AR model across different tasks, which contributes to the smaller distilling cost of DD than training AR models from scratch.
Effect of the intermediate timestep. From Fig. we find that the convergence speed and perfor- mance of different intermediate timesteps are similar, indicating that DD is not sensitive to it.
Effect of the dataset size. From Fig. 8 we find that the DD still work when there is only 0.6M (data, noise) pairs. Additionally, with more (data, noise) pairs, the performance improves. The results demonstrate DD's robustness limited data and its data scaling ability.
16

---

Published as a conference paper at ICLR 2025
step HDD vs Epech of VD
22.5
step RD vs Epach of VAR:DD
stop FID w Epech or UamaGen
step FID vs Eeech er UamaGen
30
4,-5
20.0
4,-5
24
4,m80
4,=120
17
4,=120
4,m00
25
k,m
16
17.5
20
15.0
18
50
100
50
100
100
200
300
100
200
300
Epoch
Epoch
Iter/0K
Iter/10K
Figure 7: The training curve of FID vs. epoch or iteration for different intermediate timesteps. FIDs are calculated with 5k generated sample.
1 step FID Epoch of VAR-DD
Step FID vs Epoch of VAR-DD
30
0.9M data
18
0. data
25
1.6M data
1.6M data
16
20
50
100
100
Epoch
Epoch
Figure 8: The training curve of FID vs. epoch for different dataset sizes. FIDs are calculated with 5k generated sample.
F MORE RELATED WORK DISCUSSIONS
In this section, we provide expanded discussions on two related works: DMD (Yin et al. 2024) and
MAR (Li al. 2024a).
DMD Yin et al. 2024) uses distribution matching to distill diffusion models into few steps. Specif- ically, the main idea of DMD to minimize the distance between the generated distribution and the teacher distribution at all timesteps. Additionally, it constructs data-noise pairs and conducts direct distillation as a regularization loss term to avoid mode collapse. There are several key differences between DD and DMD: (1) Target: DMD focuses on diffusion models, whereas DD focuses AR. These are two different problems; (2) Technique: In diffusion models, the diffusion process natu- rally defines the data-noise pairs, which can be directly used for distillation in DMD. In contrast, AR models do not have a pre-defined concept of "noise". How to construct noise and the data-noise pair (Sec. 3.2 one important and unique contribution of work.
That being said, the distribution matching idea in DMD very insightful, and could potentially be combined with from a high-level perspective, the noisy image in DMD is similar to the partial data sequence in AR, while the next token logits given by the pre-trained AR model can be viewed as the score function in DMD. The objective can be set as minimizing the distance between the output of a fake logits prediction network and the pre-trained AR all timesteps. As long as the next-token conditional distribution of the generated distribution is the same as the distribution given by the pre-trained AR model, the modeled one-step distribution will be correct. There no requirement for any data-noise pair in this casc.
achieve better approaches for few-step sampling of AR models. For example,
MAR (Li et al., 2024a) proposes to replace the cross-entropy loss with diffusion loss in AR, which shares some similarities with DD at high level since both works view decoding as a denoising process. The goals of these two works are different though. MAR targets removing the vector quantization in image AR models for better performance. while DD aims compress the sampling steps of AR, without any modification to the codebook. In this sense, these two works are orthogonal and could potentially be combined to develop new method that leverages the strengths of both.
17

---

Published as a conference paper at ICLR 2025
THE
Figure 9: Generation results with VAR model TTin al. 2024). From left to right: one-step model, two-step DD model, DD-pre--rrinedd44-66 and the pre-trained VAR model.
G
VISUALIZATION
In this section, demonstrate samples generated by DD. Results of label conditional generation demonstrated in Figs. to 12 Examples text conditional generation are demonstrated in Figs. B 15 where most texts are taken from the LAION-COCO dataset.
18

---

Published as a conference paper at ICLR 2025
Figure 10: Generation results with VAR model (Tian et al. 2024). From left to right: one-step DD model, two-step DD model, DD-pre--tainedd 4-6, and the pre-trained VAR model.
19

---

Published as a conference paper at ICLR 2025
Figure 11: Generation results with LlamaGen model (Sun et al. 2024). From left to right: one-step
DD model, two-step DD model, pre-trained-44-- and the pre- traintrdd LlamaGen model.
20

---

Published as a conference paper at ICLR 2025
Figure 12: Generation results with LlamaGen model (Sun et al. 2024 From left to right: one-step DD model, two-step DD model, fr--pr-traiied441 and the pre-trained LlamaGen model.
21

---

Published as a conference paper at ICLR 2025
Hammaaen-226ster Sun et al., 2024)
DD-2step (Ours)
Text Prompts
"Red Celtic Heart Knot T-Shirt" "
"Diamanttavla Pink Skull Beauty 50x50"
T TITICCINN AS6IL
"CHRYSLER Town Country"
"2 Drawer Nightstand by PoliVaz"
"Relative Risks etc"
MA
"1 Pottery Barn Emery Linen Drapes Panels Curtains Blackout Lining 50x63 Red"
Figure 13: Comparisons of text-to-image results between DD and the pre-trained LlamaGen model (Sun et al. 2024). Prompts without are taken from the LAION-COCO dataset.
22

---

Published as a conference paper at ICLR 2025
LlamaGen- 256step Sun et al. 2024)
DD-2step
Text Prompts
(Ours)
Teleod Bool Veuse Dieno tmm
"WEEKLY or MONTHLY. Soft Touch Genuine Leather Sectional in BARK T-Shirt"
06
"Toy Watch Men's s Plasteramic Diver Watch"
21:0
"Wood Wall Behind Tv Ask A Designer Series Mistakes Made To Tv Walls Nesting With Grace" "
"Activate Upholstered Fabric Armchair"
"Womens Ladies Suudded Ankle Strap Espadrilles Platform Shoes Wedge Sandals Size"
"A pair of shoes on the floor "*
Figure 14: Comparisons of text-to-image results between DD and the pre-trained LlamaGen model Sun al. (2024). Prompts without are taken from the LAION-COCO dataset.
23

---

Published as a conference paper at ICLR 2025
LlamaGen 256ssep
DD-2step (Ours)
Text Prompts
Sun al. 2024
"Butterfly Women's 7-Shirt"
"chaeeecniiss-oolltttonsssttisss 2016-16--oooffill-leessss---ifi (58)"
CASAANVE 9.4S MASSTAE
ARA
"A free hill"*
2o
"Meeting Facility, Holiday Inn Munich- Munich-Unterhachinng
PIAG 1-4-0 P FE
bani one nooe 10000
"A bridge over a river"*
"A bowl of fresh fruits"*
Figure 15: Comparisons of text-to-image results between DD and the pre-trained LlamaGen
model Sun et al. 2024). Prompts without are taken from the LAION-COCO dataset.
24


Reviews:
[{"review_id": "JZCfXTQQAh", "reviewer": "Reviewer_cVWg", "summary": "The method maps autoregressive model output distributions to a Gaussian distribution through flow matching, creating a deterministic transformation. The goal is to then learn this mapping with a neural network. This enables parallel token generation while preserving the conditional dependencies between tokens in the AR model's output distribution.", "questions": "1. Your method depends on DPM-Solver to generate the trajectories. Do you think this affects results in any way? What other dependencies does your method have that might affect results?\n2. Do the hyperparameters for DPM-Solver affect the results drastically? Why did you pick the current hyperparams?\n3. Do you think a larger amount of timesteps might be needed to get satisfactory results? Is this compatible with your method?\n4. Skip baselines seem conceptually weak. The one-step baseline is guaranteed to fail given results on toy examples. Are there no better baselines - for example progressive distillation, that is used for diffusion models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The work presents a method that leverages deterministic flow matching to create training data (from an AR model) for a one-step image generation model. When trained on this data this model is a distilled version of the original AR model. The idea of using determinstic flow matching to create the data is novel and seems like a good and innovative candidate idea to achieve this. The paper evaluates the claims on class-to-image generation on ImageNet and compares to simple baselines, achieving acceptable FID scores, that are high but not too high.", "weaknesses": "A. It seems like the FID increases, although seemingly acceptable in numerical terms, give rise to blurry and artifact-ridden images, many of which don't even preserve the structure of the class they are trying to generate (monkeys without eyes etc.). Also, another thing that undermines my confidence is that no images are shown in the main paper and instead shown in the appendix. At least some examples are shown.\n\nB. One big problem from (A) is that, since the paper's main premise is to distill an AR model into a one-shot model, good samples are required to demonstrate that this approach is viable. Given the current results we do not know if this approach can actually generate satisfactory images even when scaled or taken into another regime with larger data such as T2I.\n\nI think these are the main issues for me to be convinced of this approach. My question to authors are:\n- Do you expect the approach to become good enough to generate satisfactory images with different scale (data, model size)?\n- If so, what leads you to believe this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vlDohRAj1i", "reviewer": "Reviewer_ziid", "summary": "This paper introduces Distilled Decoding (DD), a novel method to reduce the sampling steps of autoregressive image generation models. The approach reframes the traditional next-token or scale prediction process into a denoising procedure guided by flow matching. Starting from fully noisy tokens, DD employs an autoregressive process to map these tokens into image tokens in a flexible number of sampling steps. Experiments with the LlamaGen and VAR models demonstrate that DD significantly outperforms conventional few-step sampling baselines.", "questions": "- The writing is somewhat unclear. It's not obvious whether $q_t$ represents tokens in the discrete codebooks or outcomes of flow matching. Algorithms 1-4 are much clearer than the main text, so refining the writing to enhance intuitiveness would benefit the readers.\n\n- The performance gap between few-step AR samplers and the baseline remains quite large.\n\n- Some baseline values, such as VAR/LlamaGen in Table 1, appear to be significantly lower than the original paper's reported results.\n\n- It would be interesting to explore how DD performs in text-conditioned image generation tasks.\n\n- The structure of DD closely resembles DMD in diffusion distillation, where a <noise, clean image> dataset is constructed; however, DMD isn't addressed in this paper. Adding a discussion on DMD would be beneficial. Additionally, transforming next-token prediction to AR-based token denoising at a high level seems conceptually similar to MAR (Li et al., 2024, cited). Including further discussion on MAR could also be helpful.\n\n===\n\nUpdate: I've read the author response and it has addressed my concerns, I will therefore keep my score and recommend acceptance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors proposed a new framework to make few-step AR distillation possible. \n- The conversion from next image token prediction to next (set of) image token denoising is a very smart design. The method naturally combines the best of worlds in diffusion models / flow matching and autoregressive image modeling.\n- The performance gain against baseline few-step samplers is huge.", "weaknesses": "- The authors propose a novel framework enabling few-step autoregressive (AR) distillation.\n\n- Converting next-image-token prediction to next-image-token denoising is an ingenious design choice, seamlessly integrating the strengths of both diffusion models and autoregressive image modeling.\n\n- The performance improvement over baseline few-step samplers is substantial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V11AaMEJPs", "reviewer": "Reviewer_2cqS", "summary": "The paper is concisely written with a clear line of thought. The approach of constructing a continuous embedding space in AR and matching it to a Gaussian distribution is particularly interesting. This construction effectively combines the discrete cross-entropy model from AR-based methods with the probability distribution strategies that have proven successful in diffusion models, allowing the concept of consistency distillation from diffusion to be successfully applied within the AR field. The comparisons are comprehensive, and the experiments are well-executed.", "questions": "My questions lean more toward potential future design possibilities, as the explanations provided in the paper are relatively clear.\n\nQ1: The approach seems very similar to a variant of the Consistency Model (CM) applied to the AR domain, replacing the diffusion model in CM with an AR model. What are the advantages of this substitution? Line 319 mentions that, compared to diffusion flow matching, DD “has an easy way to jump back to any point in the trajectory.” This is noted as a characteristic, but what practical benefits does it provide? Additionally, the ODE sampling trajectory in CM is also theoretically fixed and reversible. If my understanding is incorrect, please correct me.\n\nQ2: Is there a way for this method to be trained from scratch, similar to Consistency Training?\n\nQ3: Currently, the method requires a dataset size of 1.2M. How significantly does the data volume impact performance? Related distillation methods, such as DMD[1], use relatively smaller datasets. Is there a possibility of a similar approach to DMD2[2] that could discard the need for noise-data pairs?\n\nQ4: Q: In line 234, is $\\pi$ the same as z used later on? There may be a misalignment in the notation here.\n\n[1]. Yin T, Gharbi M, Zhang R, et al. One-step diffusion with distribution matching distillation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 6613-6623.\n[2]. Yin T, Gharbi M, Park T, et al. Improved Distribution Matching Distillation for Fast Image Synthesis[J]. arXiv preprint arXiv:2405.14867, 2024.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is concisely written with a clear line of thought. The approach of constructing a continuous embedding space in AR and matching it to a Gaussian distribution is particularly interesting. This construction effectively combines the discrete cross-entropy model from AR-based methods with the probability distribution strategies that have proven successful in diffusion models, allowing the concept of consistency distillation from diffusion to be successfully applied within the AR field. The comparisons are comprehensive, and the experiments are well-executed.", "weaknesses": "The paper successfully applies the consistency distillation (CD) technique from diffusion models to the AR field. However, the current results still fall short compared to the pre-trained model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SiAg33cp5s", "reviewer": "Reviewer_ni7B", "summary": "This paper presents a method called Distilled Decoding (DD) that distills autoregressive models into fewer steps. It employs flow matching to establish a deterministic mapping from Gaussian noise to the output distribution of the pretrained autoregressive models. Extensive experiments demonstrate that DD outperforms the baselines, accelerating autoregressive models with minimal performance degradation.", "questions": "In table 1, why VAR-DD with one step achieves better performance than VAR-DD with two steps?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow.\n2. The presented idea of compressing AR models with Flow Matching is intersting. Given that autoregressive models tend to be slow, accelerating them is a crucial challenge. The efforts presented in this paper could provide valuable insights for the community.", "weaknesses": "1. Lines 201-203 state that modeling the joint distribution of several tokens is impractical due to the vast possible space. However, it seems that the proposed method uses Flow Matching to learn the joint distribution of tokens (the full sequence in the one-step case). This is somewhat confusing—how does the proposed method tackle this issue?\n2. The image quality shows significant degradation (as indicated by both FID scores and the figures), raising concerns about the practicality of the proposed method.\n3. It would be beneficial to include experiments with text-conditioned LlamaGen to demonstrate that the method can be adapted to more complex scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)