Distilled Decoding 2: One-step Sampling of Image Auto-regressive Models with Conditional Score Distillation
Authors: Enshu Liu
Qian Chen
Xuefei Ning
Shengen Yan
Guohao Dai
Zinan Lin
Yu Wang
Venue: NeurIPS 2025 poster ()
Keywords: Image auto-regressive model; One-step generation; Score distillation
Primary area: deep_learning
TL;DR: We propose a new method that reduce the sampling steps of image auto-regressive models to only 1

Abstract: Image Auto-regressive (AR) models have emerged as a powerful paradigm of visual generative models. Despite their promising performance, they suffer from slow generation speed due to the large number of sampling steps required. Although Distilled Decoding 1 (DD1) was recently proposed to enable few-step sampling for image AR models, it still incurs significant performance degradation in the one-step setting, and relies on a pre-defined mapping that limits its flexibility. In this work, we propose a new method, Distilled Decoding 2 (DD2), to further advances the feasibility of one-step sampling for image AR models. Unlike DD1, DD2 does not without rely on a pre-defined mapping. We view the original AR model as a teacher model which provides the ground truth conditional score in the latent embedding space at each token position. Based on this, we propose a novel \emph{conditional score distillation loss} to train a one-step generator. Specifically, we train a separate network to predict the conditional score of the generated distribution and apply score distillation at every token position conditioned on previous tokens. Experimental results show that DD2 enables one-step sampling for image AR models with an minimal FID increase from 3.40 to 5.43 on ImageNet-256. Compared to the strongest baseline DD1, DD2 reduces the gap between the one-step sampling and original AR model by 67\%, with up to 12.3$\times$ training speed-up simultaneously. DD2 takes a significant step toward the goal of one-step AR generation, opening up new possibilities for fast and high-quality AR modeling. Code is available at https://github.com/imagination-research/Distilled-Decoding-2.

Distilled Decoding 2: One-step Sampling of Image Auto-regressive Models with Conditional Score Distillation
Enshu Liu Tsinghua University Beijing, China
Qian Chen Tsinghua University Beijing, China
Xuefei Ning Tsinghua University Beijing, China
Shengen Yan Infinigence-Al Beijing, China
Guohao Dai Shanghai Jiaotong University Shanghai, China
Zinan Lin*t Microsoft Research Redmond, WA, USA
Yu Wang Tsinghua University Beijing, China
Abstract
Image Auto-regressive (AR) models have emerged as a powerful paradigm of visual generative models. Despite their promising performance, they suffer from slow generation speed due to the large number of sampling steps required. Although Distilled Decoding (DD1) was recently proposed to enable few-step sampling for image AR models, it still incurs significant performance degradation pre-defined mapping that limits its flexibility. In
the
one-step setting, and relies this work, we propose a new method, Distilled Decoding 2 (DD2), to further advance the feasibility of one-step sampling for image AR models. Unlike DD1, DD2 does not without rely on a pre-defined mapping. We view the original AR model as teacher model that provides the ground truth conditional score in the latent embedding space at each token position. Based on this, we propose a novel conditional score distillation loss to train one-step generator. Specifically, we train a separate network to predict the conditional score of the generated distribution and apply score distillation every token position conditioned on previous tokens. Experimental results show that DD2 enables one-step sampling for image AR models with minimal FID) increase from 3.40 5.43 and 4.11 to 7.58 on ImageNet-256, while achieving 8.0x and 238x speedup with VAR and LlamaGen models, respectively. Compared to the strongest baseline DD1, DD2 reduces the gap between the one-step sampling and original AR model by 67%, with up to 12.3 x training speed-up simultaneously. DD2 takes a significant step toward the goal of one-step AR generation, opening up new possibilities for fast and high-quality AR modeling. Code available at https: //gs:://iitee thub. com/ reseentttrn-rrercrchhh.eee
Decoding-2
Introduction
Image autoregressive (AR) models have recently achieved state-of-the-art performance in high-fidelity image synthesis, surpassing other generative approaches such as VAEs, GANs, and diffusion models [38, 3, 6, 27, 15, 45, 2, 16, 37, 35, 17, 9, 14, 8, 31, 10, 34].
Project Advisor: Ziman Lin. I "Correspondence to Zinan Lin (zinan] www.omicooooo.commm and Yu Wang (yu-vangdmail. www.aaaaillttinnnnuaaaaaaa edu. ed).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite their strong generation ability, a key limitation of AR models lies in their inherent sequentially modeling manner, which leads to the token-by-toker sampling process and significantly slower inference speed. Numerous methods have been proposed to reduce sampling steps [41, 1, 18, 36, 11, 2, 37, 17], but nearly all fail to achieve single-step sampling without significant performance degradation, leaving room for further speedup. Please refer to Sec. 4.1 for more details.
Distilled Decoding 1 (DD1) [21] marks a significant breakthrough in reducing the sampling steps for AR mod- els, as it is the first method capable of compressing the sampling process of an AR image model to only single step. DDI introduces flow matching [22, 20] into the AR sampling pipeline. Specifically, instead of sampling the next token directly from probability vector output by the AR model, DD1 leverages flow matching in the codebook embedding space to transform a noise token into a data to- ken. This enables token- wiiee deterministic mapping from noise to data while preserving the output distribution of the original AR model. By iteratively conducting this process following the original AR sampling order, DD1 obtains a complete mapping from a noise token sequence to a data token sequence. Then, new model is distilled to directly learn this mapping, allowing the generation of the entire token sequence in a single forward pass.
1 token/set needs 1 step
step
Autoregrastive Model
Distill
One-step
Generator (Oum)
Our goal is to distill multi- step AR model in to a one-step generator while keeping its distribution.
Figure 1:
However, the constructed mapping is inherently challenging for the model to learn, resulting in noticeable performance drop compared to the original AR model. addition, training generative model to directly fit predefined mapping may impose constraints on the flexibility. In contrast, models like GANs and VAEs, which do not learn explicit input-output correspondences, have shown broad applicability across downstream generation tasks [19]. This insight leads us to ask:
Can we train one-step generative model whose output distribution matches given AR model,
without relying on any predefined mapping?
To answer this problem, we propose Distilled Decoding (DD2) as a completely new method. Inspired by DD1, our key motivation is to reinterpret the AR model, which originally outputs discrete probability vector for the next token di. as a conditional score model that predicts the gradient of the log conditional prohability density (i.e., the conditional score) in the codebook embedding space. Specifically, we view the generation of each token as a conditional flow matching process. Based on this, given all previous tokens 91.....f-1 the condition, we can use the teacher model's output probability vector to define a conditional score s(g;, tlg=s) = Vag log p(g qcs). where t denotes the flow matching timestep. Unlike DDI, where the conditional score is used solely to construct an ODE-based mapping, we aim to make fuller use of this signal. We borrow ideas from score distillation methods (e.g., [26, 40, 24, 43, 48)), which match the score of ax one-step generator's ss distribution to that of a teacher diffusion model and have recently shown strong performance in diffusion- based generation. Specifically, our DD2 jointly trains a one-step generator and a conditional guidance network that learns the conditional score of the generator distribution. We propose a novel Conditional Score Distillation (CSD) loss for training, which aligns the conditional score between the guidance network and the teacher AR model at every token position. We show that when the CSD loss is minimized to its optimality, the output distribution of the one-step generator matches exactly that of the original AR model.
is important to highlight that our method is fundamentally different from diffussion score distil- lation. Although both approaches involve aligning scores, AR models and diffusion models follow completely different modeling approaches and generation processes. As a result, the goals and challenges in this paper are inherently distinct from previous works. More discussion about the differences between the two methods can be found at Sec. 6.
To validate the effectiveness of DD2, we follow the evaluation setup of DD1 and conduct experiments on Im2geNet-256 [4] with two strong autoregressive models: VAR [37] and LlamaGen [35]. On VAR, we reduce the sampling steps from 10 to 1 with a marginal FID increase less than 2.5 (e.g.. from 4.19(t to 6. 6.21), achieving a up to 8. x speedup. Compared to DDI, DD2 reduce the performance gap between the 1-step model and the original AR model by up to 67%. On LlamaGen, we compress the sampling process from 256 steps to with an FID degradation from 4.11 to 8.59, resulting in a 238x speedup. Compared to DDI, our 1-step model achieves an FID improvement of 2.76. Further
2

---

VAR
LlamaGen
300
(Ous, stap!
D02 toun. SARR)
stup)
noo
300
Protrainad Uamradoo
Lamaaiendk stop)
50
stop!
50-
FID
1.4
10
12
10
8-
Ours
Ours
50
100
150
200
250
10
102 10a
10*
Time per Image/ms
Time per Image/ms
Figure 2: Comparison of DD2 models, DD1 models, pre-trained models, and other acceleration methods for pre-trained models. DD2 achieves significant speedup compared pre-trained models while outperforming DD1 by a large margin. Other methods fail to achieve one-step sampling. For DD2, DD1, and the pre-trained model, each point corresponds to a different model size, whereas for the skip-last method, each point corresponds to a different number of skipped final steps.
comparisons between DD2 and other baseline methods are presented Fig. Additionally, DD2 highly efficient to train: compared to DD1, it achieves up to 12.3x training speedup. We hope this work can inspire future research toward making image AR models maximally efficient while keeping their superior sample quality.
2
Preliminary
In this section, we introduce the formulation of standard image AR models to understand DD2.
2.1 Image Tokenizer
To train an image AR model, we first need to convert continuous-valuce images into discrete token sequences, so that the probability of each token can be explicitly outputted by the model. Recent AR models mostly rely on vector quantization (VQ) [39], which leverages an encoder E, a quantizer Q, and decoder
discretize and reconstruct visual content.
The process begins by encoding the input image x E x into a latent representation: ='= Exx), where z' = (21, 22,.... = E RC'hxx is lower-resolution feature map containing x w embeddings, each of dimension C. For each embedding 2;, the quantizer selects the nearest code vector gi from a learned codebook V (c1, C2..... . cv) E RVXC" The resulting discrete token sequence is denoted
(g1, ................................ Qhxw), where each qi is token Cj in V. To reconstruct the original image, the decoder takes z as input and produces:
= D(z). During training, reconstruction loss ((. æ) is used to ensure fidelity between the original and the reconstructed image. This VQ-based framework underpins many state-of -tate-of-the-ar -het image AR models [16, 2, 37, 35, 17].
rugo-segees Modeling
Once well trained image tokenizer is available, an AR model can be employed for image generation. We assume that an image is represented as a sequence of discrete tokens 2 = (q1. ... .4)), where each gi corresponds an embedding from the codebook V: qi E {cI.... cv The AR model is trained to estimate the conditional probability distribution of each token given all previous tokens: p(g: (q<s) = aiqilgi-1: -:-2-" ... @1 (Pi..... ........PP...................... .......... .Py). where p, denotes the probability that the next token corresponds to the j-th entry in the codebook.
At generation time, the model samples tokens one by one in order, and the likelihood of the full sequence is given by: p(Z) = II (<x)). This generation procedure requires n autoregressive steps, which is often large number, resulting slow inference speed and limited efficiency.
3 Distilled Decoding 2
In this section, we first introduce the formal problem definition in Sec. 3.1. Then, we propose Conditional Score Distillation (CSD) loss as the core component of DD2 in Sec. 3.2. Then we discuss our initialization method in Sec. 3.3, which plays a crucial role in training speed and performance, Finally, we present the full training pipeline of our approach.

---

Noise Seq
Generator
Generated Seq
Noise Seq
Generator
Generated Seq
Condition
Noisy Taken
Condition
Noisy
Generator Gradient
Guidance Gradient
Teacher
Guidance
(Pu
Guidance
fake
falke
score
d(d", al 2,
Score
spore
L2 loss
Score distillation loss
(a) Training loss of the generator.
(b) Training loss of the guidance network.
Figure 3: Training process using CSD loss. For the generator, the teacher AR model and the guidance network give the true and fake conditional score each noisy token based on all previous clean tokens, respectively. Then the true and fake conditional score are used to calculate the score distillation loss, which produces gradient to train the generator. The guidance network learn the conditional score of each noisy token given all previous clean tokens, by optimizing with standard AR-diffusion loss [17]. The generator and the guidance model are trained alternately.
3.1 Problem Formulation
Suppose a image can be encoded as sequence of length and we have a well trained teacher AR model pa. which gives the next token probability conditioned on all previous tokens pa po<xilze (x)s) and will be fixed. Our goal is to train a one-step generator Go, which can output generated sequence 28 = (41.... 9n in one run given latent variable drawn from the prior distribution: 28 = Go(e).
We hope the distribution of match the distribution of the teacher AR model.
3.2 Conditional Score Distillation Loss
In this section, we first introduce our idea of viewing teacher AR model as a conditional score model in Sec. 3.2.1, and propose the objective based on it in Sec. 3.2.2. Then, present how train the conditional score model for the generator distribution in Sec. 3.2.3, as it is required by the objective.
3.2.1 Teacher AR as a Conditional Score Model
Considering the generation process of the i-th token 9: given previous tokens (q1...., ---1) as the condition, we have the probability vector p = (Pi..... .PV) outputted by the teacher AR model, where Pj 2 0 denote the probability of j-th token ej and Σ)= 1. Inspired by DD1 [21], view the sampling process as continuous transformation of flow matching [22, 20] from source Gaussian distribution at = to a sum of Dirac function 5(-) weighted by p at t = 0:
p(g:)= Σy=1 p,0(g= cj). By choosing the noise schedule of RectFlow [22], the score function can be expressed in closed form as:
(1)- 21
s(x,, t.p) = Ey= (x+ - (1 f)c;)e
0eg)2
t Ey P
203
For more details the derivation of this expression, refer App. C.I. By substituting % as x and (Pi....,Py'
Po (c)())))) to Eq. (1), we rewrite the left side of Eq. (1) in the form of conditional score function s(x,, t, p) = se (of qf = t)q: + tc, c N0(0; I). The term conditional score refers to the score of of conditioned on Izc. Note that the condition term qzi consists of previous clean tokens without any noise injection, so it can also be noted as 22.-
Here of is noisy version of the clean token q::
3.2.2 Training Objective
With access to the true score function, we aim to make full use of this information rather than using it merely to construct an ODE mapping in DDI. To this end, we draw inspiration from score distillation methods. These methods seek align the distribution generated by model with that of teacher by matching their respective score functions.

---

We first present general formulation of score distillation. Let z E RC be random variable. Denote Po and So as the probability density function and its score function given by the teacher model 4. Po and fake as the probability density function and its score function of the generator 0. A general score distillation loss can be given as:
+ oxe, (), 8 S (0s300 + are, 1), where d a function satisfying that minimal LSD guarantees Vo E RC, ,po(x) = po (z)) In practice, choose SiD loss [48] due to its effectiveness, giving:
Lsn [o~ (o.
d = w (1) (s4 Sfake) (s4 + E a(ss 8yake)),
(3)
where (t) is the weight function and is syper-parameten, which we set 1.0.
In our scenario, however, we are not aligning the distribution of a single random variable like score distillation for diffusion models [43, 24, 48), but a sequence of random variables with auto-regressive correspondence. Specifically, we aim to match the generator's 's conditional distribution at each token position with that of the teacher AR model. This motivates us minimize the score distillation loss
on all token positions. Additionally, we have to replace the score term in Eq. (2) with the conditional score given all previous tokens and a,, ot with the noise schedule of RectFlow [22]. By incorporating above modifications to Eq. (2), we propose our conditional score distillation (CSD) loss:
LCSD = (asslonn-aa))peeee
Σ d(se (a t,1sg(4<s)) S Sfake (g. t, t((gg(<<))),
(4)
where of (1 t)ys + te and (-(-) means the stop gradient operation. We give the following proposition to show the correctness of our CSD loss, with brief proof in App. A.
Proposition 1. Minimal LCSD guarantees V/2 (91...., 9n R po(2) po
Intuitively, Eq. (4) encourages progressive alignment of the token sequence distributions. Consider the first token qı, which has no constraints by any other tokens. Its associated loss term reduces to standard score distillation loss Et (0. di ss (gi²t f2), fake(9f fz)), which encourages (2()1) ) to align with pe (21). Once the first token's 's distribution is aligned, we then consider the loss for the second token: Eta mop PP0 (o.)) dl( (g²² 12 (sg()))) )), fake (g²² 1g2 (sg())) Optimizing this ensures po (42 44) = pv (92 (1). Given that po = pa (g)) has already been achieved, it follows po(91. 92) py (q1, 92) By sequentially matching the distribution on each token position, we can finally align the entire distribution pe(g (91..... 9n) with pa (q1.....4... 9n))
3.2.3 Learning the Conditional Score of the Generator
To optimize Eq. (4), we need to access the conditional score of the generator Syake (g tzlg<s). Following previous works of diffusion score distillation [24, 43, 42, 48], we train a separate model e) output this term, which refer to the conditional guidance network.
Specifically, our guidance network consists of decoder-only transformer backbone and lightweight MIP head with negligible cost. The training procedure is inspired by MAR [17]. Given generated token sequence backbone, yielding sequence of hidden features ,fi....,f.. fn). tokens q<i and thus captures strictly causal context. For each token position i, the MLP takes as input noised version of the token gi', the corresponding timestep t,, and the contextual feature fi. Since fi only corresponds to q<s as the conditioning, we denote the outputted score function as the fake conditional score ( q.z)). We train the model across all AR positions in parallel and then present the following loss:
9n.) from the generator, we first process with the causal transformer
Each feature f; only corresponds to
Lrcs = F www eo
e (O:D)
Σ 1s falges) V log logp(1""aa)))
(5)
where of = -1- 1)q + le, and V logr(1. lg:) can be simplified to [34]. The MLP and transformer backbone are jointly optimized with Lecs.
In practice, the guidance network s and generator 0 are trained alternately using Eq. (5) and Eq. (4), respectively. During generator training with Eq. (4), the score term fake is entirely replaced by 8g-
with gradients blocked from propagating into th. The training algorithm and an illustration of the pipeline are provided in Alg. I and Fig. 3.
5

---

Seq from Dataset
Teacher
Guidance
Copy
Alternabolly
Iram
Wel Ene REE
trained
Condition
Notay Tokon
Random MUP
Cenurator
Generator
AR-diffusion Gradient
Teacher
AR-diffusinn
(Pa-
(optional)
Talia score
Figure 5: Overall pipeline. Performance alignment optional technique in the CSD training stage, which is introduced in App. B.4.
Scove
Scoce loss
Figure 4: Training loss for initialization.
3.3 Initialization of Generator and Guidance Network
With the training procedure outlined in Alg. 1, we are now ready for DD2 training. However, directly applying this method does not yield satisfactory results. We attribute this to the poor model initialization. We delve into this issue and propose our solutions in the following part of this section.
We find that good initialization is crucial for score distillation methods: poor initialization can lead to slow convergence or even training collapse. To validate this, we conduct diffusion distillation experiments on the ImageNet-64 dataset using the original DMD [43, 42] approach under different initialization schemes: (1) Default: both the guidance and generator models are initialized from pretrained teacher diffusion model, (2) Random Guidance: the guidance model is randomly initialized, (3) Random Generator: the generator is randomly initialized, and (4) Partial Random Generator: only the final layer of the generator randomly initialized. As shown in Fig. 6, improper initialization of either the guidance or the generator leads to significant training degradation. Even randomly initializing just the final layer of the generator severely impacts the performance. This is because initialization determines both the internal knowledge stored in the network and the generator's initial distribution, both of which are critical to stable and efficient score distillation training discussed in [46].
In our setting. both the generator and the conditional guidance network output continuous values. while the teacher AR model produces probability vectors. This structural mismatch makes it impos- sible to directly reuse the model weights from the teacher AR model to initialize the output heads. To address this, we propose a novel initialization strategy: we first replace the teacher AR model's classification head with a lightweight MLP, and fine-tune the new model with AR-diffusion loss [17] to align its distribution with the teacher AR model. This process is similar to the training of the conditional guidance network and MAR model [17] but with a key difference: we introduce Ground Truth Score (GTS) loss by replacing the Monte Carlo Estimation in Eq, (5) with the ground truth score calculated using the teacher AR model with Eq. (I), giving:
LaTs = (.
|| talgez) - s.lal' (.
(6)
This loss significantly improves training stability and convergence speed, as demonstrated in experi- ments in Tab. 7. The training process is shown at Alg.
For both generator and guidance network, we adopt the same architecture composed transformer backbone and lightweight MLP head, both initialized from the tuned AR diffusion model. For generator, we sample noise sequence E = (e1....,f., .,.) as the latent variable input, where each ~ (O; I). This sequence is fed directly into the MI.P, while a one-step offset version is provided to the transformer backbone. Model architectures are shown in Fig. 7.
Such strategy serves as a strong initialization for both generator and guidance network, improving the training significantly as demonstrated in Sec. 5.5.
Overall pipeline The complete training process consists of two stages: an initialization tuning phase with Eq. (6) and main training phase with Eq. (4) (for the generator) and Eq. (5) (for the guidance network). The workflow is illustrated in Alg. and Fig. 5. More techniques can be found in App. C.
Multi-step sampling involving the teacher model. To achieve more flexible trade-off between sample quality and steps, we can use the teacher model to refine the last several steps of the one-step generated sequence. Details of this sampling method are shown in App. D and Alg, 4.

---

Algorithm 1: Training with CSD loss
Algorithm 2: Tuning the AR-diffusion model
Require:
Require: dataset D, the pre-trained teacher AR model 4, AR-diffusion model V.
the pre-trained teacher AR model db
I: while not converged do
1: initialize the backbone of w with the backbone of o, randomly initialize the MLP head of
2:
(q....., 9m.) + Go(e)
2: while not converged do
Train generator 8 3: Sample L = (h.....4u,, Ln), sample
3: Sample 2 - (q). .....))..)))))))) (n) ~ D
4:
Sample f = (f1...., f.), sample be (e1...., from Gaussian distribution
2, = (c1....., cn) from Gaussian
distribution
5: z' = (q) ....,... ) 6 (1 t)2 tz.
4: 2f = =."""....,
..... ghh)))))))))rr ) (1
6: update V with Cors(2" , 2,,, M, 2) Eq. (6).
5: update @ with LOSD Eq. (4).
7:
end while
8:
return V/
// Train guidance network x/ Sample another t = another (El....,En) En) from Gaussian
A Algorithm 3: Overall Pipeline
6:
1, sample
Require: dataset D, the pre-trained teacher AR model 0%
distribution
1: train the AR-diffusion model Vi with Alg.
7: 2' = (o) .....9h" .....g")) t- (1- 1)2 tz.
2: duplicate the trained v/ as generator and guidance network /b
8:
update v with FFcs (20, 22,, Eq. (5).
3: train generator 0 and guidance network & with Alg. 1
9: end while
4: return B
10: return A
4 Related Works
4.1 Reducing the Sampling Steps of AR Models
Many prior works have attempted to reduce the sampling steps of AR models. Set prediction is a commonly used approach in image AR modeling, where the model is trained to predict the probability of a set of tokens simultancously [2. 16, 37, 17J. It significantly reduces the number of sampling steps to around 10. However, this method struggles to sample with very few steps (e.g., 1), due to the complete loss of token correlation within each set. As the set size increases, this loss becomes increasingly detrimental to sample quality as discussed in [21]. For example, consider the case where the dataset contains 2 data samples with 2 dimensions: DD = {(0, 0). (1, 1)). The one-step sampling yields uniform distribution among { {(0, 0), (1, 1), (0, 1), 01) which is incorrect. For more details, please refer to the Section 3.1 of DD1 [21]. Speculative decoding is another method of step reduction, which is widely used in large language models (LLMs) [41, 1, 18] due to its training-free frren property. It generates several draft tokens with a more efficient sampling method and then verifies them in parallel using the target model. Speculative decoding can achieve only a limited
compression ratio of sampling steps (less than 3x) ) in image AR generation [36, 11], due to the weak modeling capacity of the draft generator.
Distilled Decoding (DDI) [21] is the first work that compress the sampling steps of image AR models to without performance collapse. The key idea of DDI is construct deterministic mapping between a sequence of noise tokens and a sequence of target tokens. Specifically, given the probability vector outputted by the teacher AR model when generating the next token, DD1 replaces the multinomial sampling process of the original AR model with flow-matching sampling. The required velocity field can be accurately calculated through Eq. (1). By conducting this process following the original AR order, DDI can map a noise sequence to a data sequence. Then DD1 simply train a neural network fit this mapping. Although DD1 enables one-step sampling, it suffers from significant performance degradation and relatively slow training. Moreover, its reliance on a predefined mapping limits flexibility. As a new one-step training framework for AR models, DD2 effectively alleviates all the issues above.
4.2 Score Distillation for Diffusion Models
Similar to AR models, diffusion models (DMs) also suffer from a large number of sampling steps required by solving the diffusion ODE/SDE. Score distillation [26, 40] serves as a method to distill the multi-step teacher DM into a one-step generator [24, 43, 42, 48]. The main idea of score distillation is to make the distribution of the generator indistinguishable to the distribution of the teacher DM. general formulation of score distillation generator loss has been given in Eq. (2). guidance network is introduced to approximate the score function fake of the generator. Both models are trained in turn. Different score distillation methods choose different types of generator loss. For example, [24, 43, 42) take the KI. divergence between the two distributions as the objective, which gives d = 8fake se- SiD [48] aims to optimize the l/ distance between the score functions of the two distributions, giving Eq. (3) as the loss. Compared to traditional diffusion distillation method
7

---

based on ODE mapping [23, 29, 33, 13, 32], score distillation methods are more flexible and have better results. By introducing training data, [46] eliminates the teacher guidance in score distillation through class-ratio estimation. Recently, there are also methods applying DMD to temporal causal data type like video [44]. The main difference between our paper and [44] lies in the problems they aim to solve: our work focuses on few-step sampling for AR models, where an AR model is available as the teacher, while [44] targets to decrease the step of DMs and assumes access to a teacher DM.
5 Experiments
In this section, we apply DD2 to existing pretrained AR models to demonstrate DD2's strong ability to compress AR sampling into a single step.
5.1 Sctup
Base Models and Benchmark. In line with DD1 [21]. we choose VAR [37] and LlamaGen [35] as the base AR models due to their popularity and strong generation quality. Moreover, these two models differ significantly across several key aspects, which makes them ideal testbeds for evaluating the generality of DD2: (1) Tokenizer training: VAR's s codebook is trained to support multi multi-resolution image tokens, while LlamaGen's 's tokens are derived solely from the original resolution space; (2) Token ordering: VAR constructs the full sequence by concatenating sub-sequences across different resolutions, whereas LlamaGen follows traditional raster-scan order; (3) Generation steps: VAR has 10 sampling steps, while LlamaGen requires 256 steps. These differences allows us to evaluate how DD2 performs across a wide range of AR setups. We choose the popular and standard ImageNet- 256 dataset as the benchmark.
Generation. We use the one-step sample quality main results in Tab. 1. Additionally, following DDI, we involve the teacher AR model in sampling for smoother trade-off of quality and steps. Results are listed in Tab. 2.
Baselines. Since DD1 [21] is the only method that enables few-step sampling for image AR models, we take it as our main baseline. We also report the results of several weak baselines in the DD1 paper: (1) directly skip last several steps, and (2) predicting the distribution of all tokens in one step. which is the extreme case of set-of-token prediction method. Details of baseline can be found in App. E.S.
5.2 Results of One-step Generation
We demonstrate the main results of DD2 in Tab. 1. Since the model parameter sizes and inference latency of DD2 and DDI are similar, we compare them under the same number of sampling steps. The key takeaways are:
The performance gap between DD2 and the teacher AR model is minimal. For VAR models across all model sizes, compressing the teacher model to 1 step and achieving up to 8. x speedup with an mere FID increase of less than 2.5. For LlamaGen models, DD2 achieves 238 speed-up in a FID increase of only 4.48. Such performance drop is acceptable.
DD2 outperforms the strongest baseline DD1 significantly. For VAR models across all model sizes, DD2 decreases the performance gap between the teacher AR model and one-step model by up to 67% compared DDI. DD2 even outperforms the step sampling results of DD1 by large margin for all VAR models. For LlamaGen model, DD2 also achieves a 2.76 better FID than DD1, All weak baselines fail to generate in one-step. These results show the effectiveness of DD2.
5.3 Results of Multi-step Sampling
is better to offer a smoother trade-off curve between quality and step. To achieve this, we use the teacher model to refine the last several AR positions of the generated content. The detailed algorithm is shown at Alg. 4. Results are reported in Tab. 2. For DD1 baseline, we use its default multistep sampling schedule and control the number of sampling steps to ensure fair comparison. The sample quality increases consistently with more sampling steps, offering more choices for the users.
5.4 Training Efficiency
In addition its superior performance over DDI, DD2 offers another significant advantage: much faster convergence. As shown in Tab. 3, DD2 requires substantially much fewer GPU hours to train, achieving up to a 12.3x training speedup while having better performance than DDI. Detailed analysis of DD2's 's training cost can be found at App. B.4.
8

---

Table Generative performance on class-conditional ImageNet-256. "#Step" indicates the number of model inference to generate one image. "Time" is the wall-time of generating one image in the steady state, Results with are taken from [21]. while * denotes results obtained with more training.
Type
Model
FID
IS1
Pret
Rect
#Para
#Step
Time
GAN!
StyleGan-XL
2.30
265.1
0.78
0.53
166M
0.3
Dift.
ADM [5]
10.94
101.0
0.69
0.63
554M
250
168
Dift.
LDM-4-G [28]
3.60
247.7
400M
250
Dift.
DIT-L/2 [25]
5.02
167.2
0.75
0.57
458M
250
31
1
MaskGIT [2]
6.18
182.1
0.80
0.51
227M
0.5
ART
VQGAN [6]
15.78
74.3
1,4B
256
24
ART
ViTVQ [45]
4.17
175.1
1.7B
1024
24
ART
RQTran. [15]
7.55
134.0
3.8B
21
AR
VAR-d16 [37]
4.15
278.7
0.85
0.41
310M
10
0.133
AR
VAR-d20 [37]
3.40
305.1
0.84
0.47
600M
0.184
AR
VAR-d24 [37]
2.86
312.9
0.82
0.51
1.03B
0.251
AR
LlamaGen-L. [35]
4.11
283.5
0.85
0.48
343M
256
5.01
Weak Buseline'
VAR-skip-2
40.09
56.8
0.46
0.50
310M
0.098
Weak Buseline"
VAR-onestep*
157.5
Weak Baseline"
LlamaGen 156
80.72
12.13
0.17
0.20
343M
100
1.95
Weak Baseline"
LlamaGen-onestep
220.2
DD1
VAR-d16
9.94
193.6
0.80
0.37
327M
0.021
DDI
VAR-d16
7.82
197.0
0.80
0.41
327M
0.036
DDI
VAR-d20
9.55
197.2
0.78
0.38
635M
0.027
DDI
VAR-d20
7.33
204.5
0.82
0.40
635M
0.047
DDI
VAR-d24
8.92
202.8
0.78
0.39
1.09B
0.034
DDI
VAR-d24
6.95
222.5
0.83
0.43
1.09B
0.059
DDI
LlamaGen-L
11.35
193.6
0.81
0.30
326M
0.023
DD1
LlamaGen-L
7.58
237.5
0.84
0.37
326M
0.043
DD2 (ours)
VAR-d16
6.21
213.0
0.84
0.39
329M
0.019 (7.0x)
DD2 (ours)
VAR-d20
5.43
233.7
0.85
0.41
619M
0.023 (8.0x)
DD2 (ours)
VAR-d24
5.06
254.7
0.85
0.39
1.04B
0.031 (8.1x)
DD2 (ours)
VAR-d24*
4.91
282.2
0.87
0.39
1.04B
0.031 (8.1))
DD2 (ours)
LlamaGen-L.
8.59
229.1
0.77
0.32
335M
0.021 (238x)
DD2 (ours)
LlamaGen-LL*
7.58
238.7
0.77
0.34
335M
0.021 (238x)
Table 2: Generation quality of involving the pre-trained AR model when sampling. The notation
pre- pre-trained-n- means that the pre-trained AR model is used to re-generate the n + 1-th to m-th tokens in the sequence generated in the first step by the few-step generator.
Type
Model
FID
1st
Pret
Rec
#Para
#Stop
VAR-d16 [37]
230.2
0.84
0.48
310M
10
DDI
VAR-d16.pre pn-ttne---
6.54
210.8
0.83
0.42
327M
3
DDI
AnR-ddl--rr--miieee--333 3-5
5.47
230.5
0.84
0.43
327M
4
DDI
'AR-d16-pr--miine---55
5.03
242.8
0.84
0.45
327M
6
DD2 (ours)
VAR- (ru-l---r---riied---.
5.24
238.9
0.85
0.40
329M
3
DD2 (ours)
VAR-dI6-pre-miined-----
4.88
248.7
0.86
0.41
329M
DD2 (ours)
VAR-d16-pr--tuined-----0
4.47
277.8
0.87
0.42
329M
5.5
Ablation Study: the Importance of Initialization
As discussed in Sec. 3.3, initialization is dispensable for DD2. We provide results on LlamaGen-L and VAR-d24 models verify this, by only initializing one of the generator and guidance network with the tuned AR-diffusion model, while the other uses the backbone from the teacher AR model and a randomly initialized output head. Results are shown Tab. 4. We find that missing proper initialization for either of them can lead to significant performance degradation or even collapse, highlighting the importance of good initialization for both components.
6 Discussions
6.1 Distinction with diffusion score distillation
In this section, we discuss the differences between score distillation for diffusion models and DD2.
Fundamentally Different Task. Traditional score distillation methods aim to reduce the number of sampling steps for diffusion models. In contrast, our work focuses on one-step sampling for pre-trained AR models, which is a fundamentally different generative paradigm from diffusion models. Despite the competitive or even superior performance of AR models compared to diffusion models, one-step sampling for AR models is under explored, which highlights the contribution of DD2.

---

Table 4: Impact of Initialization.
Table 3: Training cost of DD2 and speed-up com- pared with DD1. All experiments are done 8 NVIDIA A800 GPUs.
Parum
FID-Sk
v
LlamGGestt. 335M
14.77
x
Llumakies-L.
33564
16.08
Method
Model
Param
Cost (SxGUU)
Speod-up
4
LlamaGen-L
335M
21,76
DDI
VAR-d16
327M
296.9
1x
/
VAR-d24
1.48
11.53
DDI
/
x
VAR-424
1,04
(001apeets200)
DDI
DD1
LlamaCien-L
VAR-024
326M
1.998
647.7
604.2
1x
VAR
L.:B
Collapset=200)
DD2 (ours)
VAR-d16
329M
115.5
2,6%
Table 5: Perceptual path length of DD2 and DD1.
DD2 (ours)
VAR-d20
619M
174.4
2,8%
(ours)
VAR-d24
1,048
96.1
6,3%
DD1
DD2
DD2 (ours)
LlamaGie-LL
335M
52.6
12.3%
PPLL 18437.6
7231.9
Technical Adaptations. Directly applying standard score distillation to AR models is not feasible. We have made multiple technical innovations to tackle this problem: (1) train the guidance network to learn the conditional score of the generator instead of the score, (2) we replace the classification layer in AR models with MLP head to ensure continuous output, and (3) propose to adapting the pre-trained AR model into an AR-diffusion model as an initialization. We further replace the standard AR-diffusion loss with our GTS loss for better convergence of this process.
Our strong results offer a new perspective on training one-step generative models. Currently, the dominant strategy for training such models focuses diffusion- frameworks. In contrast, method demonstrates that distilling an AR model is also a highly competitive approach, as our results surpass many representative diffusion distillation techniques shown in Tab. 6.
6.2 Benefits of Eliminating Pre-defined Mapping in DD1
Compared to DD1, key feature of DD2 is that does not rely on any pre-defined mapping, which brings several potential benefits: (1) More efficient utilization of model knowledge. In DD1, the pre-defined mapping provides only single end-to-end signal, whereas DD2 explicitly trains the model at every token position, offering a more fine- aneeggrined supervisory signal. (2) Reduced accumulation of errors. DDI, if the model fails to correctly learn the noise-to-data mapping at a certain position, this error propagates to subsequent positions because their conditions depend on carlier predictions. In contrast, in DD2, the teacher model provides ground-truth distributions for each token position based on the generator's 's current condition. Since the teacher model possesses strong generalization ability, the impact of imperfect conditions is greatly mitigated. (3) Smoother latent representations. More generally, training generative models without pre-defined mappings allows them to automatically discover smoother latent representations of the target data distribution, which benefits learning because smoother representations are easier to optimize. To quantify this property, we measure the Perceptual Path Length (PPL. metric [12] for both DD2 and DD1, where a lower value indicates smoother interpolation in the latent space. As shown in Tab. 5, DD2 achieves significantly smoother latent interpolation than DDI.
Future Works and Limitations
Compatibility with Image AR Models without VQ. In addition to the commonly used discrete- space AR models based on VQ hidden space, continuous-space AR models [17] has recently gained increasing popularity. These models generate each token through a diffusion process. Our method is naturally compatible with such models as well, since they directly provide the conditional score. We leave the application of our approach to this class of models as future work.
Scaling to Larger Tasks, Image AR models have also been used in larger-scale tasks, such as text-to-image task [9. 47]. Extending our method to these models offers practical impact.
Performance Gap to the Teacher Model. Although DD2 achieves significant speedup, the distilled models still exhibit certain performance gap compared the original AR models. Addressing this performance drop to make one-step AR models match or even surpass the quality of pretrained AR models remains an important and promising direction for future research.
Acknowledgement
This work was supported by National Natural Science Foundation of China (62506197, No. 62325405, 62104128, U19B2019, U21B2031, 61832007, 62204164, 92364201), Tsinghua EE Xilinx Al Re- search Fund, and Beijing National Research Center for Information Science and Technology (BNRist). We would like to thank all anonymous reviewers for their suggestions. We also thank all the support from InIinigence-AL.
10

---

References
[1] Tianle Cai, Yuhong Li, Zhengyang Geng. Hongwu Peng. Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple I1m inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774 2024.
[2] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[3] Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autoregressive generative model. In International conference machine learning, pages 864-872. PMLR, 2018.
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[5] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
[6] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[7] Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. arXiv preprint arXiv:2410.12533 2024.
[8] Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 13:41441994,4, 2020.
[9] Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. arXiv preprint arXiv:2412.0443) 2024.
[10] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:684006851, 2020.
[11] Doohyuk Jang, Sihwan Park, June Yong Yang. Yeonsung Jung, Jihun Yun, Souvik Kundu, Sung- Yub Kim, and Eunho Yang. Lantern: Accelerating visual autoregressive models with relaxed speculative decoding. arXiv preprint arXiv:2410.03355 2024.
[12] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pages 1401-4410 2019.
[13] Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji. and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279 2023.
[14] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
[15] Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523-11532 2022.
[16) Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2142-2152, 2023.
[17] Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint 2XXx:2446.11888 2024.
11

---

[18] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXi::201.15007, 2024.
[19] Zinan Lin, Kiran Thekumparampil. Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcen- trality: Self-supervised model training and selection for disentangling gans. In international conference on machine learning, pages 6127-6139. PMLR, 2020.
[20] Yaron Lipman, Ricky TQ Chen, Heli Ben- Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[21] Enshu Liu, Xuefei Ning, Yu Wang, and Zinan Lin. Distilled decoding I: One-step sampling of image rego-segressive models with flow matching. arXiv preprint arXi::2412.17153. 2024.
[22] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 2022.
[23] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388, 2021.
[24] Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36:76525-76546, 2023.
[25] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[26] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 1498888 2022.
[27] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances neural information processing systems, 32, 2019.
[28] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[29] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint 12X 2022.
[30] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[31] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[32] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310. /4189, 2023.
[33] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, arXiv preprint arXiv:2303.014669 2023.
[34] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456, 2020.
[35] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525. 2024.
[36] Yao Teng, Han Shi, Xian Liu, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. Accelerating rego-segeeseive text-to-image generation with training-fres free speculative jacobi decoding, arXiv preprint 240.0994,, 2024.
12

---

[37] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive mod- eling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905 2024.
[38] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, al. Con- ditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
[39] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[40] Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 118800688441 2023.
[41] Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909-3925, 2023.
[42] Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and Bill Freeman. Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems, 37:47455-47487, 2024.
[43] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6613-6623, 2024.
[44] Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional fast causal video generators. arXiv preprint arXiv:2412.07772. 2024.
[45] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:21 00.04627, 2021.
[46] Mingtian Zhang, Jiajun He, Wenlin Chen, Zijing Ou, José Miguel Hernández-Lobato, Bernhard Schölkopf, and David Barber. Towards training one-step diffusion models without distillation. arXiv preprint wrXiv:2502.08005. 2025.
[47] Qian Zhang, Xiangzi Dai, Ninghua Yang, Xiang An, Ziyong Feng, and Xingyu Ren. Var-clip: Text-to-image generator with visual auto-regressive modeling. arXiv preprint 1rX11.2408.01181 2024.
[48] Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin. and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims are supported by experimental results in Sec. 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
13

---

The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: See App. A.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
14

---

Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Sec. 3, App. and App. E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification:
We have not released our code yet.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
www.abiisssooooooiii...... for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new spun-source benchmark).
15

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: for more details.
/ /nips. /ttpp.cccppblliiii.icc w/w.idgs////////eeooco Caee//Coduubmiicc
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Sec. 3, App. C Cnd App. E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Some of our baselines are taken from previous papers, which do not contain eтror bars. Besides, believe that the metrics used in our paper not significantly affected by randomness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
16

---

8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Tab. 1, Tab. 3 for the computational cost.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp://///eu.....eeeee cc http//uulcc/tttieeeeeeeeee e
Answer:
[Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: Our method is an acceleration method for AR models, which is not related to any domains that may have societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
17

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All models and datasets are properly cited and discussed in Sec. 5,
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ com/datssete has curated licenses for some datasets. Their licensing guide can help determine license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have not released any assets yet.
Guidelines:
The answer NA means that the paper does not release new assets.
18

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development this research does not involve LLMS as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
19

---

Please refer to our LLM policy o0t.rp/s././//fffeeeeee////////1 for what should or should be described.
20

---

A Discussion of Prop.
In this section, we provide the proof of Prop. using a simple induction.
Proof. Assuming the neural network has sufficiently large capacity, minimizing LOSD is then equiv- alent to minimizing each individual term LeSD, d(s (1. (1(80((<))) 8oae (9 (1(8g(1<))) for any i.
Base case (i - 1): At the first position = 1. the corresponding term LOSDI ((s t1). Saa t2)) degrades to traditional score distillation loss, that minimizing this term guarantees (1( = (4()1)
Inductive Hypothesis (i models the distribution of all tokens before this position: po (g(c))) = po (gc*))
1): Assume that for token position = k. the generator correctly
Inductive Step k) Minimizing LCSDK d(s (g (x]sg(9<<))) 8fak (9k ,t (x(sg((<))) ( so that pol9<k+1) = po(a pekoo
guarantees po
po (gx (<<)4pppppp (g<x) = pa (gck+1) holds
Conclusion: By mathematical induction, minimizing LESD guarantees po(a....... pa (o0(9
B More Experimental Details
In this section, we provide some additional experimental settings and results.
B.1
Comparison with Diffusion Distillation Methods
We compare DD2 with several commonly used diffusion distillation methods to show our effectiveness. Results are shown in Tab. 6.
Table 6: Comparison between DD2 and diffusion distillation methods. Results are taken from the paper of Shortcut model [7].
VAR-VI
DD2-VAR-D2
DD2- VAR-d44
PD[29]
CD[33]
CT(33]
Reflow [22]
Shorteut[]]
FID
6.21
5.43
4.91
35.6
136.5
69.7
44,8
10.6
B.2 Training Curve of the Original DMD Method
We demonstrate our reproduced FID-iteration curve of the original DMD method with different initialization setting in Fig. 6. As discussed in Sec. 3.3, an inappropriate score distillation initialization strategy can lead to slower convergence and bad training stability, which motivates us to use the AR-di Riffifuusoon model for initialization.
B.3 Performance of the AR-diffusion Model for Initialization
Table Performance of AR-diffusion models. FIDs are evaluated wit 5k generated images.
Loss
Model
Param
FID-5k
Training Iter
GTS
VAR-d16
329M
11.41
330k
GTS
VAR-d20
619M
11.41
330k
GTS
VAR-d24
1.04B
11.24
230k
GTS
LlamaGen-L
335M
15.66
100k
diffusion
VAR-d16
329M
17.98
500k
21

---

FID Score over Different Initialization Methods
500
random jnii_nniittti ceit__gn__aaee jast_layer
random jnit _init__ennnn
fandom_init_com
default
FID Score
200
100
5000
10000
15000
20000
25000
30080
35000
40000
Train Step
Figure Training results for different initialization strategies. default indicates that both the generator and the guidance components are initialized using the teacher model [43], refers the setting where only the last layer of the generator is randomly initialized, while the rest of the generator and the guidance are initialized from the teacher model. random init leenggnnn means the entire generator is randomly initialized, whereas the guidance is initialized from the teacher model. random_ _i____l_____ denotes the case where only the guidance is randomly initialized, with the generator initialized from the teacher model.
_i__jnniiigee aitit_ge_alsst fayt_layer
Table 8: Wall time (hour) of each training stage under different settings. All time costs are profiled on 8 NVIDIA A800 GPUs.
Model
VAR-d16
VAR-d20
VAR-d22
LlamaGen-L
Continuous Adaptation
3.3
AR-diffusion Tuning (Head Only)
60.1
72.6
59,9
AR-diffusion Tuning (Full Model)
8.0
9.1
17.2
8.6
Main Training
42.6
88.0
19.0
40.7
Performance Alignment
4.8
4.7
Overall
115.5
174.4
96.1
52.6
We report the performance of the tuned AR-diffusion Model in Tab. 7, which serves as the initialization of both generator and guidance network. We use a 10-step Euler solver for the sampling process of every token. To verify the effectiveness of the proposed GTS loss Eq. (6), we also report the results of using traditional diffusion loss where the target a Monte Carol the ground truth score. The key takeaways are: (1) the tuned AR-diffusion model demonstrates strong sample quality, making a suitable choice for initialization, and (2) performance degrades significantly we use traditional diffusion loss, highlighting the effectiveness and necessity of our GTS loss.
B.4 Training Details and Cost
The training mainly cost consists of two parts: AR-diffusion tuning and the main training process with CSD loss.
In the first stage, we use the same setting in evaluation to generate the training data. Specifically, for VAR models, we apply top_k as 900 and classifier-free guidance scale as 2.0; for LlamaGen model we use top_k k as 8000 and classifier-free freeigddnceee scale as 2.0.
22

---

Table Converged performance before/after using performance alignment operation.
Type
Model
FID|
1St
Pret
Reef
#Para
#Step
DD2 (Before)
VAR-d16
8.35
176.7
0.80
0.42
329M
DD2 (Before)
VAR-d20
6.57
201.4
0.81
0.43
618M
DD2 (After)
VAR-d16
6.21
213.0
0.84
0.39
329M
1
DD2 (After)
VAR-d20
5.43
233.7
0.85
0.41
618M
1
For VAR models, the first stage, we first fine-tune only the output head, and subsequently train the entire model. For the second part, we apply performance alignment dl6 and d20 models, which takes additional cost of training the guidance network. We list the cost of each part in Tab. 8.
For LlamaGen model, there is an additional stage where we fine-tune the teacher model to support continuous input. For AR-diffusion tuning, we directly train the entire model. We don't apply performance alignment. Detailed results are listed in Tab. 8.
B.5 The Effectiveness of Performance Alignment
provide the convergence performance of the d16 and d20 models before and after performing performance alignment procedure in Tab. 9, demonstrating the effectiveness of this operation.
C
Implementation Techniques
In this section, we introduce several techniques we use in our pipeline.
C.1 Computing Score Function with Probability Vector
In this section, we derive Eq. (1) starting from the preliminaries of flow matching.
Flow matching [20, 22] defines an invertible transformation with ordinary differential equation (ODE) dr Viz,, t)dl between two distributions to(r) and (x). The velocity function under linear noise schedule vt
1)20 + (x) can be given as:
V Vi t)
Ez (m (1 (21 x1 t) + fx =x),,
where no.1 (20, x1) is any joint distribution that satisfies temporal boundary conditions at both ends: 1 21 dro ni (11) and a,,1 (20, 21 )dx (zoo)
Since the source distribution ni (x) Gaussian distribution here, the relationship between the score function s(r,, t) and the velocity V(x,, t) is as follows:
o(z,, t) = (s(1,, 1) xt.
(8)
1-t
And the score function can be given as:
s(2,, t) E 1 1(1 t)zu + tæa =xe)
(9)
In our problem, the target distribution is weighted sum of Dirac functions: To(2) = Ey= P;0(z Gj), and is independent of the source distribution, resulting in only a finite number of possibilities. Therefore, we only need compute the product of the source distribution probability and the target distribution probability for each possible case, and then use this the weight function to compute the expectation of ø1, With the above explanations, we can easily arrive at Eq. (1).
C.2 Multiple Noisy Samples for Fake Conditional Score Learning
As discussed in Sec. 3.2.2, the output of the guidance network sy (g (g<s) is computed in two stages. First, transformer backbone process the input sequence (f).... . (n) and outputs
a feature sequence (fi....,fw fn), where each f, is causally conditioned q:z.. Then, a lightweight MLP takes
23

---

the noisy token q'' timestep is and the feature f; as input, then outputs the estimated conditional score ( 4: g<)).
Every conditioning f; defines continuous distribution over noisy inputs (x,z t), and the model must learn to predict score function across the entire space and all timesteps. To ensure sufficient training and improve generalization, we draw inspiration from the MAR implementation (see https: Hom/LTH14// /mar for more details) and apply a multi-sample training strategy. Specifically,
/ //github for a generated sequence (91..... ........ 9n), we sample multiple noise sequences (c1, (cI...... E.......... to
create multiple noisy versions of the generated sequence. For each noisy sequence, we apply Eq. (5)
as the loss function and take the average across all m samples. The resulting training objective is:
m
LFCs = =E )mppp
455 tasll<s) - V g.ss log log Th *aa)) (0,
(10)
where fú is the j-th sample at the i-th token position, 9:" denotes the j-th noise sample at the i-th token position.
ti = (1 tz,59g + Ci with <~ (((;I)
C.3 Performance Alignment for both Generator and Guidance network
We find there are two issues during training: (1) there is discrepancy between the guidance network and the generator's 's score, and (2) unstable training dynamics. Specifically, we observe that the samples generated by the conditional guidance network via AR-diffusion tend to under-perform those generated by the generator, indicating training gap of the guidance network. Additionally, we notice that the generator's FID fluctuates significantly during training. However, applying the Exponential Moving Average (EMA) technique to the generator leads to a much more stable performance curve. These findings motivate us introduce performance alignment procedure for both generator and guidance network after an initial phase of training. Specifically, (1) for the generator, we replace the regular model weights directly with EMA weights, then (2) we fix the generator and only train the conditional guidance network for a certain period to adapt to the new generator distribution. Once this alignment process is complete, that this technique is particularly helpful for training VAR-d16 and VAR-d20 models, significantly improving performance even after the models have already converged. Results shown at Tab. 9.
resume the standard training process. We empirically found
C.4 Larger Update Frequency for the Guidance Network
Training the guidance network is crucial, as it is responsible for producing accurate conditional scores of the generator distribution. However, this task is challenging because the generator distribution is also evolving during training. To address this issue, we adopt higher update frequency for the guidance network, following the strategy used in DMD2 [42]. Specifically, in each training iteration, we update the guidance network K times with (K >> 1) while updating the generator only once. The specific values of K used for different models are provided in App. E.
C.5 Details of Model Architectures
We the same architecture for both generator and guidance network. Inspired by MAR [17], our model architecture consists of a transformer backhone and a lightweight MIP head. As discussed in Sec. 3. for the guidance network, the transformer backbone takes the token sequence as input and output a causal feature sequence, while the MLP head takes the feature, noisy token and timestep as input and output the predicted conditional score. For the generator, the transformer backbone takes the shifted noise sequence as input, while the MLP head takes the noise sequence and the feature sequence as input and give the final generated sequence. We demonstrate the model architectures in Fig. 7.
D Algorithms of Multi-step Sampling Method
In this section, we present the pseudo algorithm of the multi-step sampling method in Sec. 3. Suppose we have a sequence = (q)...., .,, gn). We denote the indexing operations X[(] = q: and
X t| (gI.....:--1. The pscudo algorithm is presented in Alg. 4, with results reported in Tab. 2.
24

---

Teacher AR model
Guidance Network
One-step Generator
Condition Tokens
Condition Tokens
Noise Tokens (Shitted)
Transformer Backbone
Noisy token
Timestep
Feature
Noise Tokens
T-1
T
Output Head
Pn
Probability of the next token
Conditional Score
Target Sequence
s(q', fl9123)
Figure 7: Demonstration of the model architectures and the corresponding inputs/outputs.
Table 10: Hyperparameters used for AR-diffusion model tuning in the initialization phase. Actual BS/iter refers the actual batch size used in each training iteration. The training iterations for VAR are reported in the format of only head + full model' to reflect the two-phase training procedure.
Hyperparameter
VAR-d16
VAR-d22
VAR-d24
LlamaGen-L.
Learning Rate
2e-4
2e-4
2e-4
le-4
Batch Size
512
512
512
512
Grad Accumulation
4
16
Actual BS/iter
128
64
32
128
Adam 80
0.9
0.9
0.9
0.9
Adam B1
0.95
0.95
0.95
0.95
Training Iterations
300k+30k
300k+30k
200k+30k
100k
Algorithm Sampling with the teacher AR model
Require: The distilled one-step model 6 the pre-trained AR model 4 total sampling steps k: >> 1.
Sampling Process
1: X (91......00 .n. one-step sampling by
for (n + 2.....17 n) do
3: Sample o/ ~ pe p (-1x(1
4: [({]] + g:
5: end for
6: return X
Detailed Experimental Settings
In this section, we present the settings of our training process.
E.1 Model Parameterization
In this work, we parameterize the network as velocity prediction network, due to its widely verified good properties. Specifically, we use v( d) = (,s(2,,t)=2 to convert between velocity and score. We use the velocity function train the AR-diffusion model and the guidance network. For the generator, 2g Ug its final output, where is the input noise and is the model output.
25

---

Table 11: Hyperparameters used for DD2. Actual BS/iter refers to the actual batch size used each training iteration. "Gen" and "Gui" stand for the generator and guidance network, respectively. The training iterations of the generator for VAR-d16 and VAR-d20 are reported in the format of "before performance alignment + after performance alignment", while the training iterations of the guidance network for VAR-d16 and VAR-d20 are reported in the format of "before performance alignment + alignment + after performance alignment".
Hyperparaneter
VAR-d15
VAR-d20
VAR-d24
LlamaGenn
Gui
Gen 6
Gui
Gen #
Gni
Gen
Gni
Gea
Start Learning sale
le-6
le-6
le.6
Ic-6
lc.6
le-6
Ie-6
Ie-6
End Learning
2e-4
2e-4
2o-4
2e-1
4c-4
5c-5
2e-4
le-4
Batch size
512
512
512
1924
1024
1024
Accumulation
4
4
8
Actual BS/iter
128
64
128
Adumn lpp
0.9
0.9
0.9
0.9
0.9
Adam Bi
0.95
0.95
0.95
0.95
0.95
0.95
0.95
0.95
Training ilerations
10k+Sk+15k 15k
8k+7.5k
80k+8k+16k
16k-18% 18k
30R
15k
60k
12k
Gokdance Update Froq
S(Refore Align), 2(After Align)
S(Bafore Align), Z/After Align)
2
E.2 Stage I: AR-diffusion Tuning
For VAR models, we first freeze the transformer backbone and tune the output MLP for certain iterations. Then we remove the constraints and tune all parameters. For LlamaGen model, we directly tune all parameters from the start. Settings are listed in Tab. I0.
E.3 Stage 2: Training with CSD Loss
Calculation the Real Conditional Score We follow the default sampling settings of original AR models for probability vector calculation. For VAR models, we set classifier-free guidance scale to 2.0, top-k to 900 and top-p to 0.95. For LlamaGen model, we set classifier-free guidance scale to 2.0, top-k to 900 and top-p to 1.0.
Optimization Settings We list the optimization settings in Tab. 11. For the learning rate of the generator, we apply a linear warm up strategy from the start learning rate to the end learning rate 40K guidance network training iterations for VAR-d16, VAR-d20 and LlamaGen--L models. For VAR-d24, we set the warm up length as 20K.
EMA We find that EMA is very important for the stability of training. Since the model performs badly at the beginning of the training process, we use a progressive EMA rate. Specifically, we use small EMA rate in the early stage of training. Then use dynamic EMA rate miin(..9999) (iter 1)/ (iler + 10)), where iler is the training iteration. This progressive EMA schedule ensures both training stability and fast convergence.
E.4 Continuous Input Adaptation for LlamaGen Models
For VAR teacher models, we directly conduct our workflow since they naturally support continuous embeddings as input. However, for LlamaGen teacher model, we need modify the model's input head to accept continuous latent embeddings instead of discrete token indices. This allows the model to handle the continuous outputs from the generator. Specifically, we replace the original nn.embedding layer emb with a MLP mlp. We first train this MLP with loss ΣV= [Imlp(c,)) emb(i)]l where i the token and V the total number of tokens in the codebook. This process is fast, but incurs performance loss. Then we fine-tune the whole model with standard AR loss implemented by LlamaGen for 200K iterations, to align its performance with the original model. Finally, we obtained an AR LlaaaGen model that incurs performance loss and supports continuous embeddings as input, which we use as the teacher model.
E.5 Baselines
Set Prediction Method. Set prediction is a commonly used technique for reducing the sampling steps of Image AR models [2, 37, 17]. However, it is fundamentally incapable of reducing the sampling process to single step. As pointed out in the DD1 paper [21], when training a model under the one-step sampling setting with this approach, the optimal solution is equivalent to independently sampling each token according to the overall token frequency at this position in the dataset. Therefore.
26

---

we can directly evaluate its one-step generation performance without actually training model. This method completely ignores the dependencies between tokens, which leads to the failure case reported in the onestep* R rows of Tab. 1.
Pre-trained AR Models. For VAR models, top_k, top_ and clasifier-free-guuddnne scale as 900, 0.95 and 1.5, respectively. For LlamaGen models, we use 8000 as top_ 1.0 as top_p, and 2.0 as classifier-free-ee guidance scale
F
Visualizations
We show some generated examples in Figs. 8 to 11.
Figure 8: Generated by DD2- VAR-d16 model.
27

---

Figure 9: Generated by DD2-VAR-d23 VAR-d20 model.
Figure 10: Generated by DD2-VAR-d22 model.
28

---

Figure 11: Generated by DD2-LlamaGen-- model.
29


Reviews:
[{"review_id": "jDef4cn8pm", "reviewer": "Reviewer_A7Mc", "summary": "The paper introduces Distilled Decoding 2 (DD2), which enables one-step sampling in image autoregressive models, addressing the slow generation speeds limitation in traditional token-by-token AR sampling. Unlike prior work such as DD1, which relies on learning an explicit mapping via flow matching, DD2 reframes the AR model as a conditional score model predicting gradients of log-probabilities in the latent embedding space. It proposes a Conditional Score Distillation loss that trains a one-step generator and a guidance network to match the conditional scores of the original AR model, ensuring the output distribution remains faithful to the teacher model without predefined mappings.", "questions": "1. For an autoregressive model, the entropy is usually higher at the beginning of the sentence and lower later in the sentence, which makes the scores of later tokens harder to model than those of the initial tokens. I am wondering whether any weighting scheme can be used for tokens in different positions. A simple choice is to normalize by the norm of the score function across the sequence length. This would be different from the score normalization used in SID (across diffusion time).\n\n2. Since you have access to the ground truth score of the teacher model, why not use that in equation 5 (like the loss of equation 6)? It should be more sample-efficient than denoising score matching loss.", "limitations": "1. The quality of the one-step generation can still be improved.\n2. The advantage compared to the one-step diffusion model (distilled from a diffusion model) is not very clear.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The quality and clarity are good. I commented other pros and cons below.\n\nStrengths:\n\n1. Usually, it is difficult to perform distillation from a clean, non-diffusion teacher model. In this case, since the teacher is a categorical distribution whose Gaussian convolution has a closed form (as used by DD1), this is a smart choice. It avoids the need for teacher score estimation, which is often required when the target distribution is an intractable EBM (e.g., [1]).\n\n2. The proposed conditional score distillation method is straightforward and natural.\n\n3. The most challenging part of this type of distillation research is that the student needs to be initialized from the teacher model, which is not straightforward when the architectures of the teacher and the student differ. It is great that the authors conducted a detailed study of this problem. The proposed method is heuristic but intuitive.\n\nWeaknesses:\nSee the questions section, especially question 2.", "was_revised": true, "final_justification": "My conerns are all resolved. I will keep my score. Hope the generation results can be furthur improved.", "rebuttal": "- Thank you for the insightful questions! Please see our detailed response below. We’re happy to continue the discussion if any concerns remain.\n\n> ### **Weakness 1: Whether weighting schemes can be applied to different token positions**\n  - Thank you again for raising this insightful point!\n  - In our default setting, we adopt a weighting scheme similar to SID[1], where each data sample is weighted by the norm of its score function. We use different weights for different data samples, which is different from your suggestion, where different weights are applied to each token position. To investigate this idea, we experimented with your proposed approach, where each token position is weighted by the norm of its score function.\n  - In addition, to further explore the impact of token-position-dependent weighting schemes on training, we also explored two static, token-wise weighting strategies:\n    - The weights linearly decrease from 3 to 1 across token positions from the first to the last.\n    - The weights linearly increase from 1 to 3 across token positions from the first to the last.\n  - We conducted experiments using the VAR-d16 model. Due to time constraints of the rebuttal period and limtied computational resources, we trained under the default setting with only 40,000 guidance iterations and 8,000 generator iterations. The results are summarized below:\n| Schedule | Default | Token Wise Norm | Static from 3 to 1 | Static from 1 to 3 |\n|:----------:|:---------:|:-----------------:|:--------------------:|:--------------------:|\n| FID      | 8.35    | 10.85           | **6.91**           | 10.44              |\n  - The results show that adjusting the token-wise loss weighting schedule is indeed beneficial. Assigning higher weights to tokens at the beginning of the sequence leads to much better performance compared to our default setting.\n    - **Possible reason**: During training, the generator's objective at each token position is to align the conditional distribution of that token (given the previously generated tokens) with the corresponding distribution provided by the pretrained autoregressive model. This implies that tokens earlier in the sequence are responsible for providing accurate conditioning for subsequent tokens. If these early tokens are not well-trained, the learning targets for later tokens may also become unreliable. Therefore, increasing the loss weight for early tokens can help improve their learning quality and, in turn, enhance overall training effectiveness.\n\n  [1] Zhou et al., Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation.\n\n> ### **Weakness 2: Why not use Eq. (5) to train the conditional guidance network?**\n  - Thank you for your careful reading!\n  - The loss in Eq. (6) is used to train a model to output the conditional score of the **teacher (i.e., the pre-trained AR model) distribution**. It relies on the pretrained AR model to provide the **ground truth conditional probability** of each token given all its previous tokens. This allows us to explicitly compute the ground-truth score for each token. \n  - In contrast, Eq. (5) defines the training objective for the guidance network, which aims to learn the conditional score of the **one-step generator distribution**. Since the one-step generator directly **outputs the generated sample** without explicitly modeling the conditional distribution of each token given the previous ones as the pretrained AR model does, we are unable to compute the exact ground-truth score for the generator.\n  - As a result, we follow the common practice in traditional diffusion and AR-diffusion frameworks and train the guidance network by approximating the score function using a Monte Carlo estimatation for each token's score, as formulated in Eq. (5).\n\n> ### **Limitation 1: The quality of the one-step generation can still be improved.**\n  - This is because our training process relies solely on the information provided by the pretrained AR model. Thus, the performance is inherently bounded by that model. Nevertheless, DD2 has already achieved a relatively small performance loss compraed to the pretrained AR model. We believe that incorporating real samples in the training process with adversarial objectives (e.g., GAN loss as used in [2][3]) could further overcome this limitation, which we leave for future work.\n\n  [2] Kim et al., Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion.\n\n  [3] Yin et al., Improved Distribution Matching Distillation for Fast Image Synthesis.\n\n> ### **Limitation 2: The advantage compared to the one-step diffusion model is not clear.**\n  - We compared several representative and recent methods in diffusion distillation on the ImageNet 256×256 benchmark, as shown below. Results of diffusion-based are taken from [4]. As can be seen, DD2 achieves significantly better performance compared to these methods.\n\n  |     | DD2-VAR-d16 (ours) | DD2-VAR-d20 (ours) | DD2-VAR-d24 (ours) | PD[5] | CD[6] | CT[6] | Reflow[7] | Shortcut Model[4] |\n  |-----|:--------------------:|:--------------------:|:--------------------:|:-------:|:-------:|:-------:|:-----------:|:-------------------:|\n  | FID |  6.21               | 5.43               | 4.91               | 35.6  | 136.5 | 69.7  | 44.8      | 10.6              |\n   - We would like to emphasize that the problem we aim to address is **fundamentally different from diffusion distillation**. Specifically, we focus on pre-trained AR models, while diffusion distillation is based on pre-trained diffusion models, which is a fundamentally different type of generative models. As a result, existing diffusion distillation methods cannot be directly applied to AR models. \n\n  - Notably, AR models have already achieved performance comparable to, or even surpassing, that of diffusion models [8][9][10]. Therefore, addressing the distillation of AR models presents an independently meaningful and valuable research direction.\n\n   [4] Frans et al., One Step Diffusion via Shortcut Models. \n\n   [5] Salimans & Ho, Progressive Distillation for Fast Sampling of Diffusion Models.\n\n   [6] Song et al., Consistency Models.\n\n   [7] Liu et al., Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow.\n\n   [8] Tian et al., Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. \n \n   [9] Sun et al., Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation.  \n\n   [10] Han et al., Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis."}, {"review_id": "1lmRhywTjB", "reviewer": "Reviewer_ddiT", "summary": "This paper proposes Distilled Decoding 2 (DD2), a method to accelerate sampling in image autoregressive (AR) models by enabling one-step generation. Unlike Distilled Decoding 1 (DD1), which relies on flow-matching and predefined mappings, DD2 reframes the AR model as a conditional score model. A Conditional Score Distillation (CSD) loss is introduced to match the token-level conditional scores between a pre-trained teacher AR model and a one-step generator. A separate guidance network is trained to approximate the conditional score of the generator, and the generator is optimized to minimize the divergence from the teacher’s score. The method significantly improves both training efficiency (up to 12.3× faster) and generation speed (up to 238× faster inference) with minimal degradation in sample quality on ImageNet-256 using strong AR backbones like VAR and LlamaGen.", "questions": "I think this paper is overall good. Please refer to the weakness part.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n1. The idea to view an AR model as a conditional score model and utilize it for training a one-step generator is quite novel and interesting.\n2. DD2 shows substantial improvements over DD1, reducing the FID gap between one-step and teacher AR models. \n3. Experiments span multiple strong AR backbones (VAR, LlamaGen) and analyze one-step and multi-step performance with meaningful baselines. A careful study of initialization reveals its importance in training stability, with a tailored strategy to reuse teacher model knowledge.\n\n**Weakness**:\n1. Only ImageNet-256 is used for evaluation. It would be better to test on larger resolutions and more diverse datasets, or conditional generation settings like text-to-image.", "was_revised": true, "final_justification": "This paper is overall good for me, so I will keep my rating. I hope the authors could provide more evaluation results, e.g., larger resolution.", "rebuttal": "- Thank you for your positive feedback! Please see our detailed response below. We’re happy to continue the discussion if any concerns remain.\n\n> ### **Weakness 1: Lack of large-scale experiments**\n  - Thank you for the suggestion! We agree that larger-scale experiments would further strengthen the paper.\n  - Due to the limited time and computational resources available during the rebuttal period, we are unable to conduct larger-scale experiments at this time. However, we plan to include them in the revision and future work.\n  - Additionally, we would like to offer the following clarification to address your concern: ImageNet-256 is a sufficiently large and diverse dataset, and generation on this dataset is considered challenging. It has been widely adopted as the main benchmark in many recent works on new generative paradigms [1][2][3][4][5]. Our choice of dataset follows the setting in DD1. Therefore, we believe that using ImageNet 256 provides a meaningful and representative evaluation of our method.\n\n  [1] Tian et al., Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction.  \n\n  [2] Sun et al., Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation.  \n\n  [3] Frans et al., One Step Diffusion via Shortcut Models. \n\n  [4] Liu et al., Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching.\n\n  [5] Geng et al., Mean Flows for One-step Generative Modeling."}, {"review_id": "rm3ORs0rG0", "reviewer": "Reviewer_cqPi", "summary": "Distilled Decoding 2 (DD2) is a method for collapsing an image autoregressive (AR) model’s multi-step sampling into a single forward pass without a predefined mapping. The key idea is to reinterpret the teacher AR model’s next-token probabilities as conditional score functions in the VQ embedding space, and then to train a one-step generator and  a conditional guidance network that estimates the generator’s own conditional score. Conditional Score Distillation (CSD) loss aligns the teacher and student score functions, minimizing this loss forces the student’s full-sequence distribution to match the teacher’s. The authors also introduce a two-stage initialization, fine-tuning an AR-diffusion model with a Ground-Truth Score (GTS) loss, to stabilize training. On ImageNet-256 with two strong AR backbones (VAR and LlamaGen), DD2 reduces sampling from 10 or 256 steps to one step with minimal FID increase from 3.40 to 5.43.", "questions": "see weaknesses, and\n1. Why do the student network and the guidance network both use the MAR architecture? Is this necessary, and could the paper’s improvements be due to MAR’s continuous representations?\n2. To verify the importance of initialization, the authors conduct diffusion distillation experiments on the ImageNet-64 dataset using the original DMD method. Why did you choose to use DMD for these experiments instead of validating it on CSD?\n3. Compared to DD1, DD2 eliminates the pre-defined mapping. However, the benefits of this change are not analyzed in the paper.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "strengths \n1. Compared with DD1, DD2 achieves much better one-step sample quality and dramatically faster training.\n2. The AR-diffusion fine-tuning with GTS loss is insightful and clearly shown to be crucial for stable, efficient distillation.\n3. The paper’s method is evaluated on multiple types of AR models (VAR, LlamaGen), which differ significantly across several key aspects. This demonstrates the effectiveness and generalizability of DD2.\n\nweaknesses\n1. The authors emphasize that their method is “fundamentally different from diffusion score distillation, with different goals and challenges from previous work”; however, this distinction is not clearly highlighted in the paper. In my opinion, the most critical aspect of applying score distillation methods to AR models is the definition of the AR model’s score function (or velocity), which has already been addressed in [1]. The subsequent methods presented in the paper seem largely similar to those in diffusion score distillation.\n2. The paper uses the GTS loss to fine-tune the teacher model due to structural mismatch. However, this seems to be caused by the use of MAR in the guidance and student model. If MAR is not used and the student model shares the same structure as the teacher model, fine-tuning the teacher model may not be necessary.\n3. In Eq (6), It seems that $s_\\psi(q_i^{t_i},t_i | q_{<i})$ should be  $s_\\Psi(q_i^{t_i},t_i | q_{<i})$. The authors should carefully review the formulas and symbols for possible errors.\n\n[1] Liu, Enshu, et al. \"Distilled decoding 1: One-step sampling of image auto-regressive models with flow matching.\" arXiv preprint arXiv:2412.17153 (2024).", "was_revised": true, "final_justification": "I thank the authors for their answers. I suggest that the authors explicitly explain in the final version of the paper why their method is fundamentally different from diffusion score distillation, as well as clarify the advantages of the statement in the abstract, \"Compared to DD1, DD2 eliminates the pre-defined mapping.\" This will help readers better understand the contributions of the paper. Based on this, I'll raise my score.", "rebuttal": "- Thank you for the insightful questions! Please see our detailed response below. We’re happy to continue the discussion if any concerns remain.\n\n> ### **Weakness 2 & Question 1: MAR architecture**\n\n   Our response is organized into three sub-questions:\n  1. **Why do the student network and the guidance network both use the MAR architecture?**\n      - The generator must use an MAR-style structure (i.e., using an MLP to output continuous embeddings instead of probability vectors), because a one-step generator must should directly produce continuous samples. Token-wise probabilities (as in AR models) lead to independent sampling at each position and prevent gradient from back propagating through the CSD loss. Thus, a differentiable MLP head is essential.\n      - The guidance model is trained to predict the conditional score of the generator’s distribution, which is also a continuous vector in the codebook embedding space. Therefore, it also requires an MLP architecture similar to MAR to operate in this continuous domain.\n  2. **Is AR-diffusion training necessary?**\n      - As explained above, AR-style models (outputting token probabilities) are unsuitable as generator or guidance networks in our framework. Hence, we cannot use a shared architecture between student and teacher. In this case, AR-diffusion initialization becomes important (see Tab.4), and the proposed GTS loss proves effective (Tab.5).\n  3. **Is the performance gain simply due to using MAR’s continuous representations?**\n      - We believe that the performance improvement is **not brought by the MAR architecture**. DD1 also employs an MLP head that outputs continuous representations like MAR (see Appendix B of [1]) for details). Since DD2 outperforms DD1 with similar architectural, we argue that the improvement stems from the new training strategy, not the continuous representations.\n\n  [1] Liu et al., Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching.\n\n> ### **Weakness 1: Distinction with diffusion score distillation**\n  - Thank you for your thoughtful question! We would like to take this opportunity to clarify the novelty and contributions of our work.\n  - **Technical Contribution/Novelty**:\n    - We discuss the high level non-trivial differences between our methods and diffusion score distillation & DD1 below:\n      - **Compared to Diffusion Score Distillation**:\n        - The problem we addressed is fundamentally different. Diffusion score distillation focuses on reducing the number of sampling steps in diffusion models. In contrast, our work targets one-step sampling for pre-trained AR models, which is under explored despite AR models achieving competitive or even superior performance compared to diffusion models[2][3][4]. We believe this makes our direction independently meaningful.\n        - To the best of our knowledge, no prior work has directly applied score distillation to AR models. We are the first to enables this application by introducing a way to compute conditional scores with AR models. Accordingly, our guidance network is redesigned from a diffusion-based model to an AR-diffusion model, which brings a non-trivial architectural and conceptual shift.\n        - From a high-level perspective, our method can be seen as decomposing score distillation into a sequence of AR steps. As discussed in Sec. 3.2.2, DD2 can be viewed as applying score distillation to each token position sequentially following the AR order, essentially splitting the original global distillation task into a series of simpler sub-tasks. This token-wise perspective is unique to our approach.\n      - **Compared to DD1**:\n        - While DD1 introduced a way to compute conditional velocities, it mainly used them to construct a static mapping and did not leverage this information in a more principled way. In contrast, DD2 explicitly integrates this information into the training objective, leading to more effective guidance.\n    - **Furthermore, directly applying score distillation to AR models is not feasible without several key adaptations**. Beyond our high-level idea, we have made multiple technical innovations that serve as standalone contributions:\n      - As discussed before, one-step generators must output generated samples directly, rather than probability distributions. Therefore, we replace the classification head of the pretrained AR model with an MLP that outputs continuous embeddings.\n      - We identify the sensitivity of diffusion-style distillation to initialization. To address this, we propose adapting the pre-trained discrete AR model into an AR-diffusion model as an initialization. This modification has proved important (see Tab 4).\n      - We observed that standard AR-diffusion loss is not efficient. Thus, we propose the GTS loss, which trains the AR-diffusion model using ground-truth scores as targets and converges much better (see Tab 5).\n\n  - **Significant Performance Gains**:\n    - As far as we know, DD1 is the only prior work that successfully compresses AR models into one-step generation. DD2 achieves **significantly better performance and lower training cost** compared to DD1, thus pushing the boundary of one-step AR generation by a large margin.\n  - **Broader Impact**:\n    - Our strong results offer a new perspective on training one-step generative models. Previously, the dominant strategy for training such models focused on diffusion-based frameworks. In contrast, our method demonstrates that following the AR schedule is also a highly competitive approach. As shown in the results below (adapted from the Shortcut model paper [5]), our method surpasses several representative and recent diffusion distillation techniques, offering an effective and scalable alternative.\n\n    ||DD2-VAR-d16|DD2-VAR-d20|DD2-VAR-d24|PD|CD|CT|Reflow|Shortcut|\n    |-----|:-------------:|:-------------:|:-------------:|:------:|:-------:|:------:|:--------:|:----------------:|\n    |FID|6.21|5.43|4.91|35.6|136.5|69.7|44.8|10.6|\n\n  [2] Tian et al., Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction.  \n\n  [3] Sun et al., Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation.  \n\n  [4] Han et al., Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis. \n\n  [5] Frans et al., One Step Diffusion via Shortcut Models. \n\n> ### **Weakness 3: Wrong symbol**\n  - Thank you for pointing this out! You're right — we should have used a different letter, such as $\\Psi$, to represent the AR-diffusion model, since $\\psi$ is already used for the guidance network. We will revise all related notations and double-check the formulas in the revision.\n\n> ### **Question 2: Why did you choose to use DMD for these experiments instead of validating it on CSD?**\n  - We have already validated the effectiveness of AR-diffusion initialization under our CSD loss setting in the **ablation studies**. As shown in **Tab 4**, omitting initialization for either the generator or the guidance network leads to performance degradation or even training collapse.\n  - In Sec 3.3, we intentionally use the original DMD setup to demonstrate that the sensitivity to initialization arises from the nature of **score distillation itself**, not from our specific modifications. This strongly motivates our introduction of AR-diffusion tuning as an effective initialization strategy.\n\n> ### **Question 3: The benefits of eliminating pre-defined mapping**\n  - We provide the following intuitive explanation for why removing the pre-defined mapping from DD1 leads to better training:\n    - More effective utilization of knowledge from the pre-trained model:\n The pre-defined mapping in DD1 provides only an end-to-end training signal. Due to the complexity of the mapping construction process, this signal is difficult for the model to learn. In contrast, DD2 directly aligns the score at each token position, providing a much richer and more localized supervisory signal.\n    - Reduced accumulation of error:\n      - DD1 constructs the mapping autoregressively, where the target token at each position depends not only on the noisy token at that position, but also on previously generated tokens. If the model fails to fit the mapping at one position, the errors will propagate to later positions, leading to cumulative error.\n      - In DD2, the model aligns the conditional distribution of the generator (given previous tokens) with that of the teacher at each position. Thanks to the strong generalization ability of the pretrained teacher model, it can still provide a meaningful conditional distribution even when the previous tokens are imperfect. This behavior is also evidenced by our **teacher-involved sampling results** shown in Table 2. As a result, DD2 suffers from significantly less error accumulation.\n  - In addition, more generally, training generative models without relying on pre-defined mappings allows the model to **discover smoother latent representations** of the target data distribution automatically. **This not only benefits training (because smoother representation is easier to learn), but also improves the model's ability to downstream tasks such as latent interpolation**, which is an important downstream task in generative modeling[6].\n    - To quantify this property, we adopt the **Perceptual Path Length (PPL)** metric proposed in [6], where a lower value indicates smoother interpolation in latent space. We evaluate this metric on both DD2 and DD1.\n\n    ||DD1|DD2|\n    |-----|:---------:|:--------:|\n    |PPL|18437.6|7231.9|\n\n    - The results are shown below, and clearly demonstrate that **DD2 achieves significantly smoother latent interpolation than DD1**. This is very likely because the pre-defined mapping in DD1 itself is not smooth, which inherently limits the smoothness of the learned latent representation.\n\n  [6] Karras et al., A Style-Based Generator Architecture for Generative Adversarial Networks."}, {"review_id": "F9d6UhMoQU", "reviewer": "Reviewer_qqvy", "summary": "This paper introduces Distilled Decoding 2 (DD2), a novel method for high-quality, one-step sampling of powerful but slow Image Auto-regressive (AR) models. DD2 re-interprets the teacher AR model as a conditional score provider and uses a novel Conditional Score Distillation (CSD) loss to train a one-step generator. This approach avoids the rigid mappings of prior work and significantly closes the performance gap to the original multi-step model, outperforming previous methods while also being much more efficient to train.", "questions": "See Weakness", "limitations": "Yes. The authors have a dedicated \"Future Works and Limitations\" section (Section 6) where they transparently discuss the remaining performance gap to the teacher model and identify promising directions for future work, such as applying the method to other types of AR models and larger tasks. The discussion is constructive and appropriate.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1.\tThe paper tackles a highly significant problem in generative modeling. Image AR models produce state-of-the-art results but are hindered by extremely slow, sequential inference. Enabling high-fidelity one-step generation makes these powerful models far more practical for real-world applications. The substantial improvement over the previous state-of-the-art (DD1) marks a major step forward in this domain.\n\n2.\tThe experimental results are comprehensive and convincing. The method is evaluated on a standard, large-scale benchmark (ImageNet-256) using multiple strong and diverse AR base models (VAR and LlamaGen). The comparisons against DD1 and other baselines are fair and clearly demonstrate the superiority of DD2 in terms of sample quality (FID) and training efficiency. The ablation study on the importance of initialization further strengthens the authors' claims.\n\nWeaknesses\n\n1.\tThe existing ablation studies in the paper are already quite convincing; however, including comparative results between the CSD loss function and other score distillation losses would make the work more comprehensive.\n\n2.\tAlthough the appendix provides detailed information, it would be beneficial to include a more high-level flowchart in the main Methods section to summarize the overall two-stage, three-step training process, giving readers a clearer and more intuitive understanding of the general framework.", "was_revised": true, "final_justification": "The authors have addressed my concerns, and I maintain my stance of accepting the paper. The authors should incorporate the revised content into the final version.", "rebuttal": "- Thank you for the insightful question! Please see our detailed response below. We’re happy to continue the discussion if any concerns remain.\n\n> ### **Weakness 1: Including comparative results between the CSD loss function and other score distillation losses**\n  - Thank you for your insightful suggestion!\n  - As we mentioned in Sec 3.2.2, our CSD loss is compatible with any valid score distillation loss function (e.g., DMD loss or SiD loss). In our main experiments, we used **SID loss with $\\alpha=1.0$** as the default setting.\n  - In our understanding, the reviewer is suggesting we compare against other types of score distillation losses. To address this, we provide additional results below comparing different loss functions. We choose SID loss with varying $\\alpha$ values and DMD loss as options. Due to time constraints of the rebuttal period and the limtied computational resources, all experiments are conducted under the default setting with 40,000 guidance iterations and 8,000 generator iterations.\n\n  |Model| SiD alpha=0.8 |SiD alpha=1.0 (default)|SiD alpha=1.2|DMD|\n  |------------|:---------------:|:-------------------------:|:---------------:|:-------:|\n  | VAR-d16 |8.28|8.35|**8.23**|15.70|\n  | LlamaGen-L |- |**7.95**|-|8.13|\n\n  - It can be seen that our method is relatively robust to different values of $\\alpha$ in the SiD loss. For DMD loss, while it performs worse on the VAR model, it achieves performance comparable to SiD loss on the LlamaGen model. Overall, we recommend our default setting based on SiD loss with $\\alpha=1.0$.\n\n> ### **Weakness 2: It would be beneficial to include a more high-level flowchart in the main Methods section**\n  - Thank you for the helpful suggestion! We agree that including a high-level flowchart would improve the clarity of our presentation.\n  - We have already illustrated the main components of our method in **Fig 5**. However, due to the limitation of space, we were unable to include the multi-stage training pipeline and other implementation details in this figure. We put them in the appendix instead. We will carefully consider your suggestion and revise the figure accordingly in the final version."}]

Author rebuttal:

Meta-review: This paper proposes a method for one-step sampling of autoregressive models using conditional score distillation. The approach is intuitive, efficient, and well-validated on ImageNet-256, showing large gains in both training and inference speed while preserving competitive quality. Reviewers were split initially, but concerns about novelty and clarity were largely addressed during rebuttal, leading to stronger support. Overall, the work is timely, impactful, and I recommend acceptance.
Decision: Accept (poster)