<figure><img src="image_1.png" /></figure>

HAL
open science

# Di[M]O: Distilling Masked Diffusion Models into One-step Generator

Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière, Vicky Kalogeiton

► To cite this version:

Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière, Vicky Kalogeiton. Di[M]O: Distilling Masked Diffusion Models into One-step Generator. 2025 International Conference on Computer Vision (ICCV 2025), Oct 2025, Hawaii, United States. hal-05413533

HAL Id: hal-05413533

https://hal.science/hal-05413533v1

Submitted on 12 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Di[M]O: Distilling Masked Diffusion Models into One-step Generator

Yuanzhi Zhu¹ Xi Wang¹ Stéphane Lathuilière² Vicky Kalogeiton¹

¹LIX, École Polytechnique, CNRS, IPP ²Inria, Univ. Grenoble Alpes, CNRS, LJK

https://yuanzhi-zhu.github.io/DiMO/

<figure><img src="image_3.png" /><figcaption>Figure 1. Unlike continuous Diffusion Models (DMs) that have been successfully distilled into one-step DM generators with performances competitive with the teacher, distilling Masked Diffusion Models (MDM) into one-step generator remains a challenge. In this paper, we propose the first one-step distillation method: Di[M]O for MDM, e.g., from the recent text-to-image masked diffusion model Meissonic [4]. We demonstrate that our Di[M]O can successfully distill the teacher model into a one-step generator, achieving competitive performance both quantitatively and qualitatively, while the teacher model's performance deteriorates rapidly with reduced generation steps, i.e., comparing our one-step results (large, bottom images) to 4-step teacher outputs (right corner of each image).</figcaption></figure>

## Abstract

Masked Diffusion Models (MDMs) have emerged as a powerful generative modeling technique. Despite their remarkable results, they typically suffer from slow inference with several steps. In this paper, we propose Di[M]O, a novel approach that distills masked diffusion models into a one-step generator. Di[M]O addresses two key challenges: (1) the intractability of using intermediate-step information for one-step generation, which we solve through token-level distribution matching that optimizes model output logits by an 'on-policy framework' with the help of an auxiliary model; and (2) the lack of entropy in the initial distribution, which we address through a token initialization strategy that imposes randomness while maintaining similarity to teacher training distribution. We show Di[M]O's effectiveness on both class-conditional and text-conditional image generation, impressively achieving performance competitive to multi-step teacher outputs while drastically reducing inference time. To our knowledge, we are the first to successfully achieve one-step distillation of masked diffusion models and the first to apply discrete distillation to text-to-image generation, opening new paths for efficient generative modeling.

## 1. Introduction

Recently, Masked Diffusion Models (MDMs) [3, 11, 40, 96] have emerged as a prominent framework in generative modeling, demonstrating strong performance in a broad spectrum of tasks. Unlike auto-regressive models that generate data sequentially, MDMs enable faster discrete data generation. Moreover, compared to continuous diffusion models [38], MDMs are particularly advantageous for visual modeling within multimodal frameworks, as they leverage a unified vocabulary to seamlessly integrate visual and textual content. Their versatility has enabled successful applications in image generation [4, 11, 12, 40, 43, 74], text generation [20, 27, 56, 67, 68], video generation [24, 104], multimodal modeling [39, 50, 108, 113], protein design [10, 28], audio synthesis [85], and motion generation [15, 75]. In addition, several algorithms have been proposed to guide the generation process in MDMs [49, 69, 80, 92, 97], further enhancing their flexibility and alignment with user-defined objectives.

Despite these advantages, current MDMs suffer from
low inference speed, i.e. high number of sampling steps required during generation [23]. Besides accelerated sampling techniques [8, 9, 47, 72, 81, 122] that yield good performances but tend to drastically deteriorate the generation with fewer inference steps, distillation techniques address this efficiency bottleneck. They transfer knowledge from a well-performing but slow teacher model to a student model capable of achieving comparable results with significantly fewer steps (even one). For this reason, distillation has been widely adopted for continuous diffusion models, typically with consistency-based [57, 100, 102] or distillation matching approaches [60, 115, 119]. These effectively reduce inference time while maintaining generation quality. However, they cannot be directly applied to masked diffusion due to fundamental differences in their mathematical formulation, i.e. the absence of a Probability Flow Ordinary Differential Equation (PF-ODE) and the inability to re-parameterize MDM outputs as denoising score functions [20, 56, 63]. Furthermore, recent work [99] points out that multi-token prediction in MDMs is inherently challenging due to their naïve conditional independence assumption.

To overcome this, the community is just starting to explore distillation alternatives for masked diffusion [20, 31]. [20, 31] are the only works attempting this. Hayakawa et al. [31] propose viewing MDMs as mixture models, progressively distilling dimension correlations into a few-step student. Meanwhile, Deschenaux et al. [20] collect the teacher's multi-step output log probability as a target for the student to learn with fewer steps. Despite their improved efficiency, these methods have critical limitations: both require a multi-round distillation process, which is computationally expensive and prone to error accumulation when further reducing student steps.

Instead, in this work, we directly address **one-step distillation for masked diffusion models**. While achieving this, we identified two main challenges. First, there is the difficulty of leveraging the intermediate steps information for one-step generator <fnref n="1" />. To distill the teacher's whole multi-step information into a *one-step* model, neither can we compute the loss between the teacher's intermediate outputs and intractable student intermediate ones, nor can we update the one-step generator using the loss gradients due to the non-differentiable sampling operation. Second, the lack of entropy in the initial distribution used during one-step generation [31]. While continuous diffusion models typically start with a Gaussian distribution that naturally incorporates randomness, MDM begins with a sequence of identical mask tokens. This lack of entropy can lead to mode collapse in one-step distillation, making effective distribution matching more difficult.

Therefore, in this work, we propose Distilling [M]asked

diffusion models into a **One-step generator**, coined Di[M]O, trained to transform an initial input distribution into outputs that closely resemble the teacher's multi-step generation distribution. Inspired by *data-free* on-policy distillation [1, 2, 5, 35, 37, 84, 93], Di[M]O matches the conditional *token-level* distribution along all intermediate states to ensure the overall generation distribution matching. Our first technical contribution is that Di[M]O approximates the training objective using pseudo-intermediate states combined with an auxiliary model that surrogates the gradient. Our second technical contribution addresses the insufficient entropy in the initial state of one-step generation. Di[M]O employs a token initialization strategy that injects randomness while maintaining similarity to the input sequences used to train the teacher, ensuring robust distillation while effectively avoiding mode collapse.

Using image generation as a representative example, we conduct experiments on class-conditional generation with MaskGit [11], and on text-conditional image generation with Meissonic [4] as the teacher model. Our results show that for both tasks, Di[M]O can efficiently distill the teacher model into a one-step generator while impressively achieving performance competitive with the multi-step outputs of the teachers. Our findings also contain training objective with generalized divergence, ablation studies discussing our design choices, limitations and future directions.

Our contributions can be summarized as: (1) We are the first to successfully achieve one-step distillation of masked diffusion models, paving the path for future work. (2) To address this, we propose Di[M]O, a on policy distillation method that enables the one-step distillation of masked diffusion models, with proposed efficient token initialization. (3) Our findings show that Di[M]O successfully reaches performance close to that of MDM teachers, while greatly enhances the sampling efficiency, verified by both class-conditional and text-conditional image generation tasks; note that, Di[M]O is the first to experiment with discrete distillation for text-to-image generation.

## 2. Related Works

**Acceleration of Masked Diffusion Models.** Improving the efficiency of MDMs has been an active area of research, with various methods proposed to accelerate inference while maintaining generation quality [8, 9, 47, 72, 81, 122]. For instance, Park et al. [72] minimize compounding decoding error to improve the allocation of discrete sampling timesteps, achieving efficiency gains without additional computational overhead. Ren et al. [81] introduce a higher-order solver that allows for larger step sizes while reducing numerical errors, enabling faster generation with minimal quality loss. Other works have explored alternative strategies to enhance efficiency. Hayakawa et al. [31] propose a distillation approach in which a few-step stu-

<footnote n="1">¹Without explicit distinction, we use the terms “one-step generator” and “student model” interchangeably when referring to our distilled model.</footnote>
dent is trained using a combination of distillation loss and consistency loss to improve sampling efficiency. However, their method relies on either simulating the teacher's trajectory or utilizing real data to obtain intermediate states for training the student, leading to a mismatch between training and inference. Similar to Progressive Distillation [87], Deschenaux et al. [20] distill a student model to approximate the teacher's multi-step output distribution via divergence minimization. However, this approach requires multiple inferences of the teacher model during student training. Additionally, since the teacher's output tokens are independent, the distillation step size is constrained, necessitating a multi-round distillation process similar to [87]. Furthermore, all these approaches still require multiple sampling steps and suffer from significant performance degradation when the number of steps is further reduced. This limitation highlights the need for further research into efficient one-step generation strategies, which is the task we propose and tackle with our proposed Di[M]O.

**Continuous Diffusion Distillation.** The goal of diffusion distillation is to reduce sampling steps, and can be generally divided into through two primary approaches [127]: regression-based and distribution-based methods [121]. Regression-based techniques [29, 54, 59, 64, 87, 102, 117, 128] train student models via regression objectives derived from the PF-ODE of the teacher model. It is challenging to apply these techniques for MDMs due to their missing of explicit PF-ODE as sampling trajectory. Distribution-based methods [18, 46, 60, 61, 66, 115, 118, 119, 123-125] instead directly match the student's output distribution to the teacher's multi-step sampling distribution, often leveraging adversarial or divergence-minimization strategies. Similar to prior Variational Score Distillation (VSD) approaches [18, 60, 66, 89, 110, 114, 118, 119], our Di[M]O approach employs an auxiliary model $\psi$ to approximate the intermediate distributions of an one-step generator. However, our method differs in three key aspects. First, it is specifically designed for masked diffusion and does not rely on the score function used in continuous diffusion models. Second, while VSD-based methods minimize the divergence of denoising distributions [32, 120] over the full space of latent images, we instead optimize the divergence of token-level conditional distributions. Third, our framework generalizes beyond the reverse Kullback-Leibler (KL) divergence employed in these prior works, supporting arbitrary f-divergences [17] for distribution matching, effectively mitigating the mode-seeking bias.

## 3. Preliminary: Masked Diffusion Models

Masked Diffusion Models (MDMs) [11, 40, 96] rely on a pre-trained discrete image tokenizer, such as VQ-VAE [22, 106]. Given an image of shape $3 \times H \times W$, the tokenizer

<figure><img src="image_4.png" /><figcaption>Figure 2. Reverse process of MDM. With a masked sequence $x_t$ as input, the MDM independently output logits $z_{\phi}^i$ at each masked position $i$, which are then used to sample the new tokens $x_{\phi}^i$ as the model prediction. In the next state $x_s$, we use $x_{\phi}^i$ to replace each masked token in $x_t$ with probability $(r_t - r_s)/r_t$ [YZ: under random strategy].</figcaption></figure>

encodes it into a sequence $x_0 = \{x_0^i\}_{i=1}^L$ of discrete tokens, where $x_0^i \in [V]$ is the token at position $i$. Here, $[V] = \{0, 1, \dots, V-1\}$ represents the set of token indices, with $V$ the vocabulary size. The sequence length is given by $L = hw$, where $h$ and $w$ correspond to the height and width of the 2D token grid produced by the VQ-VAE encoder. Similar to continuous diffusion models [38, 98, 101], MDMs generate discrete data by learning to reverse an absorbing-state forward diffusion process [3].

**Forward Process.** The forward process gradually corrupts the original token sequence $x_0$ into a fully masked sequence $x_1$. For any timestep $t \in [0, 1]$, this yields intermediate masked token sequences $x_t$, composed of mixed image and mask tokens at a mask ratio $r_t$ $^2$. The transition from $x_0$ to $x_t$ follows the probability [71, 86, 96]:

$$q_{t|0}(x_t|x_0) = \prod_{i=0}^{L-1} \mathrm{Cat}(x_t^i; (1-r_t)\delta(x_0^i) + r_t\delta([M))), \quad (1)$$

where $i \in \{0, 1, \dots, L-1\}$ denotes the token position, $[M]$ specifies the mask token, $\delta(\cdot)$ is the Dirac function, and $\mathrm{Cat}(x^i; p)$ refers to the categorical distribution over $V+1$ categories in $[V] \cup \{[M]\}$. Essentially, tokens in $x_0$ are independently masked with probability $r_t$ at timestep $t$.

**Parameterized Reverse Process.** The reverse process reconstructs the original image tokens by iteratively replacing masked tokens with predicted ones, starting from a fully masked sequence $x_1$. This process is guided by a model $\phi$, which predicts a categorical distribution over each masked token at position $i$ given $x_t$:

$$p_{\phi}(x_0^i|x_t) := \mathrm{softmax}(z_{\phi}^i(x_t)/\tau), \quad (2)$$

<footnote>$^2$In masked diffusion, each token in a sequence can only be either $[M]$ token or image token, and different timesteps influence only the proportion of these two token types in the sequence (following the mask ratio $r_t$), e.g. in half timesteps ($r_t = 0.5$), half the tokens will be masked.</footnote>
where $z_{\phi}^i \in \mathbb{R}^V$ are the independently predicted logits, and $\tau$ controls sampling diversity. As shown in Fig. 2, each predicted image token $x_{\phi}^i$ is independently sampled from the token-level predicted distribution $p_{\phi}(x_0^i|x_t)$ and is used to transition from timestep $t$ to $s$ ($0 \le s < t \le 1$). During this transition, unmasked tokens remain unchanged, whereas each masked token $x_t^i$ is updated to $x_{\phi}^i$ with probability $(r_t - r_s)/r_t$ or stay masked with probability $r_s/r_t$. The model $\phi$ operates independently across token positions, enabling parallel token prediction, hence efficiently refining $x_1$ into $x_0$ in a progressive manner.

**MDM Training Objective.** The model $\phi$ is trained to reconstruct the original token sequence $x_0$ from their masked counterparts $x_t$ by minimizing the negative log-likelihood across all mask ratios $r_t$:

$$ \mathcal{L}_{\text{MDM}} = \mathbb{E}_{x_0,t} \left[ \left( \mathbb{E}_{q_{t|0}} \left[ -\log p_{0|t}(x_0|x_t, \phi) \right] \right) \right], \quad (3) $$

with $x_0 \sim p_0(x_0)$ and $t \sim \mathcal{U}[0, 1]$. This loss is computed as cross-entropy per masked token and is the same objective used in Masked Image Modeling (MIM) approaches [6, 40] or masked generative models [25, 42], such as MaskGit [11] and Meissonic [4], where the models also take condition $c$ (class label or text prompt) as input.

## 4. Method

We introduce Distilling [M]DM into One-step generator (Di[M]O), a novel method for distilling a multi-step MDM teacher model $\phi$ into a one-step generator $\theta$. An overview of Di[M]O is illustrated in Fig. 3. The first step is to define the initial distillation objective, which aims to align the generation distribution of the one-step generator and the multi-step teacher in Sec. 4.1. In Sec. 4.2, we introduce an approximation strategy that formulates an effective gradient to update the generator. In Sec. 4.3, we propose initialization strategies to address the low-entropy issue of the initial masked state in MDMs. By combining these components, our method enables efficient and high-fidelity discrete data generation, bridging the gap between multi-step MDMs and one-step inference models.

### 4.1. One-step On-policy Distillation

Our goal is to train the student model to transform an initial random input distribution, $p_{\text{init}}$, into outputs that closely resemble the teacher's multi-step generation distribution, $p_{\phi}$ (note that we discuss the specific design of $p_{\text{init}}$ in Section Sec. 4.3.). Taking inspiration from recent works on on-policy distillation [1, 2, 5, 35, 37, 84, 93], we propose to first sample intermediate state $\tilde{x}_t$ derived from the student's own predictions and then match the conditional predicted distributions of teacher $p_{\phi}(x_0|\tilde{x}_t)$ and student $p_{\theta}(x_0|\tilde{x}_t)$. We denote this procedure as $D(p_{\phi}||p_{\theta})(\tilde{x}_t) := D(p_{\phi}(x_0|\tilde{x}_t)||p_{\theta}(x_0|\tilde{x}_t))$, where $D$ is the divergence. By

enforcing this match for all possible intermediate states $\tilde{x}_t$, we effectively ensure that the overall generation distribution of the student model aligns with that of the teacher. The overall distillation objective can be formulated as follows:

$$ \mathcal{L}_{\text{Di[M]O}}(\theta) := \mathbb{E}_{x_{\text{init}},t} [w(t) (\mathbb{E}_{q_{t|0}} [D(p_{\phi}||p_{\theta})(\tilde{x}_t)])], \quad (4) $$

where $t \sim \mathcal{U}[0, 1]$ and $w(t)$ is a weighting function. Since the generator $\theta$ can now only take $x_{\text{init}}$ as input and directly produces $x_0$, we apply the forward diffusion process to get $\tilde{x}_t \sim q_{t|0}(\tilde{x}_t|x_{\theta}(x_{\text{init}}))$ as the pseudo-intermediate state from its output $x_{\theta}(x_{\text{init}})$. This divergence is calculated across all possible $x_{\text{init}}$ and $t$ to cover every possible intermediate state.

**Token-level Distribution Matching.** The conditional divergence in Eq. (4) can be further decomposed and formulated as the averaged divergence of each masked tokens given $\tilde{x}_t$, defined as:

$$ D(p_{\phi}||p_{\theta})(\tilde{x}_t) := \frac{1}{L_M} \sum_{\substack{i=1 \\ \tilde{x}_t^i = [M]}}^{L} D(p_{\phi}(x_0^i|\tilde{x}_t)||p_{\theta}(x_0^i|\tilde{x}_t)), \quad (5) $$

where $L_M$ is the number of the mask tokens in $\tilde{x}_t$. Unlike in continuous diffusion model distillation [60, 66, 119], the divergence is computed over the fully generated latents of the models, our proposed Di[M]O instead calculates divergence at the token level with the decomposition in Eq. (5).

### 4.2. Approximation of Loss Gradient

Directly optimizing Eq. (4) is not feasible because the exact form of $p_{\theta}(x_0^i|\tilde{x}_t)$ is unknown. Similar to prior work [41, 77, 125], we seek to approximate the following gradient of training objective (see Appendix B.1):

$$ \nabla_{\theta} \mathcal{L}_{\text{Di[M]O}} = \mathbb{E}_{x_{\text{init}},t} \left[ w(t) \left( \mathbb{E}_{q_{t|0}} \left[ \nabla_{z_{\theta}} D(p_{\phi}||p_{\theta})(\tilde{x}_t) \frac{dz_{\theta}(\tilde{x}_t)}{d\theta} \right] \right) \right], \quad (6) $$

where the logits $z_{\theta}$ is the direct output of student model. In this way, we can view the loss gradient as the product of two terms: the gradient of the divergence with respect to model logits (in orange), and the Jacobian of the model output with respect to student parameters $\theta$ (in blue). By approximating these two terms separately, we can address the intractability of directly optimizing the original objective:

1. **Approximation of $p_{\theta}(x_0^i|\tilde{x}_t)$:** Given that the highlighted divergence gradient term in Eq. (6) can be viewed as a functional of the teacher output $p_{\phi}(x_0^i|\tilde{x}_t)$ and the unknown student output $p_{\theta}(x_0^i|\tilde{x}_t)$, we adopt the strategy from [60, 110, 119] and introduce an auxiliary model to approximate the unknown $p_{\theta}(x_0^i|\tilde{x}_t)$. The auxiliary model $\psi$ is trained as a $x_0$-predictor using data generated by the one-step generator and is optimized via the MDM training loss in Eq. (3) to have $p_{\psi}(x_0^i|\tilde{x}_t) \approx p_{\theta}(x_0^i|\tilde{x}_t)$.
<figure><img src="image_5.png" /><figcaption>Figure 3. Di[M]O Pipeline. Our method distills a costly multi-step MDM teacher into a one-step generator. Given $x_{init}$ sampled using our proposed token initialization strategy, the one-step generator (student $\theta$) produces logits $z_{\theta}$, from which image token sequence $x_{\theta} = \{x_{\theta}^i\}_{i=1}^L$ are sampled. These tokens are then processed to obtain an intermediate state $\tilde{x}_t$ through a forward mask diffusion process. For each intermediate state $\tilde{x}_t$, we update the one-step generator $\theta$ and auxiliary model $\psi$ alternately: the one-step generator is optimized by minimizing the conditional divergence $D(p_{\phi}||p_{\psi})(\tilde{x}_t)$ at token-level, while the auxiliary model is trained using a cross-entropy loss to model the distribution of generated tokens $x_{\theta}$ and to form the gradient to update $\theta$. The teacher $\phi$ is frozen during training.</figcaption></figure>

2. **Approximation of $z_{\theta}(\tilde{x}_t)$:** The approximation of the model's Jacobian term must satisfy a critical requirement: it needs to provide gradient information to update the model parameters $\theta$. Given this constraint, we propose approximating the intractable implicit term $z_{\theta}(\tilde{x}_t)$ by substituting it with the one-step output $z_{\theta}(x_{init})$, under the assumption of consistency property [102] (see Fig. 15 for illustration).

These approximations lead to the following gradient of loss:

$$ \nabla_{\theta} \mathcal{L}_{\text{Di[M]O}} \approx \mathbb{E}_{x_{\text{init}}, t} \left[ w(t) \left( \mathbb{E}_{q_{t|0}} \left[ \nabla_{z_{\psi}} D(p_{\phi}||p_{\psi})(\tilde{x}_t) \frac{dz_{\theta}(x_{\text{init}})}{d\theta} \right] \right) \right] \quad (7) $$

To effectively transfer prior knowledge from teacher model, we initialize both the generator $\theta$ and auxiliary model $\psi$ with the pre-trained weights of the teacher model $\phi$.

**Generalized Jeffrey Divergence.** Unlike similar works on continuous diffusion distillation [60, 66, 119] that rely on the Reverse KL (RKL) divergence, the proposed Di[M]O is not limited to this choice. Instead, it can leverage alternative token-level divergence measures, as we do not impose restrictions on the form of divergence in Eq. (7). This is an especially interesting feature as the RKL is known to exhibit unwanted mode-seeking behaviors [18].

In particular, we propose to explore the use of Generalized Jeffrey Divergence [44, 95], which is defined as a linear combination of the Forward KL (FKL) and RKL divergences and investigate its effect beyond mode-seeking (see Appendix G):

$$ D_{\text{jeffrey}}^{\beta} = (1 - \beta) D_{FKL} + \beta D_{RKL}, \quad (8) $$

with $\beta$ the hyperparameter. The explicit gradient of these divergences is provided in Appendix B.2.

## 4.3. Token Initialization Strategy

Unlike continuous diffusion models, where the initial distribution is typically a standard Gaussian, the initial state $x_1$ of MDM consists of a sequence of deterministic identical [M] tokens [3, 31]. This fixed initialization leads to mode collapse for one-step generation, as the generator can only output

fixed logits $\tilde{z}_{\theta}$ with limited sample diversity. We therefore explore three possible strategies for the token sequence initialization, including the fixed one, as follows (see left panel of Fig. 3):

1. **All masked tokens:** following the standard MDM initialization [86, 96], we initialize the sequence with only [M] tokens.

2. **All random tokens:** aiming for maximizing entropy, we choose image tokens with random values to fill the initial token sequence.

3. **Hybrid strategy:** combining the two previous strategies by fixing an initial mask ratio $r_{init}$ and replacing the remaining $1-r_{init}$ of the [M] tokens with random image tokens.

In our work, we adopt the **hybrid strategy**, guided by two key hypotheses and empirically supported by our ablation analysis in Fig. 5: (1) To prevent mode collapse, the initialization of one-step generator should include a certain level of randomness; (2) as the one-step generator is inherited from the teacher, whose objective is to predict all tokens from partially masked inputs, the initialization must preserve a similar pattern, i.e., include some mask tokens to avoid input distribution mismatch.

Moreover, inspired by prior auto-regressive distillation work [53], which proposed to replace all initial token embeddings with random Gaussian noise, we therefore add Gaussian perturbation upon our current scheme to further increase the randomness from a discrete space to a real one. Specifically, after randomly selecting $r_{init}$ [M] tokens and 1-$r_{init}$ random image tokens, we perturb all token embeddings e with Gaussian noise $\epsilon$ at a fixed noise level $\sigma_{init}$ using a variance-preserving scheme [38, 101] $\hat{e} = \sqrt{1 - \sigma_{init}^2}e + \sigma_{init}\epsilon$ as illustrated in Figure 3.

**Overview of Algorithm.** We now summarize our method in Algorithm 1 where the generator and auxiliary model are updated iteratively. In each iteration, the generator produces tokens $x_{\theta}$ from initial state $x_{init}$ in one step and is optimized with the loss gradient in Eq. (7), while the auxiliary model is trained on these generated tokens as targets.
**Algorithm 1 Di[M]O Distillation**

**Require:** Pre-trained teacher model $\phi$, condition dataset $\mathcal{D}$

1: $\theta \leftarrow \text{copyWeights}(\phi)$, $\psi \leftarrow \text{copyWeights}(\phi)$ // intialize

2: **repeat**

3: // Generate tokens $x_0$

4: Sample $x_{\text{init}} \sim p_{\text{init}}$, $c \sim \mathcal{D}$ // with strategy in Sec. 4.3

5: Get generator logits $z_{\theta}(x_{\text{init}}, c) \in \mathbb{R}^{B \times h \times w \times V}$

6: $x_{\theta} \in \mathbb{R}^{B \times h \times w} \xleftarrow{\text{sample}} p_{\theta}(x_0|x_{\text{init}}) = \text{softmax}(z_{\theta}(x_{\text{init}}, c))$

7: // Update generator $\theta$

8: Sample $t \sim \mathcal{U}[0, 1]$, $\tilde{x}_t \sim q_{t|0}(\tilde{x}_t|x_{\theta}(x_{\text{init}}, c))$ // Forward

9: Calculate $p_{\phi}(x_0|\tilde{x}_t, c)$ and $p_{\psi}(x_0|\tilde{x}_t, c)$

10: Update $\theta$ with the loss gradient $\nabla_{\theta}\mathcal{L}_{\text{Di[M]O}}$ (Eq. (7))

11: // Update auxiliary model $\psi$

12: Sample $t' \sim \mathcal{U}[0, 1]$, $\tilde{x}_{t'} \sim q_{t|0}(\tilde{x}_{t'}|x_{\theta}(x_{\text{init}}, c))$

13: Update $\psi$ with cross entropy loss (Eq. (3))

14: **until** convergence

15: **Return** one-step generator $\theta$

**5. Experiments**

**Teacher Models.** We conduct extensive distillation experiments on both class-conditional image generation and text-to-image generation. For class-conditional generation, we adopt the best-performing and publicly available class-conditional MDM, MaskGit [7], as the teacher model. For text-to-image generation, we employ the recent Meissonic [4] as the teacher and use the LAION-Aesthetics-6+ prompt dataset [14] for prompting during the distillation.

**Evaluation Metrics.** The metrics we use for comparing ImageNet results are Fréchet Inception Distance (FID) [34] and Inception Score (IS) [88]. We also use precision (Prec.) and recall (Rec.) [48], density (Den.) and coverage (Cov.) [65] to further evaluate the fidelity and diversity of generated images. For text-to-image generation, we follow [4] and measure the HPSv2 [112] and Geneval [26] score. See Appendix E for more metrics and results of both tasks. In our experiments on ImageNet 256×256, we calculate the FID using 5k generated images for ablations and 50k generated for benchmarking (50 images per class), comparing them against 50k images from the ImageNet validation set with Clean-FID [73].

**Experiment Setup.** Both teacher models adopt Classifier-Free Guidance (CFG) [36]. For the MaskGit teacher, we maintain a CFG scale of 2 during distillation, while for the Meissonic teacher, we use a CFG scale of 4. Our generator is always trained with conditioning, eliminating the need for CFG during inference, further improving the sampling efficiency. All of the ablation experiments on MaskGit are trained with a batch size of 64 and 30000 iterations. We evaluate the final checkpoints with different temperature to get the best metric results. The learning rates for MaskGit experiments and Meissonic experiments are $1 \times 10^{-5}$ and $1 \times 10^{-6}$, respectively. See Appendix D for more details.

<table><caption>Table 1. Quantitative results on class-conditional ImageNet-256. * denotes numbers estimated from the original plot.</caption><thead><tr><th>ImageNet</th><th>Method</th><th>Step (↓)</th><th>FID (↓)</th><th>IS (↑)</th><th>Prec. (↑)</th><th>Rec. (↑)</th><th>Den. (↑)</th><th>Cov. (↑)</th></tr></thead><tbody><tr><td rowspan="4">Teacher</td><td>MaskGit [7]</td><td>16</td><td>6.60</td><td>224.07</td><td>0.831</td><td>0.402</td><td>1.246</td><td>0.977</td></tr><tr><td>MaskGit [7]</td><td>8</td><td>6.66</td><td>221.57</td><td>0.827</td><td>0.397</td><td>1.233</td><td>0.974</td></tr><tr><td>MaskGit [7]</td><td>4</td><td>10.73</td><td>192.29</td><td>0.748</td><td>0.313</td><td>1.011</td><td>0.920</td></tr><tr><td>MaskGit [7]</td><td>2</td><td>91.35</td><td>13.37</td><td>0.178</td><td>0.164</td><td>0.091</td><td>0.122</td></tr><tr><td rowspan="2">Sampler</td><td>θ-trapezoidal [81]*</td><td>64</td><td>6.7</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>θ-trapezoidal [81]*</td><td>32</td><td>7.1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td rowspan="2">Distillation</td><td>di4c [31]</td><td>4</td><td>6.79</td><td>209.2</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>di4c-d [31]</td><td>4</td><td>6.57</td><td>213.6</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td></td><td>Di[M]O</td><td>1</td><td>6.91</td><td>214.0</td><td>0.828</td><td>0.377</td><td>1.255</td><td>0.967</td></tr></tbody></table>

**5.1. Class-conditional Image Generation**

In Tab. 1, we present a quantitative performance comparison of our Di[M]O with various acceleration methods, including the high-order sampler $\theta$-trapezoidal [81], other distillation techniques, and the teacher model. Compared to the high-order sampler, which requires 32 steps to achieve an FID of 7.1, our method significantly reduces the number of steps to just 1 while maintaining competitive performance, with an FID of 6.91 and an IS of 214.0. When comparing to distillation methods, our approach shows comparable results to di4c [31], which uses 4 steps and achieves an FID of 6.79 and an IS of 209.2. Our method yields an FID of 6.91 with a higher IS of 214.0 in one step. Additionally, when compared to the teacher model (MaskGit [7]) with varying steps, our method performs very closely to the teacher's performance of 6.60 FID at 16 steps, while achieving this with just a single step. These comparisons highlights the effectiveness of our method in matching the performance of multi-step teacher models while dramatically reducing the computational cost. For further details, refer to Appendix D.

**5.2. Ablations**

In this section, we present ablation studies on key hyperparameters of our method to validate our design choices: (1) the initial mask ratio $r_{\text{init}}$; (2) the Jeffrey Coefficient $\beta$; and (3) the Gaussian perturbation Strength $\sigma_{\text{init}}$:

**Initial Mask Ratio $r_{\text{init}}$.** We begin our ablation study by analyzing the impact of the initial mask ratio $r_{\text{init}}$, which is a crucial factor in our algorithm. As shown in Fig. 5a, our results confirm the hypothesis from Sec. 4.3 that both extreme values, $r_{\text{init}} = 0$ and $r_{\text{init}} = 1$ fail to work. Specifically, setting $r_{\text{init}} = 1$ results in mode collapse, while very low values ($r_{\text{init}} \approx 0$) cause unstable training, as evidenced by the broken curves in the subfigures of Fig. 5a. The optimal choice appears to be $r_{\text{init}} = 0.6$, which achieves the lowest FID. See Fig. 4 for visual demonstrations.

**Jeffrey Coefficient $\beta$.** Building on this, we conduct an ablation on the generalized Jeffrey divergence coefficient $\beta$ (Fig. 5b). Our experiments indicate that reducing $\beta$ generally improves FID, aligning with observations from prior work [116]. Surprisingly, our method remains effective
<figure><img src="image_6.png" /><figcaption>Figure 4. Visual results of ImageNet. One-step generated images from the generator trained with different $r_{init}$ in comparison with teacher generation with 16 sampling steps. The class labels of the samples from top to bottom are 388, 979 and 207 respectively.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 5. Ablation studies on ImageNet using FID as the evaluation metric. * means the training is collapsed and falls outside the comparable range with other results, we show these in the sub-figures at the right upper corner with the same x-axis range.</figcaption></figure>

even for negative values of $\beta$, with the best performance observed at $\beta = -0.2$.

**Gaussian Perturbation $\sigma_{init}$**. Finally, we examine the effect of Gaussian perturbation on token embeddings (Fig. 5c). Our results demonstrate that introducing perturbations provides additional improvements in FID, further refining the quality of the generated samples. For completeness, the corresponding results with the IS metric are provided in Fig. 8 in the Appendix.

## 5.3. Text-to-image Generation

We validate Di[M]O for text-to-image generation by evaluating our distilled one-step generator on the HPSv2 [112] and GenEval [26] benchmarks.

In Tab. 2, we show the HPSv2 [112] benchmark evaluating human preference for T2I models. Distilled from Meissonic [4], the only open-sourced text-to-image MDM, we

compare our one-step generator with not only the teacher model, but also recent diffusion models [19, 21, 76, 83] and their distilled one-step generators [18, 55, 90]. The results clearly show that our one-step generator achieves a competitive performance with teacher models generated using 16 to 32 steps, whereas teacher's performance deteriorates rapidly when fewer generation steps: a 48-step generation yields 28.83, whereas the 4 steps' drop to 24.66). Our one-step generator also outperforms other one-step continuous diffusion-based ones; however, direct comparisons are limited since they are distilled from different teachers.

To better evaluate the semantic expressiveness of our one-step generator, Tab. 3 presents a comparison with both diffusion models [21, 76, 79, 83] and our teacher, Meissonic [4]. The experimental results corroborate our previous observations: our one-step generator competes with teacher models' performance using 16 to 32 steps, while the teacher's performance degrades rapidly with reduced steps.
<figure><img src="image_8.png" /><figcaption>Figure 6. Qualitative our method distilled from teacher Meissonic. Corresponding prompts can be found in Appendix I</figcaption></figure>

<table><caption>Table 2. HPS v2.0 benchmark. Scores are collected from https://github.com/tgxs002/HPSv2. We highlight the best.</caption><thead><tr><th colspan="10">HPS v2.0</th></tr><tr><th>Model</th><th>Step</th><th>Anim.</th><th>Concept-art</th><th>Painting</th><th>Photo</th><th>Averaged</th><th>Step</th><th>Overall</th><th>Objects</th><th>Counting</th><th>Colors</th><th>Position</th><th>Color Attribution</th></tr></thead><tbody><tr><td>Latent Diffusion [83]</td><td>25</td><td>25.73</td><td>25.15</td><td>25.25</td><td>26.97</td><td>25.78</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>DALL-E 2 [79]</td><td></td><td>27.34</td><td>26.54</td><td>26.68</td><td>27.24</td><td>26.95</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Stable Diffusion v1.4 [83]</td><td>50</td><td>27.26</td><td>26.61</td><td>26.66</td><td>27.27</td><td>26.95</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Stable Diffusion v2.0 [83]</td><td>50</td><td>27.48</td><td>26.89</td><td>26.86</td><td>27.46</td><td>27.17</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>DeepFloyd-XL [19]</td><td>25</td><td>27.64</td><td>26.83</td><td>26.86</td><td>27.75</td><td>27.27</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>SDXL Base 1.0 [76]</td><td>50</td><td>28.88</td><td>27.88</td><td>27.92</td><td>28.31</td><td>28.25</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>SDXL Refiner 1.0 [76]</td><td>50</td><td>28.93</td><td>27.89</td><td>27.90</td><td>28.38</td><td>28.27</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>InstaFlow [55]</td><td>1</td><td>25.98</td><td>25.79</td><td>25.93</td><td>26.32</td><td>26.01</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>SD Turbo [90]</td><td>1</td><td>27.98</td><td>27.59</td><td>27.16</td><td>27.19</td><td>27.48</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>SwiftBrush v2 [18]</td><td>1</td><td>27.25</td><td>27.62</td><td>26.86</td><td>26.77</td><td>27.15</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td>48</td><td><strong>29.57</strong></td><td><strong>28.58</strong></td><td><strong>28.72</strong></td><td><strong>28.45</strong></td><td><strong>28.83</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td>32</td><td>29.18</td><td>28.32</td><td>28.28</td><td>27.96</td><td>28.44</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td>16</td><td>28.61</td><td>27.82</td><td>27.84</td><td>27.32</td><td>27.90</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Meissonic [4]</td><td>8</td><td>25.62</td><td>26.49</td><td>26.67</td><td>27.07</td><td>26.46</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td>4</td><td>25.01</td><td>24.95</td><td>24.87</td><td>23.80</td><td>24.66</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td>2</td><td>23.06</td><td>23.28</td><td>23.22</td><td>22.38</td><td>22.98</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Di[M]O</td><td>1</td><td>28.64</td><td>27.91</td><td>27.99</td><td>27.92</td><td>28.11</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

<table><caption>Table 3. GenEval benchmark. We highlight the best result.</caption><thead><tr><th colspan="10">GenEval</th></tr><tr><th>Model</th><th>Step</th><th>Overall</th><th>Objects<br>Single</th><th>Two</th><th>Counting</th><th>Colors</th><th>Position</th><th>Color<br>Attribution</th></tr></thead><tbody><tr><td>SD v1.5 [83]</td><td>50</td><td>0.43</td><td>0.97</td><td>0.38</td><td>0.35</td><td>0.76</td><td>0.04</td><td>0.06</td></tr><tr><td>SD v2.1 [83]</td><td>50</td><td>0.50</td><td>0.98</td><td>0.51</td><td>0.44</td><td>0.85</td><td>0.07</td><td>0.17</td></tr><tr><td>DALL-E2 [79]</td><td></td><td>0.52</td><td>0.94</td><td>0.66</td><td><strong>0.49</strong></td><td>0.77</td><td>0.10</td><td>0.19</td></tr><tr><td>SDXL [76]</td><td>50</td><td><strong>0.55</strong></td><td>0.98</td><td><strong>0.74</strong></td><td>0.39</td><td>0.85</td><td><strong>0.15</strong></td><td><strong>0.23</strong></td></tr><tr><td></td><td>48</td><td>0.54</td><td><strong>0.99</strong></td><td>0.66</td><td>0.42</td><td><strong>0.86</strong></td><td>0.10</td><td>0.22</td></tr><tr><td></td><td>32</td><td>0.46</td><td>0.92</td><td>0.53</td><td>0.33</td><td>0.80</td><td>0.08</td><td>0.13</td></tr><tr><td>Meissonic [4]</td><td>16</td><td>0.37</td><td>0.82</td><td>0.39</td><td>0.20</td><td>0.70</td><td>0.05</td><td>0.08</td></tr><tr><td></td><td>8</td><td>0.20</td><td>0.58</td><td>0.12</td><td>0.05</td><td>0.40</td><td>0.02</td><td>0.04</td></tr><tr><td></td><td>4</td><td>0.09</td><td>0.31</td><td>0.02</td><td>0.01</td><td>0.18</td><td>0.01</td><td>0.01</td></tr><tr><td></td><td>2</td><td>0.03</td><td>0.14</td><td>0.01</td><td>0.00</td><td>0.05</td><td>0.00</td><td>0.00</td></tr><tr><td>Di[M]O</td><td>1</td><td>0.43</td><td>0.91</td><td>0.53</td><td>0.22</td><td>0.75</td><td>0.07</td><td>0.11</td></tr></tbody></table>

**Limitations and Future Works.** While our data-free distillation achieves teacher-level performances, its application has so far been limited to our current model scope. In the future, we aim to extend our approach to stronger MDM teachers, particularly for image and text generation. While our method is data-free, it would be beneficial to also introduce real data to boost the one-step student to outperform the teacher. Lastly, while we leverage generalized Jeffreys divergence to avoid mode seeking behaviors, we intend to explore more general f-divergences to increase our method's flexibility and effectiveness.

## 6. Conclusion

In this work, we proposed Di[M]O, a novel method that leverages token-level distribution matching to distill the inference process of MDMs into one-step. Specifically, inspired by the concept of on-policy distillation, we match the token-level distribution conditioned on a pseudo intermediate state obtained from the student's one-step generation. We also conducted extensive experiments on the choice of initialization strategy and distillation objective, to increase the robustness of Di[M]O. Our experimental results demonstrate that our distilled model generates images with comparable quality to the teacher models, while requiring only 1 sampling step during inference. This work demonstrates the power of distribution matching methods on distillation for MDM, contributes to the growing community of research exploring efficient generation of discrete data.
## Acknowledgements

This work was supported by ANR-22-CE23-0007, ANR-22-CE39-0016, Hi!Paris grant and fellowship, DATAIA Convergence Institute as part of the “Programme d’Investissement d’Avenir” (ANR-17-CONV-0003) operated by Ecole Polytechnique, IP Paris, and was granted access to the IDRIS High-Performance Computing (HPC) resources under the allocation 2024-AD011014300R1 and 2025-AD011015894 made by GENCI and mesoGIP of IP Paris. We also sincerely thank Nacereddine Laddaoui for the help with infrastructure, Haoge Deng and Yao Teng for their insightful discussions that contributed to this work. We are also grateful to Nicolas Dufour, Robin Courant, and Lucas Degeorge for their meticulous proofreading.

## References

[1] Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In *The Twelfth International Conference on Learning Representations*, 2024.

[2] Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Chi Kit Cheung. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. *arXiv preprint arXiv:2204.01171*, 2022.

[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. *Advances in Neural Information Processing Systems*, 34:17981-17993, 2021.

[4] Jinbin Bai, Tian Ye, Wei Chow, Enxin Song, Qing-Guo Chen, Xiangtai Li, Zhen Dong, Lei Zhu, and Shuicheng Yan. Meissonic: Revitalizing masked generative transformers for efficient high-resolution text-to-image synthesis. *arXiv preprint arXiv:2410.08261*, 2024.

[5] Ashwin Balakrishna, Brijen Thananjeyan, Jonathan Lee, Felix Li, Arsh Zahed, Joseph E Gonzalez, and Ken Goldberg. On-policy robot imitation learning from a converging supervisor. In *Conference on Robot Learning*, pages 24–41. PMLR, 2020.

[6] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. *arXiv preprint arXiv:2106.08254*, 2021.

[7] Victor Besnier and Mickael Chen. A pytorch reproduction of masked generative image transformer. *arXiv preprint arXiv:2310.14400*, 2023.

[8] Victor Besnier, Mickael Chen, David Hurych, Eduardo Valle, and Matthieu Cord. Halton scheduler for masked generative image transformer. *iclr*, 2025.

[9] Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. *Advances in Neural Information Processing Systems*, 35:28266-28279, 2022.

[10] Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. *arXiv preprint arXiv:2402.04997*, 2024.

[11] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 11315–11325, 2022.

[12] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. *arXiv preprint arXiv:2301.00704*, 2023.

[13] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. *ACM transactions on Graphics (TOG)*, 42(4):1–10, 2023.

[14] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 2818–2829, 2023.

[15] Seunggeun Chi, Hyung-gun Chi, Hengbo Ma, Nakul Agarwal, Faizan Siddiqui, Karthik Ramani, and Kwonjoon Lee. M2d2m: Multi-motion generation from text with discrete diffusion models. In *European Conference on Computer Vision*, pages 18–36. Springer, 2024.

[16] Jacob K Christopher, Brian R Bartoldson, Bhavya Kailkhura, and Ferdinando Fioretto. Speculative diffusion decoding: Accelerating language generation through diffusion. *arXiv preprint arXiv:2408.05636*, 2024.

[17] Imre Csiszár, Paul C Shields, et al. Information theory and statistics: A tutorial. *Foundations and Trends® in Communications and Information Theory*, 1(4):417–528, 2004.

[18] Trung Dao, Thuan Hoang Nguyen, Thanh Le, Duc Vu, Khoi Nguyen, Cuong Pham, and Anh Tran. Swiftbrush v2: Make your one-step diffusion model better than its teacher. In *European Conference on Computer Vision*, pages 176–192. Springer, 2025.

[19] IF DeepFloyd. Deepfloyd if, 2023. https://huggingface.co/DeepFloyd, 2023.

[20] Justin Deschenaux and Caglar Gulcehre. Beyond autoregression: Fast llms via self-distillation through time. *arXiv preprint arXiv:2410.21035*, 2024.

[21] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. *Advances in Neural Information Processing Systems*, 34:8780–8794, 2021.

[22] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 12873–12883, 2021.

[23] Guhao Feng, Yihan Geng, Jian Guan, Wei Wu, Liwei Wang, and Di He. Theoretical benefit and limitation of diffusion language model. *arXiv preprint arXiv:2502.09622*, 2025.
[24] Michael Fuest, Vincent Tao Hu, and Björn Ommer. Mask-flow: Discrete flows for flexible and efficient long video generation. arXiv preprint arXiv:2502.11234, 2025.

[25] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint arXiv:1904.09324, 2019.

[26] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132-52152, 2023.

[27] Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.17891, 2024.

[28] Nate Gruver, Samuel Stanton, Nathan Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance-Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew G Wilson. Protein design with guided discrete diffusion. Advances in neural information processing systems, 36:12489-12517, 2023.

[29] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Joshua M Susskind. Boot: Data-free distillation of denoising diffusion models with bootstrapping. In ICML 2023 Workshop on Structured Probabilistic Inference {\&} Generative Modeling, 2023.

[30] Jiaqi Han, Mingjian Jiang, Yuxuan Song, Jure Leskovec, Stefano Ermon, and Minkai Xu. f-po: Generalizing preference optimization with f-divergence minimization. arXiv preprint arXiv:2410.21662, 2024.

[31] Satoshi Hayakawa, Yuhta Takida, Masaaki Imaizumi, Hiromi Wakaki, and Yuki Mitsufuji. Distillation of discrete diffusion through dimensional correlations. arXiv preprint arXiv:2410.08709, 2024.

[32] Jiajun He, Wenlin Chen, Mingtian Zhang, David Barber, and José Miguel Hernández-Lobato. Training neural samplers with reverse diffusive kl divergence. arXiv preprint arXiv:2410.12456, 2024.

[33] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.

[34] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.

[35] Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.

[36] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.

[37] Jonathan Ho, Jayesh Gupta, and Stefano Ermon. Model-free imitation learning with policy optimization. In International conference on machine learning, pages 2760-2769. PMLR, 2016.

[38] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.

[39] Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang, Zuopeng Yang, Dacheng Tao, and Ponnuthurai N Sughanthan. Unified discrete diffusion for simultaneous vision-language generation. arXiv preprint arXiv:2211.14842, 2022.

[40] Vincent Tao Hu and Björn Ommer. [mask] is all you need. arXiv preprint arXiv:2412.06787, 2024.

[41] Zemin Huang, Zhengyang Geng, Weijian Luo, and Guojun Qi. Flow generator matching. arXiv preprint arXiv:2410.19310, 2024.

[42] Jiwan Hur, DongJae Lee, Gyojin Han, Jaehyun Choi, Yunho Jeon, and Junmo Kim. Unlocking the capabilities of masked generative models for image synthesis via self-guidance. Advances in Neural Information Processing Systems, 37:130977-130999, 2024.

[43] Thibaut Issenhuth, Ugo Tanielian, Jérémie Mary, and David Picard. Edibert, a generative model for image editing. arXiv preprint arXiv:2111.15264, 2021.

[44] Harold Jeffreys. An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 186(1007):453-461, 1946.

[45] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10124-10134, 2023.

[46] Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.

[47] Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint arXiv:2502.06768, 2025.

[48] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.

[49] Xiner Li, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokcen Eraslan, Surag Nair, Tommaso Biancalani, Shuiwang Ji, Aviv Regev, Sergey Levine, et al. Derivative-free guidance in continuous and discrete diffusion models with soft value-based decoding. arXiv preprint arXiv:2408.08252, 2024.

[50] Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang. Dual diffusion for unified image generation and understanding. arXiv preprint arXiv:2501.00289, 2024.

[51] Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxllightning: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402.13929, 2024.

[52] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014.
2014, proceedings, part v 13, pages 740–755. Springer, 2014.

[53] Enshu Liu, Xuefei Ning, Yu Wang, and Zinan Lin. Distilled decoding 1: One-step sampling of image auto-regressive models with flow matching. arXiv preprint arXiv:2412.17153, 2024.

[54] Qiang Liu. Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint arXiv:2209.14577, 2022.

[55] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. arXiv preprint arXiv:2309.06380, 2023.

[56] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, 2024.

[57] Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint arXiv:2410.11081, 2024.

[58] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: In-painting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022.

[59] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388, 2021.

[60] Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. arXiv preprint arXiv:2305.18455, 2023.

[61] Weijian Luo, Zemin Huang, Zhengyang Geng, J Zico Kolter, and Guo-jun Qi. One-step diffusion distillation through score implicit matching. Advances in Neural Information Processing Systems, 37:115377–115408, 2025.

[62] Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501.09732, 2025.

[63] Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. Advances in Neural Information Processing Systems, 35:34532–34545, 2022.

[64] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297–14306, 2023.

[65] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International conference on machine learning, pages 7176–7185. PMLR, 2020.

[66] Thuan Hoang Nguyen and Anh Tran. Swiftbrush: One-step text-to-image diffusion model with variational score distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7807–7816, 2024.

[67] Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514, 2024.

[68] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025.

[69] Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arXiv:2406.01572, 2024.

[70] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.

[71] Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.03736, 2024.

[72] Yong-Hyun Park, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, and Yuki Mitsufuji. : Optimizing sampling schedule of discrete diffusion models. CoRR, 2024.

[73] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11410–11420, 2022.

[74] Suraj Patil, William Berman, Robin Rombach, and Patrick von Platen. amused: An open muse reproduction. arXiv preprint arXiv:2401.01808, 2024.

[75] Ekkasit Pinyoanuntapong, Pu Wang, Minwoo Lee, and Chen Chen. Mmm: Generative masked motion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1546–1555, 2024.

[76] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations (ICLR), 2024.

[77] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022.

[78] Zipeng Qi, Lichen Bai, Haoyi Xiong, and Zeke Xie. Not all noises are created equally: Diffusion noise selection and optimization. arXiv preprint arXiv:2407.14041, 2024.

[79] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.

[80] Jarrid Rector-Brooks, Mohsin Hasan, Zhangzhi Peng, Zachary Quinn, Chenghao Liu,arthak Mittal, Nouha Dziri, Michael Bronstein, Yoshua Bengio, Pranam Chatterjee, et al. Steering masked discrete diffusion models
via discrete denoising posterior prediction. arXiv preprint arXiv:2410.08134, 2024.

[81] Yinuo Ren, Haoxuan Chen, Yuchen Zhu, Wei Guo, Yongxin Chen, Grant M Rotskoff, Molei Tao, and Lexing Ying. Fast solvers for discrete diffusion models: Theory and applications of high-order algorithms. arXiv preprint arXiv:2502.00234, 2025.

[82] Alfréd Rényi. On measures of entropy and information. In *Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics*, pages 547–562. University of California Press, 1961.

[83] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 10684–10695, 2022.

[84] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In *Proceedings of the fourteenth international conference on artificial intelligence and statistics*, pages 627–635. JMLR Workshop and Conference Proceedings, 2011.

[85] Samir Sadok, Simon Leglaive, Laurent Girin, Gaël Richard, and Xavier Alameda-Pineda. Ancogen: Analysis, control and generation of speech with a masked autoencoder. In *IEEE International Conference on Acoustics, Speech, and Signal Processing*, 2025.

[86] Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.07524, 2024.

[87] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022.

[88] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. *Advances in neural information processing systems*, 29, 2016.

[89] Tim Salimans, Thomas Mensink, Jonathan Heek, and Emiel Hoogeboom. Multistep distillation of diffusion models via moment matching. *Advances in Neural Information Processing Systems*, 37:36046–36070, 2025.

[90] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In *European Conference on Computer Vision*, pages 87–103. Springer, 2024.

[91] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In *European Conference on Computer Vision*, pages 87–103. Springer, 2025.

[92] Yair Schiff, Subham Sekhar Sahoo, Hao Phung, Guanghan Wang, Sam Boshar, Hugo Dalla-torre, Bernardo P de Almeida, Alexander Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. arXiv preprint arXiv:2412.10193, 2024.

[93] Florian Schmidt. Generalization in generation: A closer look at exposure bias. arXiv preprint arXiv:1910.00292, 2019.

[94] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

[95] Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning llms with best-of-n distillation. arXiv preprint arXiv:2407.14622, 2024.

[96] Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. *Advances in Neural Information Processing Systems*, 37:103131–103167, 2025.

[97] Raghav Singhal, Zachary Horvitz, Ryan Teehan, Mengye Ren, Zhou Yu, Kathleen McKeown, and Rajesh Ranganath. A general framework for inference-time scaling and steering of diffusion models. arXiv preprint arXiv:2501.06848, 2025.

[98] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In *International Conference on Machine Learning*, pages 2256–2265. PMLR, 2015.

[99] Jiaming Song and Linqi Zhou. Ideas in inference-time scaling can benefit generative pre-training algorithms. arXiv preprint arXiv:2503.07154, 2025.

[100] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310.14189, 2023.

[101] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.

[102] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023.

[103] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024.

[104] Onkar Susladkar, Jishu Sen Gupta, Chirag Sehgal, Sparsh Mittal, and Rekha Singhal. Motionaura: Generating high-quality and motion consistent videos using discrete diffusion. arXiv preprint arXiv:2410.07659, 2024.

[105] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. *Advances in neural information processing systems*, 37:84839–84865, 2024.

[106] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. *Advances in neural information processing systems*, 30, 2017.

[107] Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXiv:2503.00307, 2025.
[108] Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Dplm-2: A multi-modal diffusion protein language model. arXiv preprint arXiv:2410.13782, 2024.

[109] Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gan: Training gans with diffusion. arXiv preprint arXiv:2206.02262, 2022.

[110] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.

[111] Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao, and Ngai Wong. Rethinking kullback-leibler divergence in knowledge distillation for large language models. arXiv preprint arXiv:2404.02657, 2024.

[112] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023.

[113] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024.

[114] Sirui Xie, Zhisheng Xiao, Diederik P Kingma, Tingbo Hou, Ying Nian Wu, Kevin Patrick Murphy, Tim Salimans, Ben Poole, and Ruiqi Gao. Em distillation for one-step diffusion models. arXiv preprint arXiv:2405.16852, 2024.

[115] Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8196-8206, 2024.

[116] Yilun Xu, Weili Nie, and Arash Vahdat. One-step diffusion models with f-divergence distribution matching. arXiv preprint arXiv:2502.15681, 2025.

[117] Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal plug-and-play accelerator. arXiv preprint arXiv:2405.07510, 2024.

[118] Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. arXiv preprint arXiv:2405.14867, 2024.

[119] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6613-6623, 2024.

[120] Mingtian Zhang, Peter Hayes, Thomas Bird, Raza Habib, and David Barber. Spread divergence. In International Conference on Machine Learning, pages 11106-11116. PMLR, 2020.

[121] Mingtian Zhang, Jiajun He, Wenlin Chen, Zijing Ou, José Miguel Hernández-Lobato, Bernhard Schölkopf, and David Barber. Towards training one-step diffusion models without distillation. arXiv preprint arXiv:2502.08005, 2025.

[122] Yixiu Zhao, Jiaxin Shi, Lester Mackey, and Scott Linderman. Informed correctors for discrete diffusion models. arXiv preprint arXiv:2407.21243, 2024.

[123] Mingyuan Zhou, Zhendong Wang, Huangjie Zheng, and Hai Huang. Long and short guidance in score identity distillation for one-step text-to-image generation. arXiv preprint arXiv:2406.01561, 2024.

[124] Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, and Hai Huang. Adversarial score identity distillation: Rapidly surpassing the teacher in one step. arXiv preprint arXiv:2410.14919, 2024.

[125] Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024.

[126] Zikai Zhou, Shitong Shao, Lichen Bai, Zhiqiang Xu, Bo Han, and Zeke Xie. Golden noise for diffusion models: A learning framework. arXiv preprint arXiv:2411.09502, 2024.

[127] Yuanzhi Zhu, Hanshu Yan, Huan Yang, Kai Zhang, and Junnan Li. Accelerating video diffusion models via distribution matching. arXiv preprint arXiv:2412.05899, 2024.

[128] Yuanzhi Zhu, Xingchao Liu, and Qiang Liu. Slimflow: Training smaller one-step diffusion models with rectified flow. In European Conference on Computer Vision, pages 342-359. Springer, 2025.
# Di[M]O: Distilling Masked Diffusion Models into One-step Generator

## Supplementary Material

The supplementary material is organized as follows:

* Appendix A: Broader impacts of this work.

* Appendix B: Relevant derivations of the approximated divergence gradient.

* Appendix C: Discussion of additional related works.

* Appendix D: Detailed experiment setup.

* Appendix E: Additional experiments and corresponding findings.

* Appendix F: Failure cases where the generation quality does not match that of the teacher model.

* Appendix G: Visualization and example of the mode-seeking/covering behaviors of the generalized Jeffrey divergence.

* Appendix H: Additional visual results of one-step generations from our distilled models.

* Appendix I: List of all prompts used in this paper for image generation.

## A. Broader Impacts

Our work focuses on distilling the multi-step generation process of MDMs into one step, significantly reducing inference time and computational costs, therefore lowering the carbon footprint during the inference. This advancement has the potential to make high-quality generative models more accessible, facilitating applications in creative industries, content generation, and real-time systems. However, as with many generative modeling techniques, our method inherits biases from the teacher models. This could potentially lead to ethical concerns, including the generation of misleading or harmful content. Additionally, by enabling faster and more efficient content generation, our approach could lower the barrier to misuse, such as the creation of deepfakes or other deceptive media.

## B. Relevant Derivations to the Token-level Divergence

### B.1. Loss Gradient

Given Eq. (2) and Eq. (4), by assuming $D$ is differentiable with respect to $p_{\theta}(x_{0}^{i}|\tilde{x}_{t})$, we can calculate Eq. (6) using chain rule as:

$$ 
\begin{aligned}
\nabla_{\theta} \mathcal{L}_{\text{Di[M]O}} &= \nabla_{\theta} \mathbb{E}_{x_{\text{init}}, t} [w(t) (\mathbb{E}_{q_{t|0}} [D((p_{\phi}||p_{\theta})(\tilde{x}_{t}))])] \\
&= \mathbb{E}_{x_{\text{init}}, t} [w(t) (\mathbb{E}_{q_{t|0}} [\nabla_{\theta} D((p_{\phi}||p_{\theta})(\tilde{x}_{t}))])] \\
&= \mathbb{E}_{x_{\text{init}}, t} \left[ w(t) \left( \mathbb{E}_{q_{t|0}} \left[ \frac{1}{L_M} \sum_{\substack{i=1 \\ \tilde{x}_t^i = [M]}}^{L} \nabla_{\theta} D(p_{\phi}(x_0^i|\tilde{x}_t) || p_{\theta}(x_0^i|\tilde{x}_t)) \right] \right) \right] \\
&= \mathbb{E}_{x_{\text{init}}, t} \left[ w(t) \left( \mathbb{E}_{q_{t|0}} \left[ \frac{1}{L_M} \sum_{\substack{i=1 \\ \tilde{x}_t^i = [M]}}^{L} \nabla_{p_{\theta}(x_0^i|\tilde{x}_t)} D(p_{\phi}(x_0^i|\tilde{x}_t) || p_{\theta}(x_0^i|\tilde{x}_t)) \frac{dp_{\theta}(x_0^i|\tilde{x}_t)}{dz_{\theta}^i} \frac{dz_{\theta}^i}{d\theta} \right] \right) \right].
\end{aligned}
\quad (9) $$

By defining $\nabla_{z_{\theta}} D((p_{\phi}||p_{\theta})(\tilde{x}_{t}))$ as a vector with the $i$-th element $[\nabla_{z_{\theta}} D((p_{\phi}||p_{\theta})(\tilde{x}_{t}))]_{i} = \frac{1}{L_M} \nabla_{p_{\theta}(x_{0}^{i}|\tilde{x}_{t})} D(p_{\phi}(x_{0}^{i}|\tilde{x}_{t}) || p_{\theta}(x_{0}^{i}|\tilde{x}_{t})) \frac{dp_{\theta}(x_{0}^{i}|\tilde{x}_{t})}{dz_{\theta}^{i}}$, and $\frac{dz_{\theta}}{d\theta}$ as a vector with the $i$-th element $\frac{dz_{\theta}^{i}}{d\theta}$, we have:

$$ \nabla_{\theta} \mathcal{L}_{\text{Di[M]O}} = \mathbb{E}_{x_{\text{init}}, t} \left[ w(t) \left( \mathbb{E}_{q_{t|0}} \left[ \nabla_{z_{\theta}} D(p_{\phi} || p_{\theta})(\tilde{x}_{t}) \frac{dz_{\theta}(\tilde{x}_{t})}{d\theta} \right] \right) \right]. \quad (10) $$

By default, we apply stop-gradient to the $\tilde{x}_{t}$, since the sample operation from $z_{\theta}$ to $x_{\theta}(x_{\text{init}})$ is non-differentiable.
## B.2. Explicit Form of Divergence

We start the derivation with the FKL and RKL in the generalized Jeffrey divergence. The forward and reverse KL between the teacher $\phi$ and the one-step generator $\theta$ at each output location $i$ are:

$$D_{FKL_i}(\tilde{x}_t) = \sum_{k=1}^{V} p_{\phi}(x_0^i = k | \tilde{x}_t) \log \left( \frac{p_{\phi}(x_0^i = k | \tilde{x}_t)}{p_{\theta}(x_0^i = k | \tilde{x}_t)} \right), \quad (11)$$

$$D_{RKL_i}(\tilde{x}_t) = \sum_{k=1}^{V} p_{\theta}(x_0^i = k | \tilde{x}_t) \log \left( \frac{p_{\theta}(x_0^i = k | \tilde{x}_t)}{p_{\phi}(x_0^i = k | \tilde{x}_t)} \right).$$

The derivation of these KLs with respect to the student parameter $\theta$ can be written as:

$$\nabla_{\theta} D_{FKL_i} = \sum_{j=1}^{V} \frac{\partial D_{FKL_i}}{\partial z_{\theta_j^i}} \frac{\partial z_{\theta_j^i}}{\partial \theta}, \quad (12)$$

$$\nabla_{\theta} D_{RKL_i} = \sum_{j=1}^{V} \frac{\partial D_{RKL_i}}{\partial z_{\theta_j^i}} \frac{\partial z_{\theta_j^i}}{\partial \theta}.$$

Given that the probability corresponding to the logits $z$ as $p_{\theta}(x_0^i = k | \tilde{x}_t) = \frac{\exp(z_{\theta_k^i})}{\sum_{n=1}^{V} \exp(z_{\theta_n^i})}$, we can precalculate the following quantity for $j \neq k$:

$$\frac{\partial}{\partial z_{\theta_j^i}} p_{\theta}(x_0^i = j | \tilde{x}_t) = \frac{\exp(z_{\theta_j^i})}{\sum_{n=1}^{V} \exp(z_{\theta_n^i})} - p_{\theta}(x_0^i = j | \tilde{x}_t)^2 = p_{\theta}(x_0^i = j | \tilde{x}_t)(1 - p_{\theta}(x_0^i = j | \tilde{x}_t)), \quad (13)$$

$$\frac{\partial}{\partial z_{\theta_j^i}} p_{\theta}(x_0^i = k | \tilde{x}_t) = - \frac{\exp(z_{\theta_k^i}) \exp(z_{\theta_j^i})}{(\sum_{n=1}^{V} \exp(z_{\theta_n^i}))^2} = -p_{\theta}(x_0^i = k | \tilde{x}_t) p_{\theta}(x_0^i = j | \tilde{x}_t).$$

These are the gradients of the softmax function that we will use below to help the derivation.

### B.2.1. Gradient of FKL

For each possible token $i$, we have:

$$\begin{align}
\frac{\partial D_{FKL_i}}{\partial z_{\theta_j^i}} &= \frac{\partial}{\partial z_{\theta_j^i}} \sum_{k=1}^{V} p_{\phi}(x_0^i = k | \tilde{x}_t) \log \left( \frac{p_{\phi}(x_0^i = k | \tilde{x}_t)}{p_{\theta}(x_0^i = k | \tilde{x}_t)} \right) \nonumber \\
&= \frac{\partial}{\partial z_{\theta_j^i}} \sum_{k=1}^{V} -p_{\phi}(x_0^i = k | \tilde{x}_t) \log p_{\theta}(x_0^i = k | \tilde{x}_t) \quad \text{//From KL to cross entropy} \nonumber \\
&= \frac{\partial}{\partial z_{\theta_j^i}} \left( \sum_{k=1, k \neq j}^{V} -p_{\phi}(x_0^i = k | \tilde{x}_t) \log p_{\theta}(x_0^i = k | \tilde{x}_t) \right) - \frac{\partial}{\partial z_{\theta_j^i}} p_{\phi}(x_0^i = j | \tilde{x}_t) \log p_{\theta}(x_0^i = j | \tilde{x}_t) \nonumber \\
&= \left( \sum_{k=1, k \neq j}^{V} - \frac{p_{\phi}(x_0^i = k | \tilde{x}_t)}{p_{\theta}(x_0^i = k | \tilde{x}_t)} \frac{\partial}{\partial z_{\theta_j^i}} p_{\theta}(x_0^i = k | \tilde{x}_t) \right) - \frac{p_{\phi}(x_0^i = j | \tilde{x}_t)}{p_{\theta}(x_0^i = j | \tilde{x}_t)} \frac{\partial}{\partial z_{\theta_j^i}} p_{\theta}(x_0^i = j | \tilde{x}_t) \quad \text{//derivative of } \log(\cdot) \tag{14} \\
&= \sum_{k=1, k \neq j}^{V} p_{\phi}(x_0^i = k | \tilde{x}_t) p_{\theta}(x_0^i = j | \tilde{x}_t) - p_{\phi}(x_0^i = j | \tilde{x}_t)(1 - p_{\theta}(x_0^i = j | \tilde{x}_t)) \quad \text{//substitute Eq. (13)} \\
&= \sum_{k=1}^{V} p_{\phi}(x_0^i = k | \tilde{x}_t) p_{\theta}(x_0^i = j | \tilde{x}_t) - p_{\phi}(x_0^i = j | \tilde{x}_t) \nonumber \\
&= p_{\theta}(x_0^i = j | \tilde{x}_t) - p_{\phi}(x_0^i = j | \tilde{x}_t). \quad \text{//probability sums up to 1}
\end{align}$$
### B.2.2. Gradient of RKL

For each possible token $i$, we have:

$$
\begin{align*}
\frac{\partial D_{RKL_i}}{\partial z_{\theta_j}^i} &= \frac{\partial}{\partial z_{\theta_j}^i} \sum_{k=1}^{V} p_{\theta}(x_0^i = k | \tilde{x}_t) \log \left( \frac{p_{\theta}(x_0^i = k | \tilde{x}_t)}{p_{\phi}(x_0^i = k | \tilde{x}_t)} \right) \\
&= \frac{\partial}{\partial z_{\theta_j}^i} \left( \sum_{k=1, k \neq j}^{V} p_{\theta}(x_0^i = k | \tilde{x}_t) \log \left( \frac{p_{\theta}(x_0^i = k | \tilde{x}_t)}{p_{\phi}(x_0^i = k | \tilde{x}_t)} \right) \right) + \frac{\partial}{\partial z_{\theta_j}^i} \left( p_{\theta}(x_0^i = j | \tilde{x}_t) \log \left( \frac{p_{\theta}(x_0^i = j | \tilde{x}_t)}{p_{\phi}(x_0^i = j | \tilde{x}_t)} \right) \right) \\
&= \sum_{k=1, k \neq j}^{V} \left( -p_{\theta}(x_0^i = k | \tilde{x}_t) p_{\theta}(x_0^i = j | \tilde{x}_t) \log \left( \frac{p_{\theta}(x_0^i = k | \tilde{x}_t)}{p_{\phi}(x_0^i = k | \tilde{x}_t)} \right) - p_{\theta}(x_0^i = k | \tilde{x}_t) p_{\theta}(x_0^i = j | \tilde{x}_t) \right) \\
&\quad + p_{\theta}(x_0^i = j | \tilde{x}_t) (1 - p_{\theta}(x_0^i = j | \tilde{x}_t)) \log \left( \frac{p_{\theta}(x_0^i = j | \tilde{x}_t)}{p_{\phi}(x_0^i = j | \tilde{x}_t)} \right) + p_{\theta}(x_0^i = j | \tilde{x}_t) (1 - p_{\theta}(x_0^i = j | \tilde{x}_t)) \quad \text{//substitute Eq. (13)} \\
&= \sum_{k=1}^{V} \left( -p_{\theta}(x_0^i = k | \tilde{x}_t) p_{\theta}(x_0^i = j | \tilde{x}_t) \log \left( \frac{p_{\theta}(x_0^i = k | \tilde{x}_t)}{p_{\phi}(x_0^i = k | \tilde{x}_t)} \right) - p_{\theta}(x_0^i = k | \tilde{x}_t) p_{\theta}(x_0^i = j | \tilde{x}_t) \right) \\
&\quad + p_{\theta}(x_0^i = j | \tilde{x}_t) \log \left( \frac{p_{\theta}(x_0^i = j | \tilde{x}_t)}{p_{\phi}(x_0^i = j | \tilde{x}_t)} \right) + p_{\theta}(x_0^i = j | \tilde{x}_t) \\
&= \sum_{k=1}^{V} \left( -p_{\theta}(x_0^i = k | \tilde{x}_t) p_{\theta}(x_0^i = j | \tilde{x}_t) \log \left( \frac{p_{\theta}(x_0^i = k | \tilde{x}_t)}{p_{\phi}(x_0^i = k | \tilde{x}_t)} \right) \right) - p_{\theta}(x_0^i = j | \tilde{x}_t) \quad \text{//probability sums up to 1} \\
&\quad + p_{\theta}(x_0^i = j | \tilde{x}_t) \log \left( \frac{p_{\theta}(x_0^i = j | \tilde{x}_t)}{p_{\phi}(x_0^i = j | \tilde{x}_t)} \right) + p_{\theta}(x_0^i = j | \tilde{x}_t) \\
&= p_{\theta}(x_0^i = j | \tilde{x}_t) \left( \sum_{k=1}^{V} \left( -p_{\theta}(x_0^i = k | \tilde{x}_t) \log \left( \frac{p_{\theta}(x_0^i = k | \tilde{x}_t)}{p_{\phi}(x_0^i = k | \tilde{x}_t)} \right) \right) + \log \left( \frac{p_{\theta}(x_0^i = j | \tilde{x}_t)}{p_{\phi}(x_0^i = j | \tilde{x}_t)} \right) \right) \\
&= p_{\theta}(x_0^i = j | \tilde{x}_t) \left( \log \left( \frac{p_{\theta}(x_0^i = j | \tilde{x}_t)}{p_{\phi}(x_0^i = j | \tilde{x}_t)} \right) - D_{RKL_i} \right) \quad \text{//definition of RKL} \tag{15}
\end{align*}
$$

This is similar to the results derived for AR LLM in [111].

As a result, the approximated token-level gradients of FKL and RKL at each masked position $i$ in Eq. (5) can be calculated as follows:

$$
\begin{align}
\nabla_{z_{\psi}} D_{FKL_i} (p_{\phi}(x_0^i | \tilde{x}_t) || p_{\psi}(x_0^i | \tilde{x}_t)) &:= \nabla_{z_{\psi}} D_{FKL_i}(\tilde{x}_t) = (p_{\psi}(x_0^i | \tilde{x}_t) - p_{\phi}(x_0^i | \tilde{x}_t)), \nonumber \\
\nabla_{z_{\psi}} D_{RKL_i} (p_{\phi}(x_0^i | \tilde{x}_t) || p_{\psi}(x_0^i | \tilde{x}_t)) &:= \nabla_{z_{\psi}} D_{RKL_i}(\tilde{x}_t) = p_{\psi}(x_0^i | \tilde{x}_t) \left( \log \left( \frac{p_{\psi}(x_0^i | \tilde{x}_t)}{p_{\phi}(x_0^i | \tilde{x}_t)} \right) - D_{RKL_i}(\tilde{x}_t) \right). \tag{16}
\end{align}
$$

### B.2.3. Gradient of f-divergence

Our proposed token-level divergence can be seamlessly extended to general f-divergence [17] with the form [30, 82, 116]:

$$ D_{f_i} = \sum_{k=1}^{V} p_{\theta}(x_0^i = k | \tilde{x}_t) f \left( \frac{p_{\phi}(x_0^i = k | \tilde{x}_t)}{p_{\theta}(x_0^i = k | \tilde{x}_t)} \right). \quad (17) $$

When the generator function $f$ is differentiable, we can calculate its gradient as:

$$
\begin{align}
\frac{\partial D_{f_i}}{\partial z_{\theta_j}^i} &= \frac{\partial}{\partial z_{\theta_j}^i} \sum_{k=1}^{V} p_{\phi}(x_0^i = k | \tilde{x}_t) f \left( \frac{p_{\phi}(x_0^i = k | \tilde{x}_t)}{p_{\theta}(x_0^i = k | \tilde{x}_t)} \right) \nonumber \\
&= \sum_{k=1}^{V} \left( \frac{\partial p_{\theta}(x_0^i = k | \tilde{x}_t)}{\partial z_{\theta_j}^i} \left[ f \left( \frac{p_{\phi}(x_0^i = k | \tilde{x}_t)}{p_{\theta}(x_0^i = k | \tilde{x}_t)} \right) - \left( \frac{p_{\phi}(x_0^i = k | \tilde{x}_t)}{p_{\theta}(x_0^i = k | \tilde{x}_t)} \right) f' \left( \frac{p_{\phi}(x_0^i = k | \tilde{x}_t)}{p_{\theta}(x_0^i = k | \tilde{x}_t)} \right) \right] \right) \tag{18} \\
&= p_{\theta}(x_0^i = j | \tilde{x}_t) \sum_{k=1}^{V} \left( -p_{\theta}(x_0^i = k | \tilde{x}_t) \left[ f \left( \frac{p_{\phi}(x_0^i = k | \tilde{x}_t)}{p_{\theta}(x_0^i = k | \tilde{x}_t)} \right) - \left( \frac{p_{\phi}(x_0^i = k | \tilde{x}_t)}{p_{\theta}(x_0^i = k | \tilde{x}_t)} \right) f' \left( \frac{p_{\phi}(x_0^i = k | \tilde{x}_t)}{p_{\theta}(x_0^i = k | \tilde{x}_t)} \right) \right] \right) \nonumber \\
&\quad + p_{\theta}(x_0^i = j | \tilde{x}_t) \left[ f \left( \frac{p_{\phi}(x_0^i = j | \tilde{x}_t)}{p_{\theta}(x_0^i = j | \tilde{x}_t)} \right) - \left( \frac{p_{\phi}(x_0^i = j | \tilde{x}_t)}{p_{\theta}(x_0^i = j | \tilde{x}_t)} \right) f' \left( \frac{p_{\phi}(x_0^i = j | \tilde{x}_t)}{p_{\theta}(x_0^i = j | \tilde{x}_t)} \right) \right]. \nonumber
\end{align}
$$

As shown in Tab. 4, the generalized Jeffrey divergence belongs to f-divergence, with generator function $f(u) = ((1 - \beta)u - \beta) \log u$.
<table><caption>Table 4. Summary of some typical f-divergences $D_f(p||q)$ together with generator functions $f$, where $f: (0, \infty) \to \mathbb{R}$ is a convex function satisfying the condition $f(1) = 0$. This table is mainly adapted from [30].</caption><thead><tr><th>Name</th><th>D<sub>f</sub>(p||q)</th><th>Generator f(u)</th></tr></thead><tbody><tr><td>Forward Kullback-Leibler</td><td>∫ p(x) log <sup>p(x)</sup>&frasl;<sub>q(x)</sub> dx</td><td>u log u</td></tr><tr><td>Reverse Kullback-Leibler</td><td>∫ q(x) log <sup>q(x)</sup>&frasl;<sub>p(x)</sub> dx</td><td>− log u</td></tr><tr><td>α-divergence (α ∉ {0, 1})</td><td><sup>1</sup>&frasl;<sub>α(α−1)</sub> ∫ (q(x) <sup>(p(x))</sup>&frasl;<sub>q(x)</sub>) <sup>1−α</sup> − (1 − α) (<sup>p(x)</sup>&frasl;<sub>q(x)</sub>) − α) dx</td><td><sup>1</sup>&frasl;<sub>α(α−1)</sub> (u<sup>1−α</sup> − (1 − α)u − α)</td></tr><tr><td>Generalized Jeffrey</td><td>∫ [(1 − β)p(x) − βq(x)] log (<sup>p(x)</sup>&frasl;<sub>q(x)</sub>) dx</td><td>((1 − β)u − β) log u</td></tr><tr><td>Jensen-Shannon</td><td><sup>1</sup>&frasl;<sub>2</sub> ∫ p(x) log <sup>2p(x)</sup>&frasl;<sub>p(x)+q(x)</sub> + q(x) log <sup>2q(x)</sup>&frasl;<sub>p(x)+q(x)</sub> dx</td><td>−(u + 1) log <sup>1+u</sup>&frasl;<sub>2</sub> + u log u</td></tr><tr><td>Squared Hellinger</td><td>∫ (<sup>√p(x)</sup>&frasl;<sub>√q(x)</sub>)<sup>2</sup> dx</td><td>(<sup>√u</sup>&frasl;<sub>1</sub>)<sup>2</sup></td></tr></tbody></table>

<table><caption>Table 5. Generative performance on class-conditional ImageNet-256. Results of methods in type AR are taken from the DD [53]. Percentage drop values (relative to the teacher) are shown in parentheses.</caption><thead><tr><th>Type</th><th>Model</th><th>FID (↓)</th><th>IS (↑)</th><th>Precision (↑)</th><th>Recall (↑)</th><th>#Para</th><th>Step (↓)</th></tr></thead><tbody><tr><td>AR</td><td>VAR-d16 [105]</td><td>4.19</td><td>230.2</td><td>0.84</td><td>0.48</td><td>310M</td><td>10</td></tr><tr><td>AR</td><td>VAR-d20 [105]</td><td>3.35</td><td>301.4</td><td>0.84</td><td>0.51</td><td>600M</td><td>10</td></tr><tr><td>AR</td><td>VAR-d24 [105]</td><td>2.51</td><td>312.2</td><td>0.82</td><td>0.53</td><td>1.03B</td><td>10</td></tr><tr><td>AR</td><td>VAR-d16-DD [53]</td><td>9.94 (137%)</td><td>193.6 (16%)</td><td>0.80 (5%)</td><td>0.37 (23%)</td><td>327M</td><td>1</td></tr><tr><td>AR</td><td>VAR-d16-DD [53]</td><td>7.82 (87%)</td><td>197.0 (14%)</td><td>0.80 (5%)</td><td>0.41 (15%)</td><td>327M</td><td>2</td></tr><tr><td>AR</td><td>VAR-d20-DD [53]</td><td>9.55 (185%)</td><td>197.2 (35%)</td><td>0.78 (7%)</td><td>0.38 (26%)</td><td>635M</td><td>1</td></tr><tr><td>AR</td><td>VAR-d20-DD [53]</td><td>7.33 (119%)</td><td>204.5 (32%)</td><td>0.82 (2%)</td><td>0.40 (22%)</td><td>635M</td><td>2</td></tr><tr><td>AR</td><td>VAR-d24-DD [53]</td><td>8.92 (255%)</td><td>202.8 (35%)</td><td>0.78 (5%)</td><td>0.39 (26%)</td><td>1.09B</td><td>1</td></tr><tr><td>AR</td><td>VAR-d24-DD [53]</td><td>6.95 (177%)</td><td>222.5 (29%)</td><td>0.83 (-1%)</td><td>0.43 (19%)</td><td>1.09B</td><td>2</td></tr><tr><td>AR</td><td>LlamaGen-B [103]</td><td>5.42</td><td>193.5</td><td>0.83</td><td>0.44</td><td>111M</td><td>256</td></tr><tr><td>AR</td><td>LlamaGen-L [103]</td><td>4.11</td><td>283.5</td><td>0.85</td><td>0.48</td><td>343M</td><td>256</td></tr><tr><td>AR</td><td>LlamaGen-B-DD [53]</td><td>15.50 (186%)</td><td>135.4 (30%)</td><td>0.76 (8%)</td><td>0.26 (41%)</td><td>98.3M</td><td>1</td></tr><tr><td>AR</td><td>LlamaGen-B-DD [53]</td><td>11.17 (106%)</td><td>154.8 (20%)</td><td>0.80 (4%)</td><td>0.31 (30%)</td><td>98.3M</td><td>2</td></tr><tr><td>AR</td><td>LlamaGen-L-DD [53]</td><td>11.35 (176%)</td><td>193.6 (32%)</td><td>0.81 (5%)</td><td>0.30 (38%)</td><td>326M</td><td>1</td></tr><tr><td>AR</td><td>LlamaGen-L-DD [53]</td><td>7.58 (84%)</td><td>237.5 (16%)</td><td>0.84 (1%)</td><td>0.37 (23%)</td><td>326M</td><td>2</td></tr><tr><td>MDM</td><td>MaskGit [11]</td><td>6.60</td><td>224.07</td><td>0.831</td><td>0.402</td><td>174M</td><td>16</td></tr><tr><td>MDM</td><td>Di[M|O]</td><td>6.91 (5%)</td><td>214.05 (4%)</td><td>0.828 (0.4%)</td><td>0.377 (6.2%)</td><td>174M</td><td>1</td></tr></tbody></table>

## C. More Discussion on Related Works

Recent research has explored the possibility of converting AR models into discrete diffusion versions [27]. Given this ongoing work, we plan to investigate the applicability of our approach to AR models. Furthermore, while existing limitations of MDM are well known, an efficient MDM could still serve as a draft model for AR speculative decoding [16]. Diffusion distillation has seen widespread adoption of the GAN objective, either directly for distillation [91, 109, 115] or to enhance performance [51, 118, 124]. However, these methods are not directly applicable to MDM due to its discrete nature. In a similar vein, recent works based on SiD [60, 61, 123–125] aim to minimize the Fisher divergence or a generalized score-based divergence for distillation. However, these approaches require backpropagating the gradient through the teacher model—akin to adversarial training—which is infeasible for MDM due to the non-differentiable sampling operation. Xu et al. [116] recently introduced a general framework for distilling continuous diffusion models using f-divergence. At the cost of training another additional discriminator, this method successfully extends the DMD framework from RKL to general f-divergence by utilizing the output of the discriminator to weight the loss gradient in DMD. However, their method relies on the assumption that the teacher model and the real data used for training the discriminator obey the same underline distribution (otherwise need to simulate the teacher model to get more accurate synthetic data). The training of additional discriminator in general increases the computational overhead. Moreover, while effective for continuous models, their method relies on an additional GAN loss to achieve better performance, making it unsuitable for MDMs. To show the performance of our method, we compare it with DD [53], which distills AR models into one-step or few-step generators. As demonstrated in Tab. 5, our method yields significantly smaller performance drops compared to the teacher models than [53]. To summarize, our method successfully address the multi-token prediction challenge pointed out in [31, 99], by transfer the stochasticity into the model input and always predict tokens from the correct joint distribution of multiple tokens.
<figure><img src="image_9.png" /><figcaption>(a) MaskGit teacher performance under difference inference steps with different CFG. Metric calculated with 5k generated samples.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>(b) Performance of one-step generator distilled from MaskGit teacher under different temperatures. From left to right the temperatures are [11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.5, 4.0, 3.0, 2.0]. Metric calculated with 50k generated samples.</figcaption></figure>

<caption>Figure 7. More results from the Maskgit experiments.</caption>

<table><caption>Table 6. Control experiments on the initialization choices to validate the hypothesis in Sec. 4.3.</caption><thead><tr><th>Initial Method</th><th>Step (↓)</th><th>FID-5k (↓)</th><th>IS-5k (↑)</th></tr></thead><tbody><tr><td>use VAE encoded code of random noise</td><td>1</td><td>189.39</td><td>4.67</td></tr><tr><td>use 1 − r<sub>init</sub> random image tokens, r<sub>init</sub> fixed class token (e.g. 1025)</td><td>1</td><td>173.02</td><td>5.00</td></tr><tr><td>use 1 − r<sub>init</sub> random image tokens, r<sub>init</sub> fixed image token (e.g. 512)</td><td>1</td><td>188.3</td><td>4.65</td></tr><tr><td>use 1 − r<sub>init</sub> random class tokens, r<sub>init</sub> fixed mask token [M]</td><td>1</td><td>174.47</td><td>4.73</td></tr><tr><td>use 1 − r<sub>init</sub> random image tokens, r<sub>init</sub> fixed mask token [M] (ours strategy)</td><td>1</td><td>12.01</td><td>132.44</td></tr></tbody></table>

## D. More Experiment Setup

For the sampling of teacher model, we use the heuristic parallel sampler by default, with temperature=1.3, schedule mode arccos, and choose token to remask randomly during sampling (as the greedy approach by keeping the most confident tokens leads to degraded generation). In Fig. 7a, we show the teacher's FID with 5k generated images under-difference inference steps with different CFG. This result suggests the limitation of the parallel sampler for test time scaling, as pointed out in [81]. Given these results, we use 16 steps and CFG=2.5 for the teacher model by default in the paper. In order to add random perturbation to the token embeddings, we fix the embedding layer of all the models during distillation. During the experiment, we use the same mask schedule for getting the intermediate state $\tilde{x}_t$ from one-step model generation and training the auxiliary model. In addition, we choose the same schedule when training the teacher model. We use the arccos schedule for MaskGit distillation and the cosine schedule for Meissonic distillation, respectively. We adopt the loss weight from [119] with $w(t) = \frac{1}{p_{\theta}(x_0|\tilde{x}_t)-p_{\phi}(x_0|x_{init})}$. Our experiment suggests that this weight can prevent the gradient of the generator from exploding (while the one with 1000 times the gradient value can still generate a regular image). We use by default mixed precision training with bfl6 and a gradient clipping gradient normalization 1. We use a constant learning rate scheduler with a linear warmup of 100 steps. We use the adam optimizer with beta1 = 0.9 and beta2 =0.999 and no weight decay for all experiments. All temperatures for the three models are fixed to 1 during distillation. Exponential Moving Average (EMA) is applied with a rate of 0.9999 for all experiments. The codebook sizes for MaskGit and Meissonic are 1024 and 8192, and the sequence length of the latent codes for MaskGit and Meissonic are 1024 and 4096, respectively. For a sequence length of L, a codebook size of V, and the number of [M] tokens to be replaced $N \approx (1 - r_{init})L$, the possible initial token configure is $\binom{L}{N}V^N$. Given that L and V are usually large numbers, the possible initial token configure are sufficient to be the initial state. This explains why the noise perturbation yield marginal improvement, even though stabilize the training. The embedding layer in the one-step generator is fixed during distillation to improve the stability. By default, the FID, precision, recall, density and coverage are all calculated with features extracted from the InceptionV3 network.

## E. More Experimental Results

In this section we show more experiment results.

1. **IS results from ablation.** We present the IS results of the ablation in Fig. 8. Unlike FID, we found that the best IS is achieved at a much lower temperature. This suggest that the temperature can control the trade-off between FID and IS. We show visually how the
<figure><img src="image_11.png" /><figcaption>Figure 8. IS results of the ablations corresponding to Fig. 5. * means the training is collapsed and falls outside the comparable range with other results, we therefore show them in the sub-figures at the right upper corner with the same range of the x-axis.</figcaption></figure>

final output image changes with the generator temperature in Fig. 9.

2. **One-step generation metric score with different temperature settings.** In Fig. 7b, we show the FID and IS metric with varying temperatures of the generator. We observe that, as the temperature increases, the IS score decreases, while the FID score initially declines, reaching its minimum at a temperature of 7, before starting to rise again.

3. **Top-k visualization.** We apply the top-k trick when sampling each token from the predicted distributions $p_{\phi}(x_0^i|\tilde{x}_t)$ and $p_{\theta}(x_0^i|x_{\text{init}})$. As shown in Fig. 10, after distillation, our one-step generation with top-1 sampling and without top-k are nearly identical, unlike in the teacher model. This suggests that each initial code almost deterministically maps to a fixed set of image tokens, justifying the use of Gaussian perturbation for token embeddings in the generator.

4. **Analysis of output distribution.** To further investigate this, we examine the number of potential output tokens with probabilities greater than 0.001, as shown in Fig. 11a. Specifically, we analyze 256 randomly generated images (total 65536 tokens), where each token in every image has a total of 1024 possible output tokens (vocabulary size). We find that for nearly half of the tokens, the output logits from our distilled model collapse into a delta distribution, confirming the trends observed in Fig. 10 and Fig. 9. For comparison, Fig. 11b presents the corresponding distribution for the teacher model, where the potential output token probabilities are more evenly spread across different possible token values.

5. **A justification for $z_{\theta}(\tilde{x}_t) \approx z_{\theta}(x_{\text{init}})$**. As stated in Sec. 4.2, an approximation of the $z_{\theta}(\tilde{x}_t)$ is required to provide meaningful gradients for optimizing the student $\theta$. To justify our choice, we conduct an ablation study by comparing all three possible candidates to approximate $z_{\theta}(\cdot)$:

(1) $x_{\text{init}}$: the input used in the main paper;

(2) $\tilde{x}_t$: no approximation is used (while the student $\theta$ is trained to model $p_{\theta}(x_0^i|\tilde{x}_{\text{init}})$, here, we consider to directly use $z_{\theta}(\tilde{x}_t)$, which estimates $p_{\theta}(x_0^i|\tilde{x}_t)$);

(3) $\tilde{x}_{\text{init}}$: constructed by applying the initial mask from $x_{\text{init}}$ to the generated $x_0$, aiming to stay close to $x_{\text{init}}$.

Fig. 12 compares the visual quality of one-step generations at various training iterations when using these different inputs for loss calculation. We observe that using $x_{\text{init}}$ yields the best generation quality and fastest convergence. The alternative choices fail because they both lie outside the training initial distribution $p_{\text{init}}$ of the student $\theta$, making them unsuitable inputs. In particular, training with $\tilde{x}_t$ diverges due to its larger deviation from $p_{\text{init}}$.

6. **Information in the initial sequence.** Similar to several recent works on the influence of initial noise on the final generated images [13, 58, 78, 126]. In this work, we designed a token initialization strategy which injects randomness in the initial sequence $x_0$. We here investigate the influence in the initial code. For a distilled model trained with $r_{\text{init}} = 0.6$, we tested its performance using initial sequences composed of either 40% random image tokens or 40% image tokens derived from encoding real images with VQ-VAE. The results, shown in Fig. 13, present three image examples with three different random seeds applied to each for both the random image tokens case and the encoded image token case. We find that, unlike using random image tokens when the one-step generator can produce diverse images, using the real image tokens instead results in generations that closely resemble the original images. This suggests that, similar to continuous diffusion processes, the information contained in randomly initialized codes significantly influences the final generation. This property enables test-time scaling techniques for our model to improve performance [62, 107].

7. **Interpolation between initial token sequences.** In addition, we show the interpolation results between random initial sequence and encoded sequence in Fig. 14.

8. **Control experiments on token initialization strategy.** In our experiments, we conducted a controlled study on the token initialization strategy to verify our hypothesis in Sec. 4.3 that the initial sequence of the student should be similar to those used to train the teacher model (see Tab. 6). Similar to our main strategy, we replaced a fraction $1 - r_{\text{init}}$ of the `[M]` tokens with random *visual* tokens. However, in this case, we replaced the remaining `[M]` tokens with a fixed token, such as image token 512 or class token 1025. Additionally, we tested another variation where we replaced $1 - r_{\text{init}}$ of the `[M]` tokens with random *class* tokens while keeping the rest as `[M]` tokens. In both settings, the models either failed to generate meaningful images. This outcome reinforces the student model's preference for a
<figure><img src="image_12.png" /><figcaption>Figure 9. One-step generation at different temperature scale. From top to bottom the temperature is [1e-6,1e-3,1,10,20,50,100]</figcaption></figure>

‘familiar’ input, a hybrid of visual and [M] tokens, during distillation. Furthermore, we experimented with sampling random sequences encoded from Gaussian noise of the same size as an image. As shown in Tab. 6, the training of these control experiments leads to divergence metric values.

9. **Additional metrics (FID and CLIP score) on Meissonic.** We evaluated the FID, Fréchet Dino-v2 [70] Distance (FDD) and CLIP Score [33, 45] on the MsCoCo 30k validation dataset [52], as shown in Table 8. We use the default CFG=9 as suggested in the official codebase. Two key observations emerge from the results: (1) as the number of generation steps decreases, the teacher model’s FID deteriorates rapidly (e.g., 8-step FID is 96.75 compared to 64-step FID of 48.27); and (2) our one-step student achieves superior FID/FDD while maintaining a CLIP Score comparable to the teacher. This suggests that (i) our generator performs competitively with the teacher and (ii) FID/FDD may not always be a reliable fidelity metric, particularly for the Meissonic teacher model. We hypothesize that the biased style of the teacher model causes discrepancies compared to realistic images from the MS COCO dataset. Consequently, the distilled student model records better performance relative to the teacher.
<figure><img src="image_13.png" /><figcaption>Figure 10. The top two rows display results from the teacher model using 16-step generation with default setup, with the first row generated without top-k filtering and the other with top-1 sampling. The bottom two rows showcase results from our distilled one-step generator. The class label is 933.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Figure 11. Histogram of number of potential output with probability greater than 0.001</figcaption></figure>

10. **Additional HPSV2 results on Meissonic.** In Tab. 7, we present a comparison of our teacher model, Meissonic [4], across different classifier-free guidance settings. Our results indicate that Meissonic operates optimally at CFG= 9 for the HPSV2 metric.

11. We experimented with the Two Time-scale Update Rule [118] to update the fake score multiple times per iteration. However, this did not improve model performance or further stabilize the training loss.

12. Inspired by [35], we added soft targets for training the auxiliary model using the following loss function:

$$ \mathcal{L}_{\text{MDM}} = \mathbb{E}_{x_{\text{init}}, t} \left[ \gamma(t) \left( \mathbb{E}_{q_{t|0}}\left[(1-\alpha)(-\log p_{0|t}(x_{\theta}(x_{\text{init}})|\tilde{x}_t, \phi)) + \alpha D_{\text{KL}}(p_{\psi}(x_0|\tilde{x}_t)||p_{\theta}(x_0|x_{\text{init}}))\right) \right) \right] \quad (19) $$

where α is a hyperparameter controlling the interpolation between the hard target cross-entropy loss and the soft target KL loss.

13. The default CFG for MaskGit during distillation is 2. We also explored adaptive CFG, where the guidance scale varies with r_t, similar to the linear CFG used in MDM parallel sampling. However, this approach did not yield improvements.

14. Inspired by Proximal Policy Optimization (PPO) [94], we introduced an entropy bonus term $-p_{\theta}(x_0|x_{\text{init}}) \log p_{\theta}(x_0|x_{\text{init}})$ to encourage generation diversity. However, this did not show practical benefits.

15. While $\beta \in [-0.3, 1]$ works well for the ImageNet teacher, we found that in the Meissonic experiment, the distillation process diverges when $\beta < -0.1$ or $\beta > 0.2$.

16. In Fig. 15, we illustrate the consistency assumption visually.
<figure><img src="image_15.png" /><figcaption>Figure 12. Ablation on different choice for the approximation of $z_{\theta}(\tilde{x}_{t})$.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Figure 13. One-step image generation from encoded image tokens and random image tokens. The class labels are 417, 279 and 497, respectively.</figcaption></figure>

## F. Failure Cases

While the distilled one-step generator performs comparably to the teacher model in most cases, certain classes exhibit slight color shifts and mode collapse, as shown in Fig. 16. This discrepancy may explain the observed gap in evaluation metrics between the teacher and the one-step generator.

## G. Mode-seeking vs. Mode-covering

In Fig. 17, we show with a Gaussian example the visualization of the mode-seeking vs. mode-covering behaviors of the generalized Jeffrey divergence with different $\beta$. When $\beta$ is small, the divergence approaches the FKL, which is known for its mode-covering tendency, assigning high importance to matching all modes of the target distribution. As $\beta$ increases, the behavior transitions toward a balanced form, and for large $\beta$, the divergence exhibits a mode-seeking tendency, akin to the RKL, which focuses on fitting high-density regions while ignoring low-probability modes.
<figure><img src="image_17.png" /><figcaption>Figure 14. Interpolation between the encoded image token (left most column) and random image token (right most column) in the initial sequence. The class labels are 279, 284 and 207, respectively.</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>Figure 15. A visual representation of the consistency assumption. Ideally, the model's prediction based on the correct intermediate state, $\tilde{x}_t$, should be identical to its prediction derived from the initial sequence, $x_{init}$.</figcaption></figure>

<table><caption>Table 7. Complete HPS v2.0 benchmark. Scores are collected from https://github.com/tgxs002/HPSv2. We highlight the best.</caption><thead><tr><th colspan="7">HPS v2.0</th></tr><tr><th>Model</th><th>NFE</th><th>Animation</th><th>Concept-art</th><th>Painting</th><th>Photo</th><th>Averaged</th></tr></thead><tbody><tr><td>Latent Diffusion [83]</td><td>25</td><td>25.73</td><td>25.15</td><td>25.25</td><td>26.97</td><td>25.78</td></tr><tr><td>DALL-E 2 [79]</td><td>-</td><td>27.34</td><td>26.54</td><td>26.68</td><td>27.24</td><td>26.95</td></tr><tr><td>Stable Diffusion v1.4 [83]</td><td>50</td><td>27.26</td><td>26.61</td><td>26.66</td><td>27.27</td><td>26.95</td></tr><tr><td>Stable Diffusion v2.0 [83]</td><td>50</td><td>27.48</td><td>26.89</td><td>26.86</td><td>27.46</td><td>27.17</td></tr><tr><td>DeepFloyd-XL [19]</td><td>25</td><td>27.64</td><td>26.83</td><td>26.86</td><td>27.75</td><td>27.27</td></tr><tr><td>SDXL Base 1.0 [76]</td><td>50</td><td>28.88</td><td>27.88</td><td>27.92</td><td>28.31</td><td>28.25</td></tr><tr><td>SDXL Refiner 1.0 [76]</td><td>50</td><td>28.93</td><td>27.89</td><td>27.90</td><td>28.38</td><td>28.27</td></tr><tr><td>InstaFlow [55]</td><td>1</td><td>25.98</td><td>25.79</td><td>25.93</td><td>26.32</td><td>26.01</td></tr><tr><td>SD Turbo [90]</td><td>1</td><td>27.98</td><td>27.59</td><td>27.16</td><td>27.19</td><td>27.48</td></tr><tr><td>SwiftBrush v2 [18]</td><td>1</td><td>27.25</td><td>27.62</td><td>26.86</td><td>26.77</td><td>27.15</td></tr><tr><td></td><td>48</td><td>29.57</td><td>28.58</td><td>28.72</td><td>28.45</td><td>28.83</td></tr><tr><td></td><td>32</td><td>29.18</td><td>28.32</td><td>28.28</td><td>27.96</td><td>28.44</td></tr><tr><td></td><td>16</td><td>28.61</td><td>27.82</td><td>27.84</td><td>27.32</td><td>27.90</td></tr><tr><td rowspan="4">Meissonic (cfg=9) [4]</td><td>8</td><td>25.62</td><td>26.49</td><td>26.67</td><td>27.07</td><td>26.46</td></tr><tr><td>4</td><td>25.01</td><td>24.95</td><td>24.87</td><td>23.80</td><td>24.66</td></tr><tr><td>2</td><td>23.06</td><td>23.28</td><td>23.22</td><td>22.38</td><td>22.98</td></tr><tr><td>48</td><td>28.52</td><td>27.44</td><td>27.54</td><td>27.17</td><td>27.67</td></tr><tr><td></td><td>32</td><td>28.59</td><td>27.54</td><td>27.60</td><td>27.22</td><td>27.74</td></tr><tr><td></td><td>16</td><td>28.49</td><td>27.52</td><td>27.65</td><td>27.20</td><td>27.71</td></tr><tr><td rowspan="4">Meissonic (cfg=4) [4]</td><td>8</td><td>27.99</td><td>27.24</td><td>27.31</td><td>26.54</td><td>27.27</td></tr><tr><td>4</td><td>26.33</td><td>26.03</td><td>26.01</td><td>24.79</td><td>25.79</td></tr><tr><td>2</td><td>23.61</td><td>23.87</td><td>23.72</td><td>22.50</td><td>23.43</td></tr><tr><td>Di|[|||O</td><td>1</td><td>28.64</td><td>27.91</td><td>27.99</td><td>27.92</td><td>28.11</td></tr></tbody></table>

## H. More Qualitative Results

In Fig. 18, we present randomly sampled ImageNet images generated in a single step by our distilled models with MaskGit teacher. Fig. 19 compares our one-step generator with the Meissonic teacher model using different sampling steps. Notably, our one-step generation achieves superior visual quality compared to the teacher model’s 16-step generation. Finally, in Fig. 20, we provide additional text-to-
<table><caption>Table 8. Comparison of FID, FDD and CLIP-Score for Meissonic [4] across varying generation steps and our one-step generator. The results are evaluated on MSCOCO-val 30k dataset. The teacher CFG is set to 9.</caption><thead><tr><th>Steps (↓)</th><th>64</th><th>32</th><th>16</th><th>8</th><th>1 (ours)</th></tr></thead><tbody><tr><td>FID (↓)</td><td>48.27</td><td>50.13</td><td>63.29</td><td>96.75</td><td>38.45</td></tr><tr><td>FDD (↓)</td><td>620.9</td><td>625.6</td><td>709.6</td><td>980.8</td><td>548.6</td></tr><tr><td>CLIP-Score (↑)</td><td>0.321</td><td>0.318</td><td>0.307</td><td>0.280</td><td>0.322</td></tr></tbody></table>

<figure><img src="image_19.png" /><figcaption>Figure 16. Distribution shift in the student. For both classes, the top two rows are results from the teacher with 16 steps and the lower two rows are the results from our one-step generator. The class labels are 950 and 985, respectively.</figcaption></figure>

image one-step generation results from our distilled model with the Meissonic teacher.
<figure><img src="image_20.png" /><figcaption>Figure 17. Toy example to visualize the mode-seeking VS mode-covering behavior of different $\beta$ values in generalized Jeffrey divergence. We grid search the mean and std to minimize the generalized Jeffrey divergence.</figcaption></figure>
<figure><img src="image_21.png" /><figcaption>Figure 18. One-step samples from our class-conditional model on ImageNet.</figcaption></figure>
<figure><img src="image_22.png" /><figcaption>Figure 19. Comparison with the teacher: Meissonic [4] on different steps, we see clearly that the teacher model's results drop very quickly (e.g., around 4 steps).</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>Figure 20. Qualitative results of our one-step generator distilled from Meissonic [4].</figcaption></figure>
# I. Misc.

**Prompts** Below is a collection of creative prompts we used to generate images in Figs. 1, 6, 19 and 20:

* A plushy tired owl sits on a pile of antique books in a humorous illustration.

* A photograph of a woman from Steven Universe with gigantic pink ringlets and a white dress.

* A white bichon frise puppy dog riding a black motorcycle in Hollywood at sundown with palm trees in the background.

* A photorealistic image of a giant floating glass sphere in a rocky landscape surrounded by a gentle mist.

* A cosmonaut otter poses for a portrait painted in intricate detail by Rembrandt.

* A beaver in formal attire stands next to a stack of books in a library.

* An egirl with pink hair and extensive makeup.

* Portrait of a monkey wearing a spacesuit and an astronaut helmet.

* Closeup of a seinen manga film still showing the interior of a shinkansen train with a leather seat and a window view, with a hyperrealistic film still from a Nepali movie projecting in the background.

* Swedish lake at night with heavy snowfall depicted in hyper-realistic and detailed art.

* Pencil sketch of Danny DeVito by Milt Kahl.

* A realistic anime painting of a cosmic woman wearing clothes made of universes with glowing red eyes.

* Photo of Ty Lee from Avatar.

* A green field with flowers and pink and yellow clouds under a bright sun at sunset, illustrated by Peter Chan in a colorful Day of the Tentacle style on Artstation.

* A raccoon in formal attire, carrying a bag and cane, depicted in a Rembrandt-style oil painting.

* A girl in school uniform standing in the city.

* Serene, anime-style landscape with vibrant flowers and trees, picturesque clouds, and no signs of human activity.

* A kangaroo wearing an orange hoodie and blue sunglasses holding a sign in front of the Sydney Opera House.

* A pikachu in a forest illustration.

* An oil painting close-up portrait of a young black woman wearing a crown of wildflowers, surrounded by hazy golden light.

* A capybara wearing sunglasses.

* A frog wearing an anime-inspired onesie.

* A blue-haired girl with soft features stares directly at the camera in an extreme close-up Instagram picture.

* Digital art of Prince of Roses.

* A landscape featuring a Kyoto Animation-style building.

* A path winding through a forest depicted in digital art.

* A close-up portrait of a beautiful girl with an autumn leaves headdress and melting wax.

* A neon-soaked cyberpunk alleyway with rain-drenched streets and futuristic holograms, gritty yet vibrant, hyper-realistic, ultra-detailed, cinematic scene.

* A serene mountain landscape at sunrise, mist rolling over rugged peaks, ultra-detailed, photorealistic, soft lighting, high-resolution, digital art.

* A hyper-detailed closeup of a dew-covered insect on a vibrant leaf, extreme macro photography style, ultra-realistic, high-resolution, intricate textures.

* An anime-style magical girl in a dynamic pose, vibrant colors, ultra-detailed costume and background, energetic, high-resolution, cinematic lighting.

* An enchanted autumn forest with falling leaves and warm, glowing light, ultra-detailed, photorealistic, rich textures, digital art, serene mood.

* An elegant Renaissance portrait of a noble figure, detailed textures, soft natural lighting, ultra-detailed, classical, high-resolution, oil painting style.

* A cybernetic humanoid robot portrait with metallic textures and neon accents, ultra-detailed, photorealistic, cinematic, futuristic digital art.

* A vibrant street art mural on an urban wall, ultra-detailed, energetic, bold colors, high-resolution, digital painting, modern art style.

* A dark fantasy warrior in intricately detailed armor standing in a stormy battlefield, ultra-detailed, hyper-realistic, cinematic, dynamic action scene.

* An intense lightning storm over a vast desert landscape, ultra-detailed, dramatic, high-resolution, cinematic, digital art, atmospheric.

* A detailed nature macro shot of a vibrant flower with dewdrops, ultra-detailed, photorealistic, high-resolution, digital painting, delicate textures.

* An ultra-realistic snowy mountain village under a starry sky, ultra-detailed, atmospheric, cinematic, high-resolution, digital winter wonderland.

* A humorous portrait of a cat dressed as a Victorian aristocrat, in vintage photorealism.

* A photorealistic shot of a bouquet of wildflowers in a clear glass vase on a sunlit windowsill.

* A mecha jet fighter engages in an air battle with an explosion as a backdrop, set against a dark, starry sky in a highly-detailed art piece by Stephan Martiniere.
* A young black woman stands in front of a ringed planet in space.

* Digital art of a cherry tree overlooking a valley with a waterfall at sunset.

* An astronaut in white futuristic cybernetic armor running on the surface of the moon, featured in an artwork illustration on Artstation.

* The image is a headshot of a happy girl with white hair in a school uniform, illustrated by Ilya Kuvshinov.

* A minimalistic heart drawing created using Adobe Illustrator.

* The image is a digital art headshot of an owlfolk character with high detail and dramatic lighting.

* Close up of an eye with the Earth inside the pupil, inspired by Wes Anderson's art.

* Landrover drives through a rain-soaked forest in a highly-detailed digital artwork by Greg Rutkowski and Artgerm.

* A snowy lake in Sweden captured in a vibrant, cinematic style with intense detail and raytracing technology showcased on Artstation.

* A heron silhouetted against a beautiful sunrise, created by Greg Rutkowski.

* A surreal portrait of a woman with a giant carnation face in a flower field at sunset with colorful clouds and a large sky, created by artist Simon Stålenhag.