Scale-wise Distillation of Diffusion Models
Authors: Nikita Starodubcev
Ilya Drobyshevskiy
Denis Kuznedelev
Artem Babenko
Dmitry Baranchuk
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion distillation
few-step models
image generation
video generation
Primary area: generative models

Abstract: Recent diffusion distillation methods have achieved remarkable progress, enabling high-quality ${\sim}4$-step sampling for large-scale text-conditional image and video diffusion models. 
However, further reducing the number of sampling steps becomes more and more challenging, suggesting that efficiency gains may be better mined along other model axes. 
Motivated by this perspective, we introduce SwD, a scale-wise diffusion distillation framework that equips few-step models with progressive generation, avoiding redundant computations at intermediate diffusion timesteps. 
Beyond efficiency, SwD enriches the family of distribution matching distillation approaches by introducing a simple patch-level distillation objective based on Maximum Mean Discrepancy (MMD). 
This objective significantly improves the convergence of existing distillation methods and performs surprisingly well in isolation, offering a competitive baseline for diffusion distillation.
Applied to state-of-the-art text-to-image/video diffusion models, SwD approaches the sampling speed of two full-resolution steps and largely outperforms alternatives under the same compute budget, as evidenced by automatic metrics and human preference studies. Project page: https://yandex-research.github.io/swd.

Published as a conference paper at ICLR 2026
SCALE-WISE DISTILLATION OF DIFFUSION MODELS
Nikita Starodubcev
llya Drobyshevskly
Denis Kuznedeley Yandex Research
Yandex Research
HSE University Yandex Research
Artem Babenko Yandex Research
Dmitry Baranchuk Yandex Research
ABSTRACT
Recent diffusion distillation methods have achieved remarkable progress, enabling high-quality ~4-step sampling for large-scale text-conditional image and video diffusion models. However, further reducing the number of sampling steps be- comes more and more challenging, suggesting that efficiency gains may be better mined along other model axes. Motivated by this perspective, we introduce SwD, a scale-wise diffusion distillation framework that equips few-step models with progressive generation, avoiding redundant computations at intermediate diffusion timesteps. Beyond efficiency. SwD enriches the family of distribution matching distillation approaches by introducing simple patch-level distillation objective based on Maximum Mean Discrepancy (MMD). This objective significantly im- proves the convergence of existing distillation methods and performs surprisingly well in isolation, offering a competitive baseline for diffusion distillation. Applied to state-of-the-an pling speed of two full-resolution steps and largely outperforms alternatives under the same compute budget, as evidenced by automatic metries and human prefer- ence studies. Project page: https: /tttps:/yandex--ee research, github. 1o/swd
cext-to-image/videe diffusion models, SwD approaches the sam-
INTRODUCTION
Designing large-scale diffusion models (DMs) for high-resolution image and video generation is
inherently challenging due to the slow sequential sampling requiring 20-50 steps. Although state- of-the-art DMs typically operate in ~~8x lower-resolution VAE latent spaces (Rombach et al. 202
generation speed remains significant bottleneck, especially for recent large-scale models with
>
billion parameters (Sauer et al. 2024 Black Forest Labs 2024 Cai et al. 2025 Wu et al. 2025)
The challenge becomes several times more pronounced for video DMs (Polyak et al.) 2024a
Wan
et al. 2025 Kong et al. 2024), as the latent resolution also scales across the temporal dimension.
Previous works have made substantial efforts in DM acceleration from different perspectives eet al., 2022: Song et al. 2020a] Wimbauer et al.) 2024 Li et al., [003] Jin et al., (2025). One of
the most successful directions distilling DMs into few-step generators (Song al. 2023 Sauer
ett al. 20233] Yin et al.) 2024b Kim et al. 2024). where recent methods (Yin et al.) 2024a 2025
achieve -tate-of-the-art performance of large-scale text-conditional DMs for <4 steps. Notably, these approaches generally focus on reducing the number of sampling steps while freezing other promising degrees of freedom, such as model architectures or input resolution.
Recently, Rissanen et al. (2023): Dieleman (2024) drew parallels between the coarse-to-fine nature of image diffusion and implicit spectral autoregression. Specifically, the reverse diffusion process was shown to progressively predict higher spatial frequencies conditioned on previously generated lower frequencies via the input. This observation connects DMs to next-scale prediction mod- els (Tian et al.) 2024 Voronov et al.] 2024 Han et al. 2024), which also sample higher spatial frequencies at each step, but do so explicitly via upscaling. Therefore, these models suggest sig- nificant efficiency potential by performing most steps at lower resolutions. However, despite this connection, tfate-o--the-ar few-step DMs still operate at a fixed resolution throughout the diffusion process, highlighting underexplored direction for improving their efficiency.

---

Published as a conference paper at ICLR 2026
Contribution. Since most state-of-the-art DMs belong to the latent diffusion family (Rombach) et al., 2021). we first address whether the spectral autoregression perspective also applies to latent representations. In this work, we conduct spectral analysis of existing VAE latent spaces and also extend it to the video domain. Our findings confirm that both spatial and temporal latent resolutions implicitly increase over the diffusion process, similarly the natural images. This suggests that la- tent DMs can avoid redundant computations at intermediate noisy timesteps, where high frequencies are largely suppressed.
Motivated by this observation, we introduce Scale-wise Distillation (SwD) framework, which transforms an arbitrary pretrained DM into a single few-step model that progressively increases spa- tial and temporal sample resolutions at each generation step. SwD integrates seamlessly with exist- ing distribution matching distillation approaches (Sauer al. 2023 Yin et al. 2024b) and leverages their few-step sampling algorithms, which appear naturally aligned with progressive generation.
In addition to the scale- wise distillation framework, we present a simple yet surprisingly effective diffusion distillation objective that minimizes Maximum Mean Discrepancy (MMD) (Gretton al.) 2012) in the feature space of pretrained DM. The proposed objective complements state-of-the-arr distillation methods and achieves strong performance even in isolation, establishing competitive baseline for DM distillation. Importantly it requires no additional trainable models, making it com- putationally efficient and easy to combine with existing distillation pipelines.
We apply SwD to state-of-the-art text-to-image and video DMs and show that our models compete or even outperform their teachers being more than 10x faster. Compared to full-resolution few- step models, SwD significantly surpasses them under a similar computational budget. For the same number of sampling steps, SwD provides ~2 speedup in text-to-image generation and ~3x in text-to-video generation, without compromising quality.
RELATED WORK
Diffusion distillation into few-step models. Diffusion distillation methods aim to reduce gener-
ation steps to 1-4 while maintaining teacher model performance. These methods can be largely
grouped into two categories: teacher-followin methods (Meng et al. 2023 Song et al. 2023 Luo
al. 2023a Huang al.) 2023] (Song& & Dhariwal [024 Kim et al. 2025) and distribution match-
ing[
Yin et al. 2024b a: Sauer et al.) 2023 2024 Luo et al. 2023b Zhou et al. 2024b a).
folloin- methods approximate the teacher's noise-to-data mapping by integrating the dif-
fusion ODE in fewer steps than numerical solvers (Song et al.] 2020a Lu et al. 2022). Distribution matching methods relax the feacher-following constraint, focusing instead on aligning student and teacher distributions without requiring exact noise-to-data mapping. State-of-the-art approaches, such as DMD2 (Yin et al. 2024a) and ADD (Sauer et al.) 2023 2024), demonstrate strong genera- tive performance in <4 steps. However, they still exhibit noticeable quality degradation at 1-2 step generation. leaving room for further improvement. Recently, DMD has been successfully adopted
for video diffusion models (Yin et al. 2025 Huang al. 2025).
Progressive generation with DMs. The idea of progressively increasing resolution during diffusion generation was initially exploited in hierarchical or cascaded DMs (Ho et al. [022]] Saharia et al.) 2022 Ramesh et al.] 2022 Kastryulin et al. [222 Gu et al. 2023b), which are strong competitors latent DMs (Rombach et al. 2021) for high-resolution generation. Cascaded DMs consist of multiple DMs operating at different resolutions, where each model performs diffusion sampling from scratch, conditioned on the lower-resolution sample. To bridge progressive generation with diffusion processes, several works (Gu et al. 2023a Teng et al. 2023 Atzmon et al.) 2024 Jin et al.)
2025 Zhang et al. [2025] Anagnostidis et al. 2025 Haji-Ali et al. (025) have presented multi-stage
pipelines, where DMs are trained to smoothly transition to nigher-resolution noisy samples during diffusion sampling. SwD follows up this line research by proposing a framework that readily integrates into existing diffusion distillation procedures and adapts arbitrary pretrained DMs into progressive few-step models.
Maximum Mean Discrepancy in generative modeling. Maximum Mean Discrepancy (MMD) is a metric between two distributions and Q, widely explored in early GAN works (Binkowski et al. 2018 Wang et al. 2018 Dziugaite al. 2015 Bellemare al. 2017 Sutherland et al. 2016).
2

---

Published as a conference paper at ICLR 2026
1m600
SD5.5
(=#00
Wan2.1 10050
10
000
107
Power
10
10/
10
10P
Hoky
10-1
Larara
Laries
10
16
10"1
32-1
10"
16
18
Spatial Frequency
Spatial Erequescy
SpatilFFeqquee
Teuperal Frequescy
Figure 1: Spectral analysis of SD3.5 VAE latents (128x (Left) and Wan2.1 (21x60x 104) for spatial and temporal dimensions (Right). Vertical lines mark the frequency boundaries for which the frequency components the right are not present in lower resolution latents. Noise masks high frequencies, suggesting that latent DMs can operate at lower latent resolutions for high noise levels. Green area indicates the allowed latent resolution at corresponding timestep. while Red area shows that further resolution reduction would lead to noticeable information loss.
Given a positive-definite finction k(x, y), the MMD can be defined as MMD²((P.Q) Exxx'=p(x<xxx>>>>_>__ x')] + Ey.y'~q (k y']] 1 (
(1)
where x and y denote samples from the generated and target distributions, respectively.
Generative Moment Matching Networks (GMMNs) (Li et al. 2013) employ the MMD with fixed Gaussian kernel (RBF) directly in data space. GANs, in contrast, typically consider learnable ker- nels, designed as the composition of a discriminator with a fixed kernel.
In diffusion modeling, the MMD has been explored for DM training (Bortoli et al. 2025) finetun- ing (Aiello et al., )023). DMMD (Galashov et al. 2025) employs noise-adapte discriminators for MMD gradient flows (Arbel al. (019)) Recently, IMM (Zhou al. (025) leveraged the MMD for consistency distillation (Song et al. 2023), computing the MMD with a fixed kernel between raw generator predictions different timesteps. In our work, we adopt the MMD between student and teacher distributions in the feature space of pretrained DM, yielding a powerful and effective distribution matching objective.
LATENT SPACE SPECTRAL ANALYSIS
Rissanen al. (023) and Dieleman 2024) showed that, in pixel space, diffusion models approx- imate spectral autoregression for natural images. Since fftte-of-theart art text-conditional diffusion models operate on VAE latent representations (Rombach al. 202T), we first investigate this spec- tral perspective for various latent spaces and also extend it to the video domain.
Following Dieleman (2024), we evaluate radially averaged power spectral density (RAPSD), i.e., the averaged spectra power across different spatial frequency components, and its one-dimensional analogue for temporal frequencies video.
We examine the latent spaces of image and video diffusion models, specifically Stable Diffusion 3.5 (SD3.5) (Esser et al., (024) and Wan2.1 (Wan al. (025). The SD3.5 VAE maps 3x 1024x 1024 images into 16x 128 128 latents, while the 1an2.1 VAE encodes x1x3xx x 832 video inputs into 21x16x600 x 104 latents. Both models use a flow-matching process (Lipman et al. 2023).
F the RAPSD of Gaussian noise (blue), clean latents (purple) and noisy latents (orange) at different timesteps. Figure ! (Left) provides the results for SD3.5 VAE latents. Figure I (Right) shows RAPSD across both spatial and temporal frequencies of Wan2. latents. Vertical lines indicate frequency boundaries: the components to the right correspond to high frequencies absent at lower resolutions, while those to the left align with the full latent resolution (128x (128x128).
Additional results, including more timesteps and SDXL (Podell et al. 2024 latents under a variance- preserving diffusion process (Ho et al. 2020 Song et al. 2020b), provided in Appendix E
Observations. First, we note that the latent frequency spectrum approximately follows a power law, similar natural images (van der Arjen Schaaf & van Johannes Hateren 1996). In contrast, how-
3

---

Published as a conference paper at ICLR 2026
ever, highest frequency components in latent space exhibit slightly greater magnitude. We attribute this to the VAE regularization terms, which may cause "clean" latents to appear slightly noisy.
We also observe that the noising process progressively filters out high frequencies, thereby deter- mining the safe downsampling range without noticeable information loss. Figure | (Left) shows that at t=800, noise masks high frequency components emerging resolutions above 32x This allows for 4x downsampling of 128x x 128 latents (green area). On the other hand, 8x downsampling would corrupt the data signal (red area), as the noise does fully suppress those frequencies.
A similar effect is observed along the temporal dimension in Wan2.1 latents, see Figure [ (Right). At (0000, the effective signal can be represented with 111 latent frames instead of the original 21.
Practical implication. Based on this analysis, we suppose that latent diffusion models may operate lower resolution high noise levels without losing the data signal. In other words, modeling high frequencies at timestep is unnecessary if those frequencies are already masked at that noise level. Note that this holds true for both spatial and temporal axes for video DMs. We summarize this conclusion as follows:
Latent diffusion allows losouttennlution modeling at high noise levels across spatial and temporal dimensions.
4
METHOD
This section introduces scale-wise distillation, SwD, framework for diffusion models. We begin by describing the SwD pipeline, highlighting its key features and challenges. Then, we present our distillation objective based on Maximum Mean Discrepancy (MMD).
4.1 SCALE-WISE DISTILLATION OF DMS
The core design principle of SwD to unify multi-scale generation within a single few-step model and single diffusion process, in contrast to cascaded approaches. To this end, we define a few-step timestep schedule, [4.....,,N], and pair each diffusion timestep 4; with latent resolution 8; from non-decreasing scale schedule, (si...., sx]. Therefore, starting the generation with Gaussian noise at the lowest scale, $1, the resolution of intermediate noisy latents Xt, is progressively increased over sampling steps.
Upsampling strategy. We first address how to upsample intermediate xt, to obtain faithful higher- resolution noisy samples? A naive approach is to directly upsample xt,- However, this distorts the noise variance and introduces local noise correlations. Consequently, recent progressive DMs (Ji al. 2025 Atzmon et al. 2024) propose dedicated techniques handle jump points and preserve sampling continuity.
2023) by predicting In contrast, few-step models enable stochastic multistep sampling (Song et a clean sample Xo and then renoising it to the next noise level. This approach naturally suggests upsampling &o prior renoising, thereby preserving the correct noise statistics at higher resolution.
To validate this intuition, we generate images with Stable Diffusion 3.5 (Esser et al.) 2024) from intermediate noisy latents, X;, obtained with different upsampling strategies. Specif- ically, given a full-resolution (128x122 real image latent, Xp. and downscaled version (64x64), x xdown,
Configuration
==
400
1=
600
1 800
Xij
neise
9.2
10.3
13.7
down
Xay
Xr
30.9
18.8
14.7
dows
nriee
spocile
122
223
327
Table 1: Comparison of noisy latent upsampling strategies (B, C) for 64-+128 in generation quality (FID-5K), Upsampling x xdown before noise injec- tion (B) aligns better with original full-resolution noisy latents (A).
consider: (A) reference setting where noise added to the original x (B) upsampling xdownn followed by noise injec- tion; (C) injecting noise into down followed by upsampling.
As shown in Table I] the naive strategy (C) produces largely oft-of--ditrrbutte (OOD) noisy latents. In contrast, strategy (B) shows comparable results the original noisy latents at high noise levels,
4

---

Published as a conference paper at ICLR 2026
desa
Scale
Upecaee asise
downsaale
Distillation
Up
ruecting 200
Timestop
Prodet
Targett
Figure 2: SwD sampling. Few-step model starts sampling from noise the low resolu- tion 81 and gradually increases over gen- eration steps. At each step, the previous de- noised prediction at the scale 8;-1 is upsam- pled and noised according to the timestep schedule, t. Then, the generator predicts clean image at the current resolution s,.
Figure 3: SwD training step. i) Sample a pair of adjacent resolutions [5i, Si+1] from scale schedule. ii) Downscale the training images to 8; and $141- iii) The lower scale versions are upsampled and noised to a timestep l with the forward process. iv) Given the noised images, the model G predicts clean data at target scale 8/41- v) Distribution matching loss is calculated between predicted and target images.
likely because heavy noise masks interpolation artifacts. The results for other upscale ratios are presented in Appendix F
Overall, we propose handle scale-transition points by upsampling predictions, followed by renoising. In practice, we use bicubic interpolation for spatial dimensions and adjacent blend- ing for the temporal one.
Sampling, SwD performs progressive stochastic multistep sampling under the discussed scale- transition approach. Specifically, given the intermediate noisy sample x; at resolution 8;-1, the model predicts a denoised sample x) To proceed to the next timestep t,, X) is upsampled to s; and renoised with the forward diffusion process, resulting in higher-resolution sample &t,- Then, the model predicts the next x/- Figure illustrates this sampling process.
Although this procedure can be applied directly to pretrained few-step models, observe in prac- tice that renoising is insufficient to completely remove upsampling artifacts, unless very high noise levels used. Therefore, we aim train a few-step generator that also serves as robust upscaler.
Training. We train single model across multiple resolutions, iterating over pairs of adjacent scales [8t, 85-1] from the scale schedule. each training step, we sample batch of full-resolution images or videos, downscale them to the source and target resolutions in pixel space, according to the s; and 9;-1 scales, and then encode them into the VAE latent space. Notably, we find that downscaling in pixel space before the VAE encoding largely outperforms latent downscaling in our experiments.
Next, we upsample the lower resolution latents from 8; to 8:41 and apply the forward diffusion process according to the timestep schedule, t,. The noised latents are then fed into the scale-wise generator, which predicts Xo at the target scale 81+1.
Finally, we calculate distillation loss between the predicted and target latents at 8,41. In our work, we use distribution matching, motivated by ADD (Sauer al. [0033] 2024) and DMD (Yin et al.) 2024b a), achieving tlate-of-the-arl performance in diffusion distillation.
The schematic illustration of this training procedure provided Figure Further implementation
details and discussions are in Appendix A
Discussion on the timestep and scale schedules. Following Section 3 the emergence of higher- frequency components at lower noise levels can provide useful initial assumptions for designing the schedules. However, since the analysis provides only averaged results and does not account for upsampling artifacts, the schedules ultimately remain hyperparameters.
In practice, we adopt default few-step timestep schedules and slightly shift them toward noisier timesteps, aligning with the intuition that noise injection helps mitigate upsampling artifacts. The scale schedules are progressively increasing, starting with 2-4x lower resolution to achieve the desired inference speedups. We also find that the method not highly sensitive specific schedule
5

---

Published as a conference paper at ICLR 2026
choices, allowing them to be selected with minimal tuning and shared across models with similar diffusion processes.
4.2 DIFFUSION DISTILLATION WITH MAXIMUM MEAN DISCREPANCY
In addition to the proposed scale-wise distillation framework, we extend the family distribution matching distillation methods with patch-level MMD loss, calculated on the intermediate spa- tial features of the pretrained DMs. Below, we discuss the loss computation for transformer-bas based DMs (Peebles & Xie 2022). dominant architecture in state-of-the-art DMs, and note that its for- mulation readily extends to convolutional backbones.
First, we leverage the ability of DMs operate different noise levels, enabling the extraction of structured and finer-graine signals at high and low noise levels, respectively. Accordingly, before feature extraction, noise both generated and target samples within predefined timestep interval. In practice, we find that low-to-mid noise interval yields slightly better performance.
Then, we extract feature maps FERNxLXC from the middle transformer block of the teacher DM for generated and target samples and denote them as Ffak and Freal, respectively. N is a batch size, number of spatial tokens, and C' a hidden dimension of the transformer. We then compute the MMD between the distributions of spatial tokens, which correspond to patch representations in vision transformers (Dosovitskiy et al. 2020).
For the MMD computation, we consider two kernels: linear (k(x. y) y) and the radial basis function (RBF) (Chang et al., (2010). The former aligns feature distribution means, while the latter also matches all higher-order moments. In our experiments, both kernels perform similarly, so we simplify LMDD using the linear kernel, i.e., calculate MSE between spatial token means per image:
MMDD
F
plake
Note that the feature means computed across the entire batch rather than per image tend to mitigate condition-speccfic information, resulting in lower text relevance in our experiments.
Discussion. LAMD with a linear kernel can be considered as a diffusion distillation adaptation of the feature matching loss, proposed for improved GAN training (Salimans et al. 2016). To our knowledge, such losses have not been explored in the context diffusion distillation. The notable differences are: LMMD leverages pretrained DM instead of a learnable discriminator; ii) it uses the feedback from different noise levels; iii) the feature means are computed per image rather than across the entire batch.
Overall objective. We incorporate LMMD as an additional loss to LOMD and LaAN in our scale-wiss framework: LswD LAMD LDMD + 3 LGAN- Interestingly, despite its simplicity, LMMD is later shown to be a highly competitive standalone distillation objective.
5 EXPERIMENTS
Models. We validate our approach in text-to-image generation by distilling SDXL Podell et al. (004). SD3.5 Medium, SD3.5 Large (Esser et al., 2024) and FLUX.1-dev (Black Forest Labs,
2024.. We also apply SwD to the recent text-to-video model, Wan2.1-1.3B (Wan et al. (025).
Data. To remain in the isolated distillation setting and avoid biases from external data, we train all models exclusively on synthetic data generated by their teacher, rather than on real data. We note that this step does not pose bottleneck for training, as the distillation process itself converges relatively fast (~3K iterations) and requires significantly less data than the DM training. The synthetic data generation settings for each model are provided in Appendix A
Metrics. For text-to-image models, use 30K text prompts from the COC02014 and MJHQ sets (Lin et al. 2013: Li et al., 2024) and evaluate the automatic metrics: FID (Heusel et al. 2017). HPSv3 (Ma et al.) 2025), Reward (IR) (Xu al., 2023), and PickScore (PS) (Kirstain et al. 2023) and GenEval GGosh et al. 2023). Note that FID was shown to correlate poorly with human perception (Kirstain et al. 2023) for text-to-image assessment but report it here for completeness.

---

Published as a conference paper at ICLR 2026
FLUX-SwD
FLUX-Schnell
FLUX
SD3.5 Large SwD
SD3.5 Large
cuctus a happy face in Sahara desert.
An astronaut rides pig through in the forest. next to river, with clouds in the sky.
cloud dragon flying mountains, its body swirling with the wind.
Figure 4: Qualitative results of and SD3.5 Large SwD. More examples are in Figure 15
CausVid
CrusVia
SwD
SwD
Cinematic closeup detailed portrait of relndeer snovy... Figure 5: Qualitative results of Van2.11SwDD More examples are in Figure
A Samoyed Golden Retrlever dog are playfully romping--
Also, we conduct user preference study via side-by-side comparisons evaluated by professional assessors. We select 128 text prompts from the PartiPrompts dataset (Yu et al. (0222), following (Yin
et al. 2024a Sauer et al. 2024). and generate 2 images per prompt. More details are in Appendix H
For T2V models, we evaluate VBench-2.0 (Zheng et al. 2025). ) and VisionReward (Xu et 2024) and VideoReward (Liu et al. 2025) 1003 prompts from MovieGenBench (Polyak et al. 2024b).
steps. text-to-image models, Setup. our main experiments, we distill the models to 512x512 512 and progress to 1024x 1024. scale schedules begin at image resolutions of 256 For text-to-vidco, we start with 21x160x227 and achieve the 480 832 resolution. We chose such starting points lower resolutions provide only marginal speed improvements. The exact timestep and scale schedules for each model are in Appendix
Also, note that SDXL and Wan2.1 are distilled using only the MMD loss, the DMD loss per- forms poorly in the scale-wise setting when the base model fails to generate plausible low-resolution images. We discuss this limitation in more detail in Appendix [
Baselines. For text-to-image, we compare with the teacher models and their publicly available dis- tilled versions: DMD2-SDXL. (Yin et al. 2024a), SDXI--Turbo (Sauer al. 2023). Hyper-SI) (Ren

---

Published as a conference paper at ICLR 2026
Model
Steps
Latency, sfimage
Model size,
PS
HPSv3 1
IR
FID
PS
HPSv3
TR
FID
GenEval
COCO 30K
MITIQ
30K
Switti
0.44
2.5
22.6
11.1
0.98
20.0
21.6
9.8
0.84
8.9
0.62
Infinity
14
0.80
2.0
22.7 7
11.8
0.99
28.1
21.5
10.5
0.98
12.9
0.69
21.6
9.2
SDXL-TTubo
0.12"
2.6
222.6
10.0
0.83
17.5
21.33
9.6
0.84
15.4
0.55
Hyper-SI)
21.6
10.1
14.7
SDXL-DMID
0.20
2.6
22.
12.0
0.87
14.1
21.6
10.1
0.86
8.3
0.58
SDXI-SwD
21.6
SD3,5-M
10
1.8
2.2
22.1
10.2
1.00
16.3
21.6
9.9
0.97
9.5
0.69
SD3,,5-M- Turbo
0.96
2:2
222-2
D.83
17.6
21.3
9.3
0.74
13.6
0.50
SD3,5-M-SWD
0.19
2.2
22.8
11.7
1.12
23.1
21.8
10.7
1.10
13.1
0.70
SD3.5-L
8.3
8.0
22.5
11.3
1.06
16.5
21.8
10.4
1.04
10.7
0.70
SD3.5.I-Tuuro
0.63
22.8
0.90
22.6
21.7
9.9
0.9
13.5
0.70
SD3.5.L.SSD
0.39
8.0
22.8
12.8
1.20
20.6
21.8
11.1
1.22
13.9
0.71
FLUX
10.0
22.9
12.4
1.03
23.6
21.7
10.7
0.93
13.0
0.06
FLUX-Turbo-Ipph
2.75
12.0
23.1
13.4
1.08
21.2
21.5
11.2
0.97
11.3
0.66
Hyper-FLUX
2.75
23.0
12.4
0.94
24.2
21.7
10.9
0.85
14.9
0.61
FLUX-Schhell
1.41
12.0
22.6 6
11.2
1.01
16.5
21.5
10.3
0.96
9.8
0.69
FLUX-SwD
0.72
23.1
26.4
21.9
14.4
0.71
(*) (DXL only that generates at 512x512, while all other models produce
images.
Table 3: Quantitative comparison of SwD against other leading open-source models. Bold indicates the best pest-performing model within each DM group, while underline denotes the second best.
2024 FLUU-Schhnel (Black Forest Labs, 2024). FLUX- Turbo-Alpha (Tea 2024). Also, we evaluate other fast state-of-the- art models, such as next-scale prediction models (Switti Voronov et al. 2024) and Infinity (Han et al.) 2024)).
al. 2024). SD3.5-Turbo (Sauer al.
For the text-to-video task, we compare with the teacher model (Wan2.1 1-1.3B (Wan et al. 2025) and its 3-step DMD distilled variant, CausVid (Yin et al. 2025). ensure a fair comparison, we train Caus Vid using the official code and setup, but on our training dataset generated with 1- whereas the public version uses high-quality internal data.
5.1
MAIN RESULTS
Text-to-image. 3 Tigure/6ppreset the comparisons of SwD with the baselines in terms of generation quality and speed. The results organized into subsections corresponding to different base diffusion models.
We find that SwD models achieve state-of-the-art performance in terms of PS, HPSv3, IR and GenEval within their respective model families. In terms of latency, SwD shows nearly 2x speedup compared to the fastest counterparts.
According to the human study, SwD outperforms most other models in terms image complex- ity and image aesthetics, including the expensive teachers and their distilled variants, while maintaining comparable levels of text relevance and defects. We observe slight degradation in de- fects for FLUX-SwD compared the Hyper-FLUX and FLUX models, which are ~4x and ~ w14 slower, respectively. SDXL-SwD also has slightly more defects than SDXL-DMD2, which we at- tribute to the use of the MMD loss alone, discussed in Section 5.3 Qualitative comparisons are presented in Figure 4 with additional results in Figures 15
18
and
19
Text-to-video. The results in Table show that SwD achieves better performance than the teacher model, while being 72x faster. Compared to Caus SwD provides comparable results and
Latency, s/video
Vision Reward
Video Reward
VBench2 Overall
Model
Wan 2.1
137
0.038
3.43
51.6
Caus Vid
4.2
0.012
6.21
52.3
Spatial SwD
2.1
0.064
6.15
62.8
achieves 2..3 faster inference. Also, we find that SwD, when applied across both temporal and spa- tial dimensions, does not degrade quality compared to the spatial-only variant and further improves in- ference speed. Visual examples are provided in Fig- ures 5 14
(*) version trained our generated dataset using the official implomentation and sctup.
Table Comparison of 4-step SwD variants with the Wan 2.1 and 3-step Caus Vid models.
8

---

Published as a conference paper at ICLR 2026
Text Relevance
Image Aesthetics
Image Complexity
Defects
400-4
MICC M6-401
0014M
100
MIXS
Tuch-M
105 96-40
1DC591.660
Tech-M
100 5-355000
DM82 5801
105536660
0502 600
500541.600
BNDO SONE
500 5-3-5-00
SHI seSt
5035-6600
50:541-60
5005-99-00
5005-6600
501541-60
SWis
Svisl
201
1014.
60154
S1-
4015
Tem
Turte:
Terted
400 51-55
Terkel
21.1
Figure 6: Human preference study for SwD against the baseline models.
Seale-wise, steps
Full-scale, 2 steps
Full-scale, steps
Relevance
Aestbetics
Scale wise.
Full.
Scale milc,,
Full
wias
Full,
Scale-wisc
Full
Foll,
Scale-riss
dutcc baroque paiuting of horse field of flowers
Image: Complesity
Defectt
Scale wise,
Full.
Scale wicc,
Foll
Don't 't Panic!
Don't T Parlic!
Don't 't Panic!
Srale-wton,
Pull,
Scale-wite,
Foll
Scale
Pall.
Scale wise,
Foll
Scale-wiee
Foll
book witk the words 'Don't Punde!' written
Figure 7:
Visual examples (Left) and human preference study (Right) of the scale-wise and full- resolution settings within SD3.5 Medium. The numbers indicate the sampling steps.
5.2 SCALE-WISE VERSUS FULL- ROLL-RESOLUTIO
Next, compare SwD against their full-resolution counterparts. The full-scale baselines the same timestep schedules but operate at a fixed target latent resolution.
We provide the quality comparisons for the SD3.5 Medium and evaluate FLUX in Appendix D Comparing the settings for the same number of steps (4 vs 4, 6 vs 6). human evaluation (Figure Right) does not reveal any noticeable quality degradation. Qualitative examples (Figure 7 Left) further confirm this. Interestingly, automatic metrics (Tables 77 and R indicate that the scale-wise variants can even outperform their full-resolution counterparts, while being more efficient.
Then, we align generation times of scale-wise and full-resolution setups (4 vs. 2, 6 vs. 2 steps) to assess quality differences. Human evaluation reveals clear advantage the scale-wise setup, par- ticularly in reducing defects and improving image complexity. Examples in Figure (Left) highlight the high defect rates of the 2-step full-resolution baseline. Consistently, automatic metrics also show notable gains in HPSv3 and PS.
Runtime. Table reports per-image generation latency (including VAE decoding and text encod- ing), and Table 5 shows average training iteration time. Compared to the full-resolution setting with the same number steps (4 steps), the scale-wise setup achieves ~2x speedup both training and sampling across text-to-image models, and ~3x for text-to-video.

---

Published as a conference paper at ICLR 2026
Sctup
Steps
SD3,5-M
SD3.5-L
FLUX
Wan2.1
Sctup
Loss
SD3.5-M
SD3.3-L
FLUX
Wan2.1
Full-scale
4
0.29
0.63
1.41
5.51
Full-scale
LswD
7.5
13.4
22.8
70.6
Full-scale
2
0.16
0.33
0.72
2.97
Full-scale
1.0
1.7
2.9
12.7
Scale-wise
6
0.19
0.41
0.97
2.61
Scale-wise
CsuD
3.2
7.8
11.3
23.9
Scale-wise
4
0.17
0.32
0.72
1.84
Scalo-wise
0.1
141
Table 4: Sampling times (sec / image) of scale- wise and full-resolution setups. The measure- ment setting is described in Appendix G
Table 5: Training times (sec / iteration) for scale-
wise and full-resolution 4-step setups using the full objective (CswD) and MMD only (LSwD-MMD).
Text Relevance
Image Aesthetics
CSUD sclup
PS f
HPSvv3 f
IR1 f
FID
500.5-M Covo
505.-M
SDAS-M Cowb
SDAS-M Expo
LsHD (Main)
21.8
10.7
1.11
13.6
FLUX Ko0
FLUX 4p00
VLUX Eayl
PLUX Zoop
CAADD only
21.5
10.5
1.15
13.8
DsHo w/o LYMD
21.2
9.7
0.91
19.5
0.3
6.5
Image Complexity
Defects
LMMD Ablation
501.5-M Crwn
SD0.5-M Lones
KDXS-M Erwal
S0L15M Arw
A: RBF kemel
21.8
10.8
1,09
13.7
B: Batch averaging
21.5
10.5
0.97
16.4
FLLX Rowz
FLUX Apec
PLUX Eayt
FLUX Dop
C: w/o noising 21.3
10.2
1.01
16.6
6.5
Figure 8: Comparison of the main SwD models against the ones distilled with LMMD alone.
Table 6: Ablation study of the LMDD objective for
SwD on MJHQ30K.
5.3 ABLATION STUDY OF MMD LOSS
Here, we study the role of the LAMD loss and its design choices. Most experiments are conducted with the 6-step SD3.5-M setup from Section 5.1 with the MJHQ results reported in Table 6
We first assess the LMDD contribution to LSwD- We observe that training with LMMD alone slightly underperforms the full LSwD but remains effective removing it from LSwD leads to a significant drop in performance. Human evaluation (Figure X show that LoMy-only models exhibit noticeable degradation in defects, though not severe. Visual comparisons (Figure 16) confirm that they provide comparable performance to the full LSwD- More- over, as shown in Table LoMD-only training enables more than 7 faster iterations since avoids training extra models.
independent distillation method, whereas
Finally, we examine several LMMD variants. The LMMD with the RBF kernel (A) shows similar results. Referring to the feature matching (Salimans et al. 2016), we consider two changes, B: the feature tokens Equation (2) are averaged across the entire batch instead of per image, and C: extracting DM features only from clean samples, rather than noising them with the diffusion process. We observe that both and C make LMMD less effective.
6 CONCLUSION
We introduced SwD, scale-wise diffusion distillation framework equipped with a novel patch-level MMD- based distillation technique. We show that both components can be readily combined with existing offte-of-the-are distillation methods and lead to further efficiency and quality improvements for few-step models. We believe the proposed loss for DM distillation offers substantial potential for further development to pave the way toward a highly effective, self-contained distillation pipeline that eliminates the need for additional trainable models.
REFERENCES
Emanuele Aiello, Diego Valsesia, and Enrico Magli. Fast inference in denoising diffusion models via mmd finetuning. arXiv preprint arXiv:2301.07966 2023.
Sotiris Anagnostidis, Gregor Bachmann, Yeongmin Kim, Jonas Kohler, Markos Georgopoulos, Art- siom Sanakoyeu, Yuming Du, Albert Pumarola, Ali Thabet, and Edgar Schönfeld. Flexidit: Your diffusion transformer can easily generate high- samples with less compute. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 28316 223266.2666 2025.
10

---

Published as a conference paper at ICLR 2026
Michael Arbel, Anna Korba, Adil SALIM, and Arthur Gretton. Maximum mean discrepancy gradient flow. In Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-But Buc, E. Fox, and R. Gameett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL htttps::/proceeddinii: neur wwwrips.cco.. httpa/addnneeappr..cccccoeeeeeee ccpppperr firrriiii files/
Yuval Atzmon, Maciej Bala, Yogesh Balaji, Tiffany Cai, Yin Cui, Jiaojiao Fan, Yunhao Ge, Sid- dharth Gururani, Jacob Huffman, Ronald Isaac, et el. Edify image: High-quality image generation with pixel space laplacian diffusion models. arXiv preprint arXiv:24 1. 1,07226, 2024.
Marc G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The cramer distance as solution to biased wasserstein gradi- ents. arXiv preprint arXiv: 10743, 2017.
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton, Demystifying mmd gans. arXiv preprint arXiv:/801.01401,, 2018.
Black Forest Labs. Flux.1. 1-dev 2024.
Ollin Boer Bohan. Tachv: Tiny autoencoder for hunyuan video. https://github: com/
madebyollin/taehy 2025.
Valentin De Bortoli, Alexandre Galashov, J Swaroop Guntupalli, Guangyao Zhou, Kevin Patrick Murphy, Arthur Gretton, and Arnaud Doucet. Distributional diffusion models with scoring rules. In Forty-second International Conference Machine Learning, 2025. URL https: wtw.://wwnet//roomiaaaaaaa------
Qi Cai, Jingwen Chen, Yang Chen, Ychao Li, Fuchen Long, Yingwei Pan, Zhaofan Qiu, Yiheng Zhang, Fengbin Gao, Peihan Xu, et al. Hidream-il: A high-efficient image generative foundation model with sparse diffusion transformer. arXiv preprint 2xXi::5505.22055 2025.
Yin-Wen Chang, Cho-Jui Hsieh, Kai-Wei Chang, Michael Ringgaard, and Chih-Jen Lin. Training and testing low-degree polynomial data mappings via linear svm. Journal of Machine Learning Research, 11(4), 2010.
Sander Dieleman. Diffusion is spectral autoregression, 2024. URL https:// aii/ 2024/09/02/ 02 wewpeetall-attorrrrreeeeeeeeeeee
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. arXiv preprint arXiv:1505.00 2015.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. CoRR, abs/2403.03206, 2024.
Alexandre Galashov, Valentin De Bortoli, and Arthur Gretton. Deep MMD gradient flow without adversarial training. In The Thirteenth International Conference Learning Representations, 2025. URL
Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt, Geneval: An object- framework for evaluating text-to-image alignment, 2023. URL
11513
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The journal of machine learning research, 13(1):723-773, 2012.

---

Published as a conference paper at ICLR 2026
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Miguel Ángel Bautista, and Joshua M. Susskind. f-DM: A multi-stage diffusion model via progressive signal transformation. In The Eleventh Interna- tional Conference Learning Representations. 2023a. URL forum? d= BdwKTsg4m
net/
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M Susskind, and Navdeep Jaitly. Matryoshka diffusion models. In The Twelfth International Conference on Learning Representations, 2023b.
Moayed Haji-Ali, Willi Menapace, Ivan Skorokhodov, Arpit Sahni, Sergey Tulyakov, Vicente Or- donez, and Aliaksandr Siarohin. Improving progressive generation with decomposable flow matching. In The Thitty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis, 2024. URL
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. arXiv preprint arXiv:2106.15282, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https::/ /ttp:::://ppnnevieww net forum?
nZeeeeeee
Tao Huang, Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowl- edge diffusion for distillation. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023. URL
Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap autoregressive video diffusion. arXiv preprint arXiv:2506.. 08009, 2025.
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong MU, and Zhouchen Lin. Pyramidal flow matching for efficient video gener- ative modeling. In The Thirteenth International Conference Learning Representations, 2025. wtw.:e/w..eeeettooiidiideeeee
URL /ttpps://openreviiss net wor.mmidd 6d2ddd66622RRuu
Sergey Kastryulin, Artem Konev, Alexander Shishenya, Eugene Lyapustin, Artem Khurshudov, Alexander Tselousov, Nikita Vinokurov, Denis Kuznedelev, Alexander Markovich, Grigoriy Livshits, Alexey Kirillov, Anastasiaa Tabisheva. Liubov Chubarova, Marina Kaminskaia, Alexan- der Ustyuzhanin, Artemii Shvetsov, Daniil Shlenskii, Valerii Startsev, Dmitrii Kornilov, Mikhail Romanov, Dmitry Baranchuk, Artem Babenko, Sergei Ovcharenko, and Valentin Khrulkov. Yaart: Yet another art rendering technology. In Proceedings of the 31 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.I, KDD 225, pp. 2313 3-244.. New York, NY, USA, 2025. Association for Computing Machinery. ISBN 97984007 2456. doi: 10.1145/3690624. 3709404. URL htt.httt///ddo/ org/ 0.
3709404
Dongjun Kim, Chich-Hsine Lai, Wei- Wei-Hsiang Liao, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. Pagoda: Progressive growing of a one-step generator from a low-resolution diffusion teacher, 2024. URL. httpthtttt///attee http://wwwwvvvooooo////////// 14822
12

---

Published as a conference paper at ICLR 2026
Yeongmin Kim, Sotiris Anagnostidis, Yuming Du, Edgar Schönfeld, Jonas Kohler, Markos Geor- gopoulos. Albert Pumarola, Ali Thabet, and Artsiom Sanakoyeu. Autoregressive distillation of diffusion transformers. In Proceedings of the Computer Vision and Pattern Recognition Confer- ence, pp. 15745-15756, 2025.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. 2023.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03600 2024.
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation, 2024.
Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang. Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17535-17544 2023.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International conference on machine learning, pp. 1718-1727. PMLR, 2015.
Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl-lightning: Progressive adversarial diffusion distillation, 2024.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Lawrence Zitmick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015, URL
org abs http://b///1/0 1405.4055..0 0312
Yaron Lipman, Ricky T. Q Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023. htthttt::: hictpss:/epenreviieet www.rrewwwwwweeeeeeeeeeeeeeeeeee Eorum? 2ow.mm/i.d/ow/wwC......
Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng. Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, et al. Improving video generation with human feedback. arXiv preprint ar arXiv:2501.139 2025.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
Simian Luo, Yiqin Tan, Longbo Huang. Jian Li, and Hang Zhao. Latent consistency models: Syn- thesizing high-resolution images with few-step inference, 2023a.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang, Diff- instruct: universal approach for transferring knowledge from pre-trained diffusion models. In seveth- Conference on Newral Information Processing Systems, 2023b. URL https:
//openreview. wor.//oorurt/eeee d=ML I551
Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score, 2025. URL httlhhttss://axiit http://axxwvvv..gaaaaa// abs /2508 03789
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint 272004221..09748. 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL
httpss//speereviee. htt.:/iwwwweeeeeeeeeeeeeeeeete de
13

---

Published as a conference paper at ICLR 2026
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih Yao Ma, Ching- Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Ja- gadeesh, Kunpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sum- baly. Sai Saketh Rambhatla, Sam Tsai, Samaneh Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya, Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei-Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali Tha- bet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena Kerr, Carleigh Wood, Ce Liu, Cen Peng. Dimitry Vengertsev, Edgar Schonfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang, John Hoffman, Jonas Kohler, Kaolin Fire, Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du Movie gen: A cast of media foundation models, 2024a. CRLLhttps://// http://aaxvvv..o httpp/////.a.i....ooaa////////// 13720
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching- Yao Chuang, et al. Movie gen: A cast of media founda- tion models. arXiv preprint arXiv:2410.1377 2024b.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URL https://aaxiisi org/abs/
2204. 06125
Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, XING WANG, and Xuefeng Xiao.
Hyper-SD: Trajectory segmented consistency model for efficient image synthesis. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:// openreview. net. httperrwwww.eeeeeoomeedddd----e- htte://ormmmdo00x6o000
severi Rissamen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissi- pation. In International Conference on Learning Representations (ICLR), 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontjo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion mod- els with deep language understanding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:://pperrett
net http//oommm111=xx--000
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 29. Cur- ran Associates, Inc., 2016. URL UR//tppps//poeeeednn.eer....oo.. paper/201 fi httpa//e.ee.a.........eee2262... Paappp
cccpppeeeepepee..pppppee..... files/
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation, 2023.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rom- bach. Fast high-resolution image synthesis with latent adversarial diffusion distillation, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv
preprint arXiv:2010.02502 2020a.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference on Learning Representations, 2024. URL https://
openreview wtw.::w..ee....eeeeeeee wtw.orruummmiddaaaayyyybooooooo.
14

---

Published as a conference paper at ICLR 2026
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
Danica J Sutherland, Hsiao-Yu Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum mean discrepancy. arXiv preprint arXiv::611.0444000 2016.
AlimamaCreative
Team. HLUX.1-Turbo-Alpha. 1ww.-oo..---o------ pha 2024. Accessed: 2025-11-21.
http::://hugginnggne co/
al
Jiayan Teng, Wendi Zheng, Ming Ding, Wenyi Hong, Jianqiao Wangni, Zhuoyi Yang, and Jie Tang.
Relay diffusion: Unifying diffusion process across resolutions for image synthesis. arXiv preprint arXiv:2309.03350 2023.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. 2024.
van der Arjen Schaaf and van Johannes Hateren. Modelling the power spectra of natural im- ages: Statistics and information. Vision Research, 36:2759-2770, 1996. URL https://app
semanticschola:: wtg/CorpusII:: 18823055
Anton Voronov, Denis Kuznedelev, Mikhail Khoroshikh, Valentin Khrulkov, and Dmitry Baranchuk.
Switti: Designing scale-wise transformers for text-to-image synthesis. arXiv preprint arXiv:2412.01819, 2024.
Team Wan, Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang. Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang. Wei Wang. Wenmeng Zhou, Wente Wang. Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lx, Yifei Li, Yijing Liu, Yiming Wang. Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng. Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503. 20314, 2025.
Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In AAAI, 2023.
Wei Wang, Yuan Sun, and Saman Halgamuge. Improving mmd-gan training with repulsive loss 2018
function. arXiv preprint
Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, al. Cache if you can: Accelerat- ing diffusion models through block caching. In Proceedings of the IEEE/CVF Conference on 2024.
Computer Vision and Pattern Recognition, pp.
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang. Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng, Lin Qu, Minggang Wu, Peng Wang. Shuting Yu, Tingkun Wen, Wensen Feng, Xiaoxiao Xu, Yi Wang, Yichang Zhang, Yongqiang Zhu, Yujia Wu, Yuxuan ort..bs/.g/////////////
Cai, and Zenan Liu. Qwen-image technical report, 2025. URL 2508. 02324
Jaazheng Xu, Xiao Liu, Yuchen WU, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In seveth- Conference on Neural Information Processing Systems, 2023. URL https: //openrevies http://orrrmm...o//eeeeeeeeo.
15

---

Published as a conference paper at ICLR 2026
Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang. Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qumlin Jin, Shurun Li, Jiayan Teng, Zhuoyi Yang, Wendi Zheng, Xiao Liu, Ming Ding, Xiaohan Zhang, Xiaotao Gu, Shiyu Huang, Minlie Huang, Jie Tang, and Yuxiao Dong. Visionre- ward: Fine-grained multi-dimennional human preference learning for image and video generation, 2024. httttt:/:///axttttt www.rrwww...rrr.ooommmmmo abs 2412. 21059
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. In NeurIPS, 2024a.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Frédo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In CVPR, 2024b.
Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. In CVPR, 2025.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Yuechen Zhang, Jinbo Xing, Bin Xia, Shaoteng Liu, Bohao Peng, Xin Tao, Pengfei Wan, Eric Lo, and Jiaya Jia. Training-free efficient video generation via dynamic token carving. arXiv preprint arXiv:2505. 16864, 2025.
Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, and Ziwei Liu. VBench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:2503.21755 2025.
Linqi Zhou, Stefano Ermon, and Jiaming Song. Inductive moment matching. In Forty-second International Conference on Machine Learning, 2025. URL /
Eorum?
Mingyuan Zhou, Zhendong Wang, Huangjie Zheng, and Hai Huang. Long and short guidance in score identity distillation for one-step text-to-image generation. ArXiv 2406.01561, 2024a. URL
https://arxiv: wttp:////////../..//////// 01561
Mingyuan Zhou, Huangjie Zheng. Zhendong Wang. Mingzhang Yin, and Hai Huang, Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In International Conference on Machine Learning. 2024b. URL htt.hhtp:://// /tapps://eeee /tss:///xxvvv org/abs/ abs/
2404. 04057
16

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
Our few-step generator is initialized from pretrained teacher DM, with trainable LoRA adapters (Hu et al. 2022) added to the attention and MLP layers. We use a LoRA rank of 64 for SDXL, SD3.5-M, and SD3.5-L, and 128 for FLUX and Wan 2.1. However, the rank has negligible impact on performance in our experiments.
Training. The model is trained to minimize LSwD LMMD + LDMD + 3 LGAN-
LDD corresponds to the reverse KL divergence, which employs the scores of the "real" and "fake" distributions. The real score is estimated with the pretrained DM, while the fake score is modeled by training a separate "fake" DM on student-generated samples during distillation. The fake model parameterized with another set of LoRA adapters added to the teacher DM. Real score estimation uses CFG (Ho & Salimans 2022) with corresponding default guidance scales for each model.
LGAN- Following DMD2 Yin et al. 2024a), also include a GAN loss for text-to-image settings. The discriminator is a 4-layer MLP head operating on averaged spatial features extracted from the 11-th (SD3.5-M), 20-th (SD3.5-L), 15-th (FLUX, Wan2. transformer blocks of the fake DM. For SDXL, the feature maps are extracted from the middle UNet block. The LoRA adapters of the fake DM are also updated using the discriminator loss.
LMMD. For the proposed MMD loss, we use the timestep interval [0, 600) ((0, 400] for SDXL) to noise input samples prior to the feature extraction. The transformer blocks for feature extraction are the same as those used in the GAN setting.
For the SD3.5-M, SD3.5-L, FLUX models, we use all losses LwD. For SDXL and Wan2.1, 1, we use LMND only since LDMD degrades the performance for these models as discussed in the following section, while LGAN does not show noticeable improvements in our experiments.
Overall, the models are trained with learning rate of 4e-6 6 and batch sizes of 64 (SD3.5-M, SDXL) and 24 (FLUX, SD3.5-L, Wan 2.1) for ~3K iterations on a single node with 8 A100 GPUs.
Data. To exclude the effect of external data, train the models on the teacher synthetic data, generated prior to distillation using the standard teacher setups. SDXL: 50 sampling steps with guidance scale of 7.5, followed by 10 refiner steps. SD3.5 Medium: 40 sampling steps with guidance scale of 4.3. SD3.5 Large: 28 sampling steps with a guidance scale of 4.5. FLUX: 30 sampling steps with guidance scale of 4.5. Wan2.1: 50 steps with guidance scale of 5.0.
B IMPORTANCE OF SCALE-ADAPTED TEACHER MODELS
SD3.5,
SD3.5 256
Ouns,
Ours, 1021 (SW)
We also address the following important ques- tion: Does the teacher model need to be capa- ble of generating images at low scales prior to scale-wise distillation? The teacher model may not inherently handle low scales effectively, making scale-wise distillation more challeng- ing than the full-scale distillation. If this limita- tion is significant, additional pretraining of the teacher on small scales might be required.
SDKI,, 1004
SDXI,
1824
We evaluate the ability of SD3.5 Medium and SDXL to generate images at a lower resolution (556x266) The results are shown in Figure
K.yonng Andler skfiowedt wilif syllio noc, noby lomnrol paiiring
Figure 9: SD3.5 generates cropped images at low- resolutions (256x20 X26)), while SDXL does not pro- duce meaningful images at all. SwD success- fully adapts the model to both fixed low-resolution sampling (Ours, 256) and scale-wise sampling (Ours, 1024 SW) thanks to LGAN and LMMD. In contrast, LOMD tends to inherit teacher's limita- tions and thereby should be applied with caution or disabled altogether in such cases.
We observe that SD3.5 produces cropped and simplified images, but the overall quality re- mains acceptable, likely due to its pretraining at 256 x 256 resolution. SwD successfully dis- tills this model, mitigating the cropping prob- lem and increasing image complexity in both the fixed low-resolution setting (Ours, 256) and under scale-wise sampling (Ours, 1024 SW).
17

---

Published as a conference paper at ICLR 2026
Setup
Steps
PS
HPSv3
IR 1
FID 4
Setup
Steps
PS
HPSv3
IR
FID
SD3.5 Medium
SD3.5 Medium
Scale-wise
22.8
11.7
1.10
23.1
Scale-wise
6
21.8
10.7
1.10
13.4
Scale-wise
22.7
11.7
1.12
23.7
Scale-wise
21.8
10.7
1.13
13.7
Scale-wise
22.6
10.6
1.09
22.3
Scale-wise
2
21.7
10.3
1.10
12.8
Full-scale
22.5
11.2
1.08
20.4
Full-scale
6
21.6
10.3
1.09
13.4
Full-scale
22.5
11.3
1.09
21.2
Full-scule
4
21.7
10.4
1.10
13.5
Full-scale
2
22.3
10.8
1.03
20.3
Full-scale
2
21.5
10.0
1.04
13.1
FLUX
FLUX
Scale-wise
23.1
14.6
1.14
26.4
Scale-wise
4
21.9
11.6
1.06
14.4
Scale-wise
23.0
14.1
1.12
26.5
Scale-wise
21.9
11.5
1.10
14.0
Full-scale
23.1
14.0
1.13
28.5
Full-scale
4
21.8
11.3
1.09
14.4
Full-scale
23.0
13.8
1.13
26.9
Full-scale
2
21.8
11.2
1.08
13.4
Table Quantitative comparison between scale- wise and full-scale setups in terms of automatic metries on COCO30K.
Table 8: Quantitative comparison between scale- wise and full-scale setups in terms of automatic metrics on MJHQ30K.
Text Relevanee
Image Acsiictie
State-elie,
Pell
Scale-Mtie,
Pall
Sctup
PS
HPSv3 1
IR
FID
Sallo-bbe,
Pall
Sallowaac, 2
Pall
COC02014
Main
[32,48, 64, 80,96, 128]
22.8
11.7
1.10
23.
s= 664 64, 64, 128
22.4
10.3
1.02
23.7
Complcity
Deffecte
32, 32,32, 128]
22.3
9.8
0.97
23.8
Nallonebs,
PalL
Scall-Mtae,
Pall
MJHQQ
Main == [32,488 48, 80,96, 128)
21.8
10.7
1.11
13.6
PalL3
Scale-Mita,
PalL
8 [61, 61, 128)
21.3
9.8
1.06
14.6
32, 32, 32, 32, 128
21.2
9.4
0.99
15.7
Stale-rieo,
Pall
Scall-Mtte,2
Pall
Table 9: Comparisons to the "constant" scale schedules for SDD.55Medium SwD.
Figure 10: Human preference study comparing scale-wise and full-resolution FLUX setups.
More importantly, SDXL fails to generate faithful images at 256x256 resolution. Interestingly, SwD is able to recover from such poor starting point during distillation.
We attribute SwD 's ability adapt lower-resolution sampling the LGAN and LAND objectives that use low-resolution target images during distillation. In contrast, LDMD does not rely on reference images the target resolution. Empirically, find that LDVD degrades student performance when applied to teachers that struggle with low-resolution generation (e.g.. SDXL and Wan2.1).
In summary, LDMn is sensitive to the teacher's ability generate in low resolution and tends to inherit its limitations, whereas LGAN and LMND are significantly more robust.
Therefore, in our main experiments, we perform scale-wise distillation for the SDXL and Wan 2.1 models using LMMD only. We excluded LGAN for these models as it did not show significant additional improvements.
C SwD TIMESTEP AND SCALE SCHEDULES
Below, provide the timestep and scale schedules used in our main experiments. The scale schedule shows the resolutions in the corresponding VAE latent spaces.
SDXL. [1000, 800, 600, 400], [64, 80, 96, 128).
SD3.5 Medium. = [1000, 945, 896, 790, 737, 602). = [32, 48, 64, 80, 96, 128).
SD3.5 Large. ( [1000, 896, 737, 602). [64, 80, 96, 128).
FLUX. = [1000, 945, 790, 602). [32, 64, 96, 128).
Wan2.1. / [1000, 896, 737, 602), [6x20x34, 11x 32x 21
18

---

Published as a conference paper at ICLR 2026
steps
Full-scale, steps
SD3.5 Large
steps
Full-scale, steps
bluck siting
white black
Mack sitting
white back
7
7
7
cut parting cryssal ball
cat patting crystal
KEEP OFF GRASS
KEEP OFF
KEEP OFF THE TH GRS GRASS
KEEP OFF THE GRASS'
KEEP. THE GRASS
KEEP OFF THE GRASS
THE GRASS
The words 'KEEP OFF THE GRASS' writeon on brick well.
The words KKEP OFF THE GRASS' writte on a brick wall.
Figure 11: Visual examples of 4-step scale-wise and 2-step full-resolution SD3.5-Large settings.
Figure 12: Visual examples of 4-step scale-wise
and 2-step full-resolution FLUX settings.
D SCALE-WISE VERSUS FULL-RESOLUTION
Here, we provide additional results to compare the various scale-wise and full-resolution settings. Table and Table 8 present the automatic metrics for SD3.5 Medium and FLUX the COCO and MJHQ datasets. The visual examples for FLUX, SD3.5 Large, SD3.5 Medium are presented in Figures
12 and respectively.
In Table 9 we evaluate the use of "constant" 6-step scale schedules s= [64, 64, 64, 64, 64, 128] and s=|32, 32, 32, 32, 32, 128) for SD3.5 Medium contrast to the progressively growing schedule s= (3, 48 64, 80, 96, 128). used in our main setup. Note that the last step is required to be made the target resolution. The results show that is important gradually increase the resolution over sampling steps.
Finally, we investigate scale-wise distillation affects high-frequency details compared to the models distilled at the target resolution. We evaluate Patch FID (Lin et al. 2024) and Patch CLIP-IQA (Wang et al. 2023) on MJHQ30K. As shown in Table 10, SwD achieves similar scores to the full-resolution distilled models, indicating noticeable degradation of high- frequency details.
Patch FID
Patch CLIP-IQA
Model
Steps
SwD
Full-scale
SwD
Full-scale
SD3.5-M
6
20.1
21.1
0.85
0.85
SD3.5-L
20.4
21.9
FLUX
18.7
18.9
0.80
0.79
Table 10: Comparison of SwD and full-resolution distilled models in nigh-frequency detail preser- vation on MJHQ30K.
E EXTENDED LATENT SPACE SPECTRAL ANALYSIS
13 provides more results for the SD3.5 and Wan2.1 models and also includes the analysis for FLUX Back For-
We provide additional results for our latent space spectral analysis in Section Figure
est Labs| 2024) and SDXL. (Podell et al. 2024). In contrast to other models, the SDXL model (Podell
al. 2024)
variance- preseeciig (VP) diffusion process (Ho al. 2020
Song et al. 2020b).
The SDXL latent space has C=44 channels and 128 x 128 spatial resolution.
F EXTENDED ANALYSIS OF NOISY LATENT UPSCALING STRATEGIES
19

---

Published as a conference paper at ICLR 2026
In Table 11 present the extended com-
Configuration
1001
== 600
= 500
parison of noisy latent upscaling strategies for 32 x 32 + 128 128 and 96 96 128 x
128- 125 sike, 1288
Xi
9.2
10.3
13.7
yoak,
xi 2250022
90.2
10.8
16.8
128 setups. Specifically, given a full-resolution
50 sooks 12x122
217
298
(128x 128) real image latent, Xn, and its down- Xu The images are generated with Stable Diffusion
scaled versions Xo x222322 64 and X, ]28x128
04:04 speak, 126x:228 XL 1288128
ND
30.9
18.8
14.7
x44x66 spuode, 122
122
223
327
3.5 Medium (Esser et al. (024)) from interme- diate noisy latents, Xt. obtained with different upscaling strategies. We use default guidance scale of and the standard 28-step timestep schedule.
Mi
d6x00 sponk, 12 108 186 195
Xi
18.8
96-06 moke,
xc,
20000 upooid 125012
32.8
58.9
120
Table II: Extended results of the noisy latent upscaling strategies. We evaluate FID-5K on COCO2014 by generating images with SD3.5- M using obtained with different upsampling strategies discussed in Section 4.1
Upscaling the noised low-resoluton latent pro- vides poor performance across all upscaling factors. For upscaling Xy-predictions, as ex- pected, lower upscale ratios consistently lead to better results across all timesteps, since the predictions less distorted by upscaling. Also, we note that even for the x setup, the perfor- mance is still not fully recovered at 1=600, suggesting that the upsampled latents are still slightly shifted and require the scale-wise model training.
G RUNTIME MEASUREMENT SETUP
In our experiments, we measure runtimes in half-precision (FP16), using torch.compile for all mod- els: VAE decoders, text encoders, and generators. Note that, under our very fast sampling settings, the computational costs of the encoder and VAE start to account for noticeable portion of the overall runtime. Thus, we replace original VAEs with VAEY
VAEs (Boer Bohan
2025 for all models.
The measurements are conducted in an isolated environment on single A100 GPU, We use a batch size of 8 for all runs, and each measurement is averaged over 100 independent runs. The latency is then obtained by dividing the average runtime by the batch size.
H HUMAN EVALUATION
The evaluation conducted using Side-by-Side (SbS) comparisons, where assessors are presented with two images alongside a textual prompt and asked to choose the preferred one. For each pair, three independent responses are collected, and the final decision determined through majority voting.
The human evaluation is carried out by professional assessors who are formally hired, compensated with competitive salaries, and fully informed about potential risks. Each assessor undergoes de- tailed training and testing, including fine-grained instructions for every evaluation aspect, before participating in the main tasks.
In our human preference study, we compare the models across four key criteria: relevance to the textual prompt, presence of defects, image aesthetics, and image complexity. Figures 20 2 21
22 illustrate the interface used for each criterion. Note that the images displayed in the figures are randomly selected for demonstration purposes.
20

---

Published as a conference paper at ICLR 2026
STXX.
[w250)
(=:50
(w6000
[=300
hear
Nale Laken
Powert
10
State
Spotial Fesquency
Spotial Fleqpeecyy
Spotial Frequency
Syedial Fespuency
S133.5
1=250
1=500
t=800
t= 800
Nikin
Notay
10F
Lakee's
22433
Soc2 1
Poonee
10
307
107
10-11
10"
10n1
10
10
10
10
50
Spaatisl Prugurzcy
Speioll Progunncy
Speriod Frequancy
Special Proguacc
FLUX
1-200
t-500
1-000
6-000
10F-
Ponoer
324-2 (544)
107
de 3.3x
10/
10'
307
100
10-
Spatial Frequency
Spatial Frequency
Spucial Frequency
Spacial Frequancy
Wand.1
1=00
107
10P
101
35x04
107
Powert
10F
10-1
Saay Idends
10-1
Nakay Armuc
Lawes
10-2
16-1
302
16-1
UP
Spatial Frequescy
Temponal Fleuueacy
Special Prequsscy
Teupprral Freqeeny
Figure 13: Extended spectral analysis from Section 3 to more timesteps and models (FLUX, SDXL).
Bone
Dume
Puae
Kadas
20:1
$132
3040
3.5
833
BASC
33
11.8
2ake
$580
Table 12: Full comparison on VBench2.0 using all 18 metrics.
21

---

Published as a conference paper at ICLR 2026
Wan2.1
CausVid
SwD
A breathtuking image of a meteor colliding with the surfuce of a planet, with bright...
Wan2.1
Caus Vid
SwD
Animated scene features a close-up of a short fluffy monster kneeling beside a..
Wan2.1
Caus Vid
SwD
3D animation of small, round, fluffy creature with big, expressive eyes explores
Wan2.1
CausVid
SwD
A white and orange tabby cat is seen happily darting through dense garden, as if... Figure 14: Qualitative results of 1-
22

---

Published as a conference paper at ICLR 2026
FLUX-SwD
FLUX-Schnell
FLUX
SD3.5 Large Sw
SD3.5 Large
Sci-fi cosmic diarama of a and jellyfish in resin cuhe, volumetrie lighting, high resolution, hdr. sharpen, photorealismm.
Flying robot koi fish with armour plating, ncon glowing eyes and wiring, 4h, unreal engine, marvel comies style.
portrait enigmatic female oracle mysticul library filled with ancient and glowing orbs knowledge...
Belle epoque, christmas, red house in the forest, photo realistic, 8k.
lighthouse giant wave, origami style.
Figure 15: Qualitative results of FLUX-SwD and SD3.5 Large SwD.
23

---

Published as a conference paper at ICLR 2026
FLUX SwD
FLUX LMD
SD3.5 Medium LwDD
SD3.5 Medium LMD
A close-up high-contrast photo of Sydney Opera House sitting next to Eiffel tower...
blue Porsche 356 parked in front of a yellow brick wall.
BONEZ.
BONEZ. Z BONEZ
BBC NECZ
DROF
Keg be
A dog wearing baseball cup backwards and writing BONEZ chalkboard.
A doorknocker shaped like a lion's 's head.
PAN
cartoon of bear birthday party.
Figure 16: Qualitative comparisons SwD trained with the full LSwD loss against the ones trained with LAMD alone. LAMD in isolation produces competitive few-step models.
24

---

Published as a conference paper at ICLR 2026
Scale-wise, steps
Scale-wise, 4 steps
Full-scale, steps
Full-seale, 4 steps
Full-scale, steps
cat drinking pint of beer
black dog sitting between bush and pair of green pants standing up with nobody inside them
bird standing stick
drawing of house on mountain
SPAM
SMAKAE
SPAM
SPAM
SPAM SPAM
can of Spam on an elegant plate
Figure 17: Qualitative examples of image generations using scale-wise and full-resolution SD3.5 Medium SwD variants for different generation steps.
25

---

Published as a conference paper at ICLR 2026
SD3.5 Medium SwD
SD3.5 Medium
SDXL
DMD-SDXL
Switti
hard of buffato stampeding at the Kremlin
close-up high-contrast photo Sydney Opera House sitting next Eiffel under blue night sky of roiling energy. exploding yellow stars, and rudiating swirts of blue
cardboard spaceship
two motorcycles facing each other
baby duikon rudish tutu walking dog
Figure 18: Qualitative comparison of SD3.5 Medium SwD against the models of the similar size.
26

---

Published as a conference paper at ICLR 2026
SDXL-SwD
DMD2
SDXL
SDXL-Turbo
Hyper-SD
close-up of blue dragonfly daffodil
an overhead view of pickup truck with flatbed
I
II
A custle made of cardboard.
diplodocus standing front the Millennium Wheel
Figure 19: Qualitative comparison of SDXL-SwD against the SDXL-based alternatives.
27

---

Published as a conference paper at ICLR 2026
Prompt:
A medieval Scottish castle with grey stone walls and turrets, positioned mirror-like loch under misty sky, Highland
cows grazing in the foreground, rugged mountains rising in the distance.
Image
Which image better according the instructions?
9
The images are uncomparable
Quality: Brightness and contrast
Quality: Acidic unnatural colors
Quality: Glow
Image 1 is better
Image is better
Image better
Image 2 is better
Image is better
Image 2 is better
The images are equal in this aspect
The images are equal in this aspect
The images are equal this aspect
images were not evaluated this aspect
The images were evaluated for this aspect
The images were not evaluated for this aspect
Quality: Image is better
Quality: Image better
Aesthetics: Visibility of the main objects
Aesthetics: Background and environment
Aesthetics: Image detail
Image better
Image is better
Image better
Image better
Image 2 is better
Image better
The images equal in this aspect
The images are equal in this aspect
The images are equal in this aspect
The images were not evaluated for this aspect
The images were not evaluated for aspect
The images were not evaluated for this aspect
3
Aesthetics: Image is better
Aesthetics: Image 2 is better
Can't decide
Figure 20: Human evaluation interface for aestheties.
28

---

Published as a conference paper at ICLR 2026
cakk door and cowhiide leather belt
Defects in composition and watermarks
Image
Image in better
Images are equal
style
The imaes have the same style
mages images uncomparable
The verdict based staps
of objects
Image is better
Image is better
verdict based the previous
of secondary objects
la better
Image to better
based previous
1
Image 1 better
Image 2 better
The uncomparable
Enor loading images
Figure 21: Human evaluation interface for defects.
29

---

Published as a conference paper at ICLR 2026
oak door and cowhide leather belt
Main objects
there are more main objects Image
there are more main objects image
both images have the number of main objects
Final
Image is better
Image better
decide
loading images
Main objects
there are more main objects in Image
there are more main objects in Image
both images have the number of main objects
Secondary objects
there are more secondary objects in Image
there are more secondary objects in Image 2
both images have the same number of secondary objects ?
Image better in
effect objects
Image 2 is better in terms of the effect extra objects
both images have the same effect of extra objects ?
Final
Image
Image better
Can't decide
Error loading images
Figure 22: Human evaluation interface for relevance.
30

---

Published as a conference paper at ICLR 2026
Image1
Image
Which image more complex according to the instructions?
Image
Image 2 better
Can't decide
Figure 23: Human evaluation interface for complexity.
31


Reviews:
[{"review_id": "U8VJcvgIIG", "reviewer": "Reviewer_iUZm", "summary": "The paper introduces Scale-wise Distillation (SwD), which progressively increases latent resolution during few-step diffusion distillation.\nMotivated by spectral analysis showing that high-noise latents mainly contain low-frequency content, the method reduces redundant computation at early steps.\nAn additional MMD-based distillation loss is proposed and combined with DMD, showing competitive results on SD3.5, FLUX, and Wan2.1 models.", "questions": "Please see the part of Weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Clear and well-motivated idea linking noise level and latent frequency spectrum.\n* Practical framework that integrates smoothly with existing distillation methods.\n* The proposed MMD loss is easy to use, and effective even alone.\n* Good results with notable speedups for both text-to-image and text-to-video generation.\n* The writing is clear and the figures (e.g., Figure 1 spectral plots) effectively communicate the intuition behind the framework.", "weaknesses": "1. **Limited and possibly unfair baseline comparisons (SDXL case):**  \nTable 3 compares SwD results on SD3.5 and FLUX but omits SwD results on SDXL, even though Appendix B (Figure 9) shows that SDXL experiments were performed. For a fair evaluation, the authors should include SDXL-SwD and compare it directly to DMD2-SDXL and SDXL-Turbo. Moreover, other recent open-source distillation baselines such as Hyper-SD [Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis] should be incorporated.\nGiven that the paper emphasizes the synergy between MMD + DMD losses, fair quantitative evidence across identical model backbones is essential to support the claimed superiority.\n\n2. **Insufficient validation for FLUX and Wan distillations:**  \nThe comparison for FLUX and Wan 2.1 models is not convincing, as no competing baselines are provided. Stronger open-source baselines such as Hyper-FLUX, CausVid, or LightX2V (all publicly available and Wan-based distillation frameworks) should be included. Without these comparisons, it is difficult to judge the SwD’s efficiency and quality againest other distillation methods.\n\n3. **a minor issue** :  \nIn Table 3, the number of function evaluations (NFE) or diffusion steps used for each model is unclear, making speed–quality trade-offs hard to assess.\n\n\n4. **Relationship between MMD loss and prior progressive distillation methods (ADD/LADD) is not well clarified:**  \nThe MMD loss seems similar with the progressive distillation in ADD/LADD with different training objective. (constrain intermediate latent representations via feature alignment.) The paper should discuss more explicitly:\n* Whether the training procedure of “MMD-only distillation” is otherwise identical to ADD or DMD pipelines aside from the loss definition.\n* What specific advantages (e.g., stability, generalization) MMD introduces beyond computational simplicity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qYpTQQSy2J", "reviewer": "Reviewer_cK3V", "summary": "This paper addresses the inefficiency of few-step diffusion distillation methods, which redundantly compute all steps at full resolution. The authors propose two main contributions.First, Scale-wise Distillation (SwD), a framework where a single generator progressively increases its operating resolution during the few-step sampling process. This avoids unnecessary computation in early, noisy steps, where high-frequency detail is absent. Second, a simple and effective MMD-based distillation loss ($\\mathcal{L}_{MMD}$) that matches student and teacher feature distributions. This loss is shown to be a highly competitive baseline on its own and significantly speeds up training. Applied to SOTA models (SD3.5, FLUX, Wan2.1), SwD provides ~2-3x speedup over full-resolution models at the same step count and achieves significantly higher quality under the same computational budget", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The core idea of unifying progressive generation with few-step distillation (SwD) is novel and elegant. The motivation is strong, grounded in a solid spectral analysis (Section 3) of VAE latents for both images and video;\n2. The paper introduces a simple MMD-based loss that is surprisingly powerful. Ablations (Table 6) show it performs competitively on its own while being remarkably efficient. As it requires no extra trainable models (unlike GAN or DMD losses), it enables >7x faster training iterations (Table 5), making it a highly valuable standalone contribution;\n3. The experiments are comprehensive. The key comparison in Section 5.2 (Figure 7, Tables 7-8) clearly demonstrates SwD's superiority: at an equivalent compute cost (e.g., 4-step SwD vs. 2-step Full-res), SwD produces significantly better-quality images with fewer defects . The main results (Table 3, Figure 6) show SOTA performance, even outperforming teacher models in human preference;", "weaknesses": "1. The framework's performance depends on the co-design of the timestep schedule $[t_i]$ and the scale schedule $[s_i]$. While the authors provide the schedules used (Appendix D), the paper offers limited intuition on the methodology for finding these optimal schedules or the model's sensitivity to them.\n2. While the $ \\mathcal{L} _ {MMD} $ only variant is exceptionally simple, the full $ \\mathcal{L} _ {SwD} $ objective used to achieve the absolute SOTA results ($\\mathcal{L} _ {MMD}+\\mathcal{L} _ {DMD}+\\mathcal{L} _ {GAN}$) inherits the training complexity of methods like DMD2 (e.g., training a \"fake\" DM). However, the paper wisely presents the $\\mathcal{L} _ {MMD}$-only path as a highly effective, simpler alternative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1lEN22HfDZ", "reviewer": "Reviewer_h2GD", "summary": "This paper advances diffusion distillation by introducing a scale-wise few-step diffusion model. Instead of operating entirely at a fixed resolution, the model begins at a lower scale and progressively increases to the final resolution, similar to standard few-step diffusion approaches. Additionally, the authors propose a Maximum Mean Discrepancy (MMD)-based distillation loss to complement the existing DMD and GAN losses, further improving training effectiveness. Experiments conducted on various architectures—including SD 3.5 and FLUX for image generation (evaluated on COCO2014 and MJHQ) and WAN 2.1 for video generation (evaluated on VBench 2.0)—demonstrate the effectiveness of the proposed method.", "questions": "1. Why does the GenEval result for Infinity reported in this paper (0.69 in Table 3) differ from the value reported in the original Infinity paper (0.73)?\n\n2. In Equation (2), how are the two presented kernels applied in practice?\n\n3. Are the models initialized from pretrained weights or trained from scratch?\n\n4. Could the authors provide additional explanation for why the temporal dimension in SwD does not lead to performance improvements in Wan 2.1? (lines 375–377)", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a detailed spectral analysis of the latent spaces across different diffusion models, offering valuable insights that motivate the first contribution—scale-wise distillation.\n\n2. The proposed distillation framework is evaluated on multiple diffusion models, covering both image and video generation tasks. The results demonstrate the potential of scale-wise distillation to achieve higher efficiency while maintaining performance comparable to existing distillation methods.\n\n3. The paper is clearly written and well-organized, with logical section flow and well-explained experimental setups that effectively support its main conclusions.", "weaknesses": "1. The experiment on different upsampling strategies (lines 192–212) lacks evaluations with alternative scale configurations (e.g., from/to 32, 80, or 96), which would provide a more comprehensive and reliable analysis.\n\n2. The paper does not explain why the temporal dimension of SwD does not contribute to performance improvement when applied to Wan 2.1 (lines 375–377).\n\n3. The human preference study charts in Figures 6, 7, and 8 lack sufficient descriptive captions or visual annotations, reducing interpretability.\n\n4. In Table 3, the performance of SwD on 8B- and 12B-scale models shows only marginal gains over other distilled variants—for instance, SD3.5-L-Turbo (0.71 vs. 0.70) and FLUX-Schnell (0.71 vs. 0.69). Moreover, the human preference study (Figure 6) indicates that SD3.5-L-SwD and FLUX-SwD are outperformed by Turbo-L and FLUX-Schnell in most aspects, calling into question the practical advantage of SwD at similar scales.\n\n5. The paper omits comparisons with other few-step video diffusion models, such as Video LCM [1], T2V-Turbo [2], and MagicDistillation [3], which are relevant baselines.\n\n6. There is no discussion of Jenga [4], which shares a similar conceptual idea of adjusting scale with timestep (smaller scales for larger timesteps and larger scales for smaller timesteps).\n\n7. Table 3 lacks a description of the scale schedule corresponding to each timestep, making the setup difficult to reproduce.\n \n[1] https://arxiv.org/abs/2312.09109 \n\n[2] https://arxiv.org/abs/2410.05677 \n\n[3] https://arxiv.org/abs/2503.13319 \n\n[4] https://arxiv.org/abs/2505.16864", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F6qUDAXtLe", "reviewer": "Reviewer_8Ahm", "summary": "This paper proposes Scale-wise Distillation (SwD), a method that distills a diffusion model into a single model capable of progressively increasing resolution while generating images with only a few sampling steps.\n\nThe key ideas are:\n(i) Spectral analysis, which shows that in the early high-noise stages, high-frequency components are largely suppressed, thus computations can be performed at lower spatial/temporal resolutions to save cost;\n(ii) A progressive sampling and training procedure, where at each step the resolution is increased, and the previous step’s\nis upsampled → re-noised to serve as the input for the next step;\n(iii) A simple distribution matching loss that aligns teacher and student distributions in the DM feature space using Maximum Mean Discrepancy (MMD), particularly a linear-kernel variant (LMMD).", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-  **Method motivation**: Through RAPSD-based analysis, the paper convincingly demonstrates why low-resolution processing is safe during high-noise stages (Fig. 1). This motivation is appropriate as a methodological rationale; however, similar motivations have actually been used several times before from the diffusion efficiency literature. See Weakness.\n\n- **Solid experiments**: The approach is also extended to both text and video domains. The comparison of efficiency in Tables 4 and 5 is very appropriate, and the performance comparison is clearly shown in Figure 7. Under the same number of steps, SWD and the full model seem to have comparable performance, but SWD shows better efficiency.", "weaknesses": "- Recently, many diffusion efficiency studies have discussed that it is reasonable to focus on low-frequency components at time steps close to the noise. For example, [1] uses a transformer with a larger patch size at earlier (noise-near) time steps, and [2] proposes a method to better capture low-frequency information at each time step. It would be better if the motivation of this work were discussed in connection with these findings.\n\n- My greatest concern is whether the proposed method can effectively capture high-frequency details. Since the method focuses largely on low-frequency content except at the final step, this potentially poses a limitation. Moreover, because large-resolution images are involved, it may be difficult for the metrics to properly evaluate whether the high-frequency information has been preserved. I strongly recommend incorporating the measurement of FID-Patch as used in the SDXL‑Lightning paper to comprehensively assess fine detail preservation.\n\n\n[1] (CVPR 25) FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Compute\n\n[2] (CVPR 25) Autoregressive Distillation of Diffusion Transformers\n\n[3] SDXL-Lightning: Progressive Adversarial Diffusion Distillation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)