GAS: Improving Discretization of Diffusion ODEs via Generalized Adversarial Solver
Authors: Aleksandr Oganov
Ilya Bykov
Eva Neudachina
Mishan Aliev
Alexander Tolmachev
Alexander Sidorov
Zuev Aleksandr
Andrey Okhotin
Denis Rakitin
Aibek Alanov
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion models
diffusion acceleration
diffusion distillation
ODE solvers
adversarial training
Primary area: generative models
TL;DR: The paper introduces the Generalized Adversarial Solver, a simplified diffusion sampling acceleration framework combining distillation and adversarial training to achieve superior generation quality over existing methods.

Abstract: While diffusion models achieve state-of-the-art generation quality, they still suffer from computationally expensive sampling. Recent works address this issue with gradient-based optimization methods that distill a few-step ODE diffusion solver from the full sampling process, reducing the number of function evaluations from dozens to just a few. However, these approaches often rely on intricate training techniques and do not explicitly focus on preserving fine-grained details. In this paper, we introduce the Generalized Solver: a simple parameterization of the ODE sampler that does not require additional training tricks and improves quality over existing approaches. We further combine the original distillation loss with adversarial training, which mitigates artifacts and enhances detail fidelity. We call the resulting method the Generalized Adversarial Solver and demonstrate its superior performance compared to existing solver training methods under similar resource constraints.

Published as a conference paper at ICLR 2026
GAS: IMPROVING DISCRETIZATION OF DIFFUSION ODES VIA GENERALIZED ADVERSARIAL SOLVER
Aleksandr Oganoy Ilya Bykov*.), Eva Meudachina".,,Miin Mishan Aliev', Alexander Tolmachev', Alexander Sidorov, Aleksandr Zuev,
Andrey Okhotin Denis Rakitin', Aibek Alanov
HSE University, Russia
Lomonosov Moscow State University, Russia
FusionBrain Lab, AXXX, Russia
A Moscow Independent Research Institute of Artificial Intelligence, Russia
ABSTRACT
While diffusion models achieve state-of-the-art generation quality, they still suffer from computationally expensive sampling. Recent works address this issue with gradient- based optimization methods that distill a few-step ODE diffusion solver from the full sampling process, reducing the number of function evaluations from dozens to just a few. However, these approaches often rely on intricate training techniques and do not explicitly focus on preserving fine- fine-grained details. In this paper, we introduce the Generalized Solver: simple parameterization of the ODE sampler that does not require additional training tricks and improves quality over existing approaches. We further combine the original distillation loss with adver- sarial training, which mitigates artifacts and enhances detail fidelity. We call the resulting method the Generalized Adversarial Solver and demonstrate its superior performance compared to existing solver training methods under similar resource constraints. Code is available at httpp:: github. com/ 3atp/t115ttttttt..
INTRODUCTION
Diffusion models (Sohl-Dickstedd et al. 2015 Ho al. 2020 Song et al. 2020b) offer state-of-the art
generation quality in diverse vision problems. including unconditional and conditional (Dhariwal &
Nichol 2021 (Ho& & Salimans 2022) generation, text-to-image (Nichol et al. 2021 Ramesh et al.
2022
Rombach et al. 2022 Esser et al. 2024), text-to-video (Blattmann et al. Brooks et al. 2024 Zheng al. 2024 Chen et al. 2024b) and even text-to-31 (Poole et al.
Saharia et al.) 2022
2023
2022
Wang et al. (023)) generation. One of the reasons for their success consists in satisfying both
high sample quality (Dhariwal & Nichol 2021; Karras et al., 2022) and mode coverage from the generative trilemma (Xiao et al.) 2021). In theory, this allows diffusion models to produce desirable samples from the target distribution given unlimited computation time.
Besides, many improvements were made to satisfy the third requirement on generation speed. One way to tackle high inference time is to train new model that utilizes the pre-trained diffusion and requires fewer inference steps. This may be achieved by straightening the generation trajectories (Liu ctt al. 2022b 2023 Wang et al. 2024) or by directly performing diffusion distillation (Salimans & Ho 2022 Song al.) 2023 Sauer al.) 2023 Yin et al. 2023) into few-step student. These baining-based methods are capable of fast generation with superior quality on large-scale scenarios. Their training procedures, however, computation and memory-hear and may be infeasible for users with resource constraints on cutting-edge problems, such as video generation.
Due to the mentioned resource requirements, the lightweight approach of directly accelerating generation is preferable most of the time. Such inference-time methods as designing specific
solvers (Song et al., 2020a] [u et al. 2022a: Zhang & Chen 2022). caching intermediate steps (Ma) et al. 2024| Wimbauer et al. 2022),, or performing quantization (Gu et al. 2022 Badri & Shaji 2003). push the boundaries of the pre-trained model by utilizing its knowledge as much possible given
"Equal contribution. "Correspondence: 31 #.5ttt@ggma com

---

Published as a conference paper at ICLR 2026
Teacher NFF-10
UniPC NFF-4
NFF-4
Reaher NFF-10
UniPC NFF-4
GAS NFF-4
Figure I: Illustration of the Generalized Adversarial Solver image generation in comparison with the training-free UniPC (Zhao et al. 2024) solver with equal number of function evaluations (NFE). Our method shows superior results that are almost identical to teacher images in terms of generation quality.
a fixed computational budget. Among them, specifically designed solvers are mostly theoretically sound and are capable of producing high-quality samples similar to the full-inference model. However, they require significant hyperparameter search (Zhou et al. [022bb Zhao et al., 2024) for each model and may be suboptimal depending on the particular setting.
natural improvement of the idea consists of training (hyper-)parameters of the inference-time "student" sampler to match the full-inference "teacher" model. The approach is free-form and allows for optimizing timestep schedule (Sabour et al., 2024; Tong et al., 2024) as well as the sampler coefficients (Kim et al.) (2044 Frankel et al. (025)) for each prediction step. Currently existing
methods for training the sampler succeed in improving test-time efficiency of the model compared to the standard solvers. At the same time, they do not realize the full potential of the paradigm and
tend to have inefficiencies that lead to nuanced and complicated training schemes. Among these are the unstable loss scale (Sabour al. 2024), limited parameter space (Tong et al. 2024) and disentanglement of the parameter subsets (Frankel et al. 2025), which we find to be harmful for training. Besides, straightforward sampler distillation into student with limited parameters may be ineffective for preserving the fine-grained details and may interfere with the generation quality.
In this paper, we aim to tackle the aforementioned issues by introducing a simple yet effective sampler parameterization and modifying the distillation loss. Specifically, we construct sampler that performs each sampling step by calculating weighted sum of the current velocity direction with all of the points and directions from previous steps. We propose to utilize pre-defined solver as a time-dependent guidance and learn correction to its theoretically derived weights to facilitate and accelerate training. On top of that, we endow the sampler distillation with the adversarial loss (Godffello et al. 2014) to further boost the sampler quality. Most importantly, we
1. Introduce a novel sampler parameterization that we call the Generalized Solver and demon- strate its significant impact on training acceleration;
2. Combine it with the adversarial training and validate its positive impact on the fine-grained generation details;
3. Show that the resulting Generalized Adversarial Solver achieves superior results compared to the existing methods of solver/timestep training several pixel-space and latent-space data sets.
2 BACKGROUND
2.1 DIFFUSION MODELS
Diffusion models (Scki-Dickstein et al. 2015 Ho et al. 2020 Song et al. 2020b) simulate the data
distribution by defining the forward process of gradual data noising and constructing its time reversal. The forward process is commonly defined by sequence {Peo [eeoor) of transition probabilities (x)xx)/x)) V (x; a,xp, IT) It perturbs the initial data distribution Paata (xo) = (xx) by destroying part of its signal and replacing it with the independent Gaussian noise. Here, a and of positive differentiable functions that define the corresponding noise schedule. Typically, their choice ensures that the sequence of the corresponding marginal distributions pa (x,) converges to a simple and tractable prior distribution pr (xT) (e.g. standard normal). For each noise schedule one
2

---

Published as a conference paper at ICLR 2026
construct the equivalent Probability Flow ODE (PF-ODE) Song et al. 2020b)
1 - - dxt [10x= -
log (x)) dt,
(I)
where setting
f(t) dlog aq di
g² - do? cllo a4 dl
0f
(2)
and sampling the endpoint xy from the prior distribution pr ensures (Lu et al. 2022a) that x ~ P
for all timesteps. Essentially, ODE formulation allows one to obtain a backward process of data generation by reversing the velocity of the particle given access to the score function Vx, log pe (x,) of the perturbed data distribution. In practice, diffusion models approximate the score function by optimizing the Denoising Score Matching (Vincent 201 I) objective
min /' (x0, (xx - Xx, log (x))(
where the score functions Vx, log Pt (Xx of the conditional Gaussian distributions are tractable and equal to -xx, a,x0)/o? Besides the score networks, one can directly approximate the ODE velocity function by setting (x(, = x())x; - (1))))(())oo(()))))
2.2 ODE SOLVERS
Sampling from diffusion model amounts to numerically approximating the solution of the corresponding PF-ODE (Eq. D Standard numerical methods for solving general-form ODE dx, = u(x,, f)dl mainly based approximating the direction X++h x, via Taylor expansion.
The first-order Fuler scheme makes a step h v(x,, 1). which is simple, yet has a large discretization error. Its higher-order modifications generally approximate the derivatives with finite differences. This correction allows Runge- Kutta methods to produce high-quality results (Lu et al. 2022a] Zhang
Chen 2022 Karras et al. 2022). However, these methods require mid-point evaluations, which harms performance in Now-NFE regimes (see e.g. (Zhang & Chen 2022, Table 2)). In contrast, Linear
Multistep solvers (Liu et al. 2022a Zhang & Chen 2022) use only previously calculated points and directions for the same approximation, thus remain useful in this setting.
Recently designed solvers such as DDIM (Song et al., 2020a), (PM--olver(+++) (Lu et al. 2022a b). DEIS Zhang & Chen, 2022), and UniPC (Zhao et al., 2024), exploit the semi-linear nature of the PF-ODE (Hochbruck Ostermann 2010). They approximate the integral in the "variation of constants" formula
J" ai (T) s T)dr,
a, s(x,,
O2
allowing more accurate steps thanks to the oont coefficient of x,,, and enabling computationally efficient multistep solvers.
Several previous works highlight the importance of choosing the timestep schedule (the set of time points at which function evaluations are performed), which has a significant impact on the image generation quality (see (Krrras al. 2022 Appendix D.I) and (Frankel al., 2025 Appendix H.3)).
2.3 SOLVER AND SCHEDULE DISTILLATION
Several recently introduced acceleration methods outsource the choice of solver coefficients and the timestep schedule to the gradient- based optimization. Specifically, LD3 (Tong et al. (0224) and S4S (Frankel et al. 2025) formulate this an instance of knowledge distillation (Hinton et al. 20 15).. Given the tre-tedined diffusion model and the corresponding ODE dx; = v(x,, t)dt, one can define the complete "teacher" sampler to be the output of multi-step high-quality approximation of the PF-ODE, which we denote by
T (xx)
ODESolve (xr, v..)), + 0 Schedule, Solver; Params).
(5)
Here, Xy is the initial value, v(..) is the corresponding velocity field and - shows the interval, where solve the ODE. "Solver" and "Schedule" define the sampling scheme and "Params" account
3

---

Published as a conference paper at ICLR 2026
for the additional parameters of the scheme. Then, one could take any parameterization of the lightweight "student"
Φ (XT 0, = ODESolve (xT, o(. - 00 Schedule(O), Solver()); Params(()) (6)
with bounded computational requirements and optimize its parameters by minimizing distance d between the corresponding outputs
min dusstll (8, (. min (pr(xr) (odd((o (Ø (xr 0,0, $7 (xr)). 0.0.E
0,045
In addition, LD3 and S4S account for the limited parameterization of the student and simplify its objective by allowing to slightly adapt the input and facilitate replication of the teacher output
min Coon (0, Φ, {) = min (p((xr) 0,0,5
min x x,(x .or)
( (x 8, Φ. E); @ (xx)),
(8)
0.6.8
where (xx,, roT) fx: x 1| < rot) } is the ball centered in Xy with radius ror controlled by the additional hyperparameter т. We thoroughly discuss parameterizations of the methods and compare them with our Generalized Solver in Section 3.1
2.4 ADVERSARIAL TRAINING
Adversarial training Goodfellow et al. 2014) is powerful way to guide free-form generator Ga (z) towards realistic outputs via optimizing the minimax objective (Nowozin al. 2016)
min max E px) f (D (Ga(z))) + (x) (-Dw(x)),
(9)
Here, f(t) is commonly equal to - log(1 + e ") the discriminator D, is trained to distinguish real samples from the fake ones, while the generator aims trick it. Family of the GAN losses with the
form of Equation |9] (Nowozin et al.]|22)) [2016] Mao et al. 2017 Lim & Yel (017)) suffers from mode
collapse (Arjovsky al. 2017 Gulrajani al. 2017). One of the alternatives is the relativistic GAN loss Jolicoeur-Martineau 2018)
min max E (x(x)panxx (Go(z))) - (x
(10)
that is specifically designed to discourage mode dropping (Sun et al. 2020). Together with the gradient penalty
Canad (,, wh) A Vx Ap Vx (Go
(11)
on discriminator outputs and architecture improvements, relativistic loss allows Huang et al. (0024)
build novel high-quality GAN baseline R3GAN which we use throughout the paper.
METHOD
In this section, construct Generalized Adversarial Solver (GAS): automatic sampler learning method that combines simple yet effective parameterization with distillation and adversarial training.
3.1 GENERALIZED SOLVER (GS)
In Section 2.2cion2.22vv we have discussed that linear multi-step solvers and their specifically designed diffusion counterparts are the preferable families under strict requirements computations. Given a timestep
schedule = 4o > 4 > IN = and order K they all have the same signature
Xa+1 Xn +
Ojn Ux,,
(12)
max(x K+100)
where the coefficients On = an an, tn+1) and Cj.. == C (f)m+1) typically depend on the current and the next timesteps. We propose several modifications to this basic signature. First, stress that the less restriction on NFE is, the fewer parameters the method has. Second, one can see that depending on the parameterization of the diffusion model the formula may also contain the weighted
4

---

Published as a conference paper at ICLR 2026
Teache
u(. E.)
6o-1x00,0)
M 2 3
disteil + Loads
Student
Studmit
Studrak.
Students
GAS
Figure 2: Illustration of the Generalized Adversarial Solver. Our student makes each sampling step by calculating the weighted average of previous points and velocity directions. We train the corresponding weights and timestep schedule via distillation and adversarial loss.
sum of previous points (e.g., if one substitutes (イx with the network predictions. We thus propose to increase the capacity of the signature by adding the weighted sum all previous points and remove the restriction on the order of the solver:
(1/2) (1,))(X, b2)) along
Xa+11
aj + Σ (x,,((),
(13)
1=0
J=D
Given this signature, we next define our parameterization that has three sets of parameters: (0, db. E)- The first set / of parameters defines the timestep schedule via the cumprod transformation: the logits on are transformed into "stick breaking" portions (0,) E [0, 1]. The timesteps are then defined
1 = (T 8) II o (0,) +
(14)
The second set @ defines solver coefficients. However, we do not straightforwardly set aj.n = aj (0) and Cin = (Ø). Instead, we use a powerful base multi-step solver (e.g. (3M) (Lu et al. 2022b)) as a source of theoretical guidance for the trained coefficients. This base solver offers time-dependent theoretical coefficients an (t (fmm++) and cy. (t) (f)n+1))) ) which we can use as a strong backbone for our solver. We then train additive corrections to these coefficients
in the following way. We set
aj, (0, = annn ( + ân,a (0),
n
(15)
(o), else,
thus adding a trainable scalar (Ø) to the current point coefficient an and training scalars (d) for all the previous point coefficients.
Next. since the "old" velocities (computed more than K steps before) do not have theoretical coefficients, we train one scalar čj (o) per timestep j S K and
(0,n(0, () Cn (o).
(16)
Finally, define the coefficients before the "recent" velocities (computed less than K steps before). Here, theoretical base coefficients are typically constructed via weighted sum of the approximations
(0) of the higher-order derivatives v(j) (x))(x,,, ( via finite differences (which themselves weighted sums of previously computed velocities). This leads to the sum of the form Σ C) (t,(tt))+))) o(f)
"Theoretically, one could represent previous points linear combination of the previous velocity vectors. However, this "averpparamettrizaaion"" may simplify training.
5

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of solver parameterizations between our GS, LD3 (Tong al. 2024) and $4S (Frankel et al. 2025). We propose to add additive guidance to several velocity coefficients with theoretical term from pre-defined solver instead of just two multiplicative terms a, and bn- The guidance is marked by the dependence of coefficients on 0. We add weighted sum of the previous points to prediction.
Method
Parameterization
LD3
Xx+1 = an((n (x+1) Xn ]=max K+100)
Σ
s.n(ly..... (,t1) 0( 19 E,)
R
S4S
Xn+1 = an (1, (2+1) Xn (l l2+1)
(j,( v(x,, 13 Eg
j=max(n-K K+100)
GS
Xn+1
On,n () X djnn (jn(()) (0) Xj +
Σ Cjn (0, v(xj, + ==0
Combined with the finite-difference approximation of the derivatives 6/6) = Ei=n-j Win v(x;, !?)) we obtain
K-1
Σ 1=0
win v(x,, E Σ
(17)
(n-j
Here, we train additive corrections Cz.n (0) for the coefficients ejan (1f)m+4)) corresponding to the derivatives approximation. We thus obtain sum
K-1
Σ
() + mn))
Σ
Win v(x; (2),
(18)
1=0
which produces recent velocity coefficients
Ci (0, (0,
Win
Σ
[Cnn(Cmm + Cgn(
(19)
We initialize the corrections with zeros obtain efficient initialization. By doing this, ensure that even sudden change of the timesteps does not completely ruin the solver performance due the meaningful dependence of its coefficients time. We show the positive impact of the theoretical guidance in Section
The last set & of parameters acts correction the timesteps that evaluate the pre-trained model on. Analogous to Tong et al. (2024) and Frankel et al. (2025) we define the decoupled timesteps E, + E, and use them for making predictions with the diffusion model. Combining the signature from Equation 13 with the introduced parameterization, we obtain the Generalized Solver (GS)
n
n
Xx+1 On (0)
X
+
aj
Xj
Σ 1=0
(0, Φ) oxx,, Eg)
(20)
and extensively compare it with the parameterizations of LD3 and S4S Table !
Taken together, the following design choices of the Generalized Solver improve its quality over existing approaches:
the use of theoretical coefficients, which form the basis of GS and improve convergence;
the signature of the linear multistep method, on which many theoretical solvers are based, determines the use of the past history of xj;
additive parameterization, which connects the theoretical and trainable solver coefficients within the signature.

---

Published as a conference paper at ICLR 2026
3.2 GENERALIZED ADVERSARIAL SOLVER (GAS)
We train the Generalized Solver on the previously established distillation loss from Equation 7 Specifically, take d from the distillation loss (Equation 7 to be LPIPS in pixel-space and L in latent-space experiments, We do not use the soft version from Equation g It is important to examine the "solver distillation" " problem from another perspective. Essentially, is an instance of the paired translation problem//earning a mapping from its input/output samples. Several works (Isola al., 2017: Ledig al., 2017) have shown that the standard regression loss could greatly benefit from adding the adversarial loss on the outputs. Recently, adversarial loss has been established as powerful tool to boost performance of the diffusion distillation (Kim et al. 2023 Sauer et al. 2023
2024
Yin et al., 2024) methods.
Given this, we augment distillation -baeed training of the GS via distillation loss and obtain the Generalized Adversarial Solver (GAS). We denote solver's output
sh (xT Φ <) ODESolve(Xx, v(.)). ).
GS( o.)).
(21)
where GS(O, Ø, <) defines the Generalized Solver signature and parameterization, defined in Sec- tion 3.1 and Equation 20 specifically. We denote the discriminator by D and train GAS on the sum of distillation and adversarial losses
min max (o,s(O, (0,s,O,,,,,.)) 1)) = min max Laisni (0, ④) E)- + Ladv (0, 0, C, v);; 0,0.5 V
0.0.5 4
(22)
(0, G. v0) Epr (x)) (De (& (xr o, E) D2 (T7 (yr))).
We note that Xy and are different initial noises for student and teacher generation sampled from the same prior distribution. We exploit R3GAN (Huang et al.] 2024) relativistic loss with f(() log(1 + e) and add the discriminator gradient penalties from Equation 11 to facilitate its training dynamics.
The incorporation of the adversarial loss is also effective in terms of removing generation artifacts in low NFE regimes, where regression task becomes harder. We will further demonstrate this in Section 4
EXPERIMENTS
We demonstrate the efficiency of the proposed method by conducting experiments several pixel and latent space experiments. We perform evaluation on pixel-space CIFARIO x2) (Krizhevsky
Hinton| 2209), FFHQ 66) (Karras et al., 2019). and AFHQv2 (64x64) (Choi al.] 2020)) Among latent diffusion models (Rombach et al. 2022) we cover LSUN Bedroom (256x4 (566x256 256) (Yu
jt al., 2015) and the class- conditional ImageNet (256x 256) (Russakovsky et al. 2015 Additionally,
we assess the Stable Diffusion (Rombach et al.] 2022) model on the MSCOCO (12x5122) (Lin et al.
2015) text-to-image dataset. We use Karras et al. (2022) and andRombache et al. (002 pretrained models for pixel and latent space experiments respectively.
We choose distance d (Equation 7 in distillation loss to LPIPS (Zhang et al. 2018 in pixel-space and L1 in latent-space experiments. We initialize timesteps using lime-uniform schedule and utilize the (PMM-Solvr+++3MM (Lu et al. 2022b) coefficients as the guiding theoretical parameters. For pixel-space models we pretrained R3GAN discriminator. For latent experiments we adapt the same discriminator architecture, but train it from scratch. We calculate FID (Heusel et al. (2017) using 50000 samples, unless stated otherwise. The additional training details can be found in Appendix
4.1 MAIN RESULTS
In Table 2
we illustrate that the proposed methods, GS and GAS, systematically enhance image sampling quality across different solvers, especially in low NFE setups. As an example, the S4S Alt (Frankel et al. 2025) algorithm reports FID score of 10.63 with NFE=4 on the FFHQ dataset, whereas GAS achives a significantly better FID score of 7.86 under the same conditions. Our approach outperforms all previously proposed methods across all evaluated datasets. Specifically, GAS achieves a FID score of 4.48 with NFE=4 on the AFHQv2 dataset and 3.79 on the FFHQ dataset using NFE=6. Additionally, achieve the FID score of 5.38 on the conditional ImageNet dataset
7

---

Published as a conference paper at ICLR 2026
with NFE=4, 4.60 the LSUN Bedrooms dataset with NFE=5, and 14.71 the MS-COCC dataset with NFE = 4.
Table 2: We evaluate FID score comparison of the proposed GS and GAS methods against existing solvers like UniPC and iPNDM, and alongside training- approaches such as GITS, DMN, LD3, and S4S. We report the FIDs of the teacher models as those utilized during our training process. The baseline scores were taken from the corresponding papers, unless otherwise noted. report utilizing teacher model having significantly difference in teacher hyperparameters, thus it cannot be fairly compared to other methods.
(a) Pixel-space datasets include CIFARIO (32x32), AFHQv2 644x64), and FFHQ (64 64)
(b) Latent diffusion models are on the LSUN- Bedroom and ImageNet datasets x 256).
Method
NFE=4
NFE=6
NPE-10
Mothod
NFE=
NFE-5
NFE=6
NFB=7
CIFAR10
LSUN-Bedroom--25 256 (latent space)
Solvers
Solvers
DPM+ (3M)
46.59
12.16
4.62
3,08
DPM+ (3M)
48.82
18.64
8.50
5.16
UniPC (3M)
43.92
13.12
4.41
3.16
UniPC (3M)
39.78
13.88
6.57
4.56
iNDDM (3M)
35.04
11.80
5.67
3.69
iNDMM (3M)
11.93
6.38
5.08
4.39
Solver optimization methods
Solver optimization methods
UniPC [GITS)
25.32
11.19
5.67
3.70
UniPC (GITS)
70.93
47.37
22.33
17.27
UniPC [DMN]
26.35
5.90
2.45
UniPC [DMN]
29.22
4.40
4.55
iPNDM [GITS]
15.63
6.82
4.29
2.78
iPNDM [GITS]
76.86
59.17
28.09
19.54
iNDDM [DMN]
28.09
9.24
7.68
3.31
iNDMM [DMN]
11.82
6.15
4.71
5.16
Best LD3
9.31
3.35
2.81
2.38
Best LD3
8.48
5.93
4.52
4.16
S4S Alt
6.35
2.67
2.39
2.18
$4S Alt'
20.89
13.03
10.49
10.03
GS (Ours)
GS (Ours)
9.83
5.32
3.77
3.34
GAS (Ours)
4.05
2.49
2.24
2.17
GAS (Ours)
6.68
4.60
3.77
3.36
Teacher
2.03
Teacher
3.06
FFHQ
Imagenet-256 (latent space)
Solvers
DPM++ (3M)
46.14
14.01
6.18
4.18
DPM+ (3M)
26.07
11.91
7.51
5.95
UniPC (3M)
53.25
11.24
5.59
3.90
UniPC (3M)
20.01
8.51
5.92
5.20
iPNDM (3M)
36.54
16.44
8.11
5.39
iPNDM (3M)
13.86
7,80
6.03
5.35
Solver optimization methous
Solver optimizarion methods
UniPC [GITS]
21.38
12.21
7.84
4.46
UniPC (GITS)
$4.88
34.91
14.62
9.04
UniPC [DMN]
25.82
9.47
6.85
3.54
UniPC [DMN]
16.72
7.96
7.54
7.81
iNDM (GITS]
18.05
9.38
5.72
3.96
iNDDM [GITS]
56.00
43.56
19.33
10.33
iPNDM [DMN]
31.30
12.12
11.00
5.24
iPNDM [DMN]
10.15
7.33
7.25
7.40
Best LD3
17.96
3.50
3.25
Best L.D3
9.19
4.46
4.32
$4S Alt
10.63
4,62
3.15
2.91
GS (Durs)
10.70
4.49
2.96
2.67
GS (Ours)
7.87
4,93
4.30
4.17
GAS (Ours)
3.79
2.87
2.66
GAS (Ouns)
5.38
4.87
4.32
4.17
Teacher
2,60
Teacher
4.10
AFHQv2
(c) Training datasct for SD consists 1000 MS- COCO samples, while FID computed across 30,000 prompts to generate images with spatial reso- lution of 512x512.
Solvery
DPM++ (3M)
27.82
10.72
4.28
3.19
UniPC (3M)
33.78
8.27
4.60
3.81
INDMM (3M)
23.20
9.55
4.49
3.19
Solver optimization methods
Method
NFF=4
NFE=5
NFF56
NFB
UniPC [GITS]
12.20
3.86
2.88
UniPC [DMN]
30.32
14.46
6.85
2.94
MS-COCO (Stable Diffusion v1.5)
iPNDM [GITS]
12.89
6.10
4.03
3.26
iPNDM (2M)
17.76
14.41
13.86
13.76
iPNDM [DMN]
33.15
16.01
10.12
3.22
iPNDM [GITS]
18.05
14.11
12.10
11.80
Best LD3
9.96
3.63
2.63
2.27
Best LD3
17.32
13.07
12.40
11.83
2.70
2.29
2.18
S4S1
16.05
13.26
11.17
10.83
GS (Ours)
5.92
2.87
2.33
2.25
GS (Ours)
11.97
GAS (Ours)
4.48
2.66
2.29
2.31
GAS (Ours)
14.71
11.91
11.73
11,36
Teacher
2.16
Teacher
14.10
12.08
11,80
11.48
4.2 ABLATION STUDY
Coefficients parametrization First, we demonstrate significant impact of solver parameterization on training efficiency. Specifically, we show the difference between our parameterization, that represents coefficients of fixed theoretical guidance and explicitly trained additive corrections,
8

---

Published as a conference paper at ICLR 2026
Table 3:
We compare our parametrization with S4S variant on CIFARIO and FFHQ datasets in terms of FID and LPIPS scores. Both setups use batch size of 24, while training dataset consists of 49k samples. Teacher dataset has FID score of 2.03 and 2.60 for CIFARIO and FFHQ datasets respectively.
FFHQ Generalized
0.25
0.20
545
ME-0
LPP5
545 MT
0.10
NFE=4
NFE=6
NFE-8
NFE=10
0.05
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
0.00
CIFARI0
0
10000
Training iteration
20000
30000
40000
S4S
31.44
0.273
2.93
0.073
2.87
0.072
2.26
0.027
Our
4.39
0.116
2,51
0.046
2.21
0.017
2,15
0.010
Figure 3: LPIPS evaluation loss for
FFHQ
training iterations comparing S4S and our parametrization. Our method re- sults in more stable training process.
S4S
24.24
0.175
11.08
0.117
7.76
0.098
3.97
0.045
10.79
0.116
4.40
0.046
2.97
0.016
2.70
0.005
and the parameterization from another high-quality method S4S (Frankel al. 2025). We ablate the theoretical guidance in Appendix [B.] and demonstrate that it yields a substantial improvement in FID. For the purpose of fair comparison with S4S, we implemented LMS + PC S4S solver type removing a constraint on the solver order. This guarantees that Generalized Solver and S4S have the same number of trainable parameters.
In Table 3 we demonstrate our parameterization's 's superior performance on different datasets and NFE. Our results are consistent with the training issue reported in (Frankel et al., 2025). Figure 3 represents the dynamics of LPIPS loss on the evaluation dataset in different training iterations of the experiment. Our parametrization shows a more efficient training process, faster convergence and more stable training behavior. We also compare GS with LMS and LMS+PC under identical configurations in Appendix B..9
We observe that training with our parametrization for GS and GAS is stable, demonstrating an improvement in FID throughout the training process. More details are provided in Appendix B.3
Adversarial training Addition of the adversarial training is crucial part of our contribution, because significantly improves the image generation quality as seen in Tables 2a 2b It is crucial for low NFE setups because teacher image can be too difficult for the student replicate, therefore smaller values of the regression loss (LPIPS or La for pixel and latent models respectively) do not always correlate with smaller FID scores (as be seen in Table 4 and occasionally result in visible artifacts. Examples of such behavior are presented in Figure 4 Adding ad adversarial loss makes the student's generation closer to teacher's distribution and thus removes appearing artifacts and makes generation more realistic, in spite of occasionally resulting in bigger LPIPS or L1 losses. Additionally, we discuss three aspects of adversarial training in the appendices: the influence of loss selection in Appendix B.2 the sensitivity to the adversarial loss weight in Appendix B.5 and the impact of the training itself on mode collapse in Appendix B.6
Table 4: Results of 10k train- ing iterations calculated on 1000 validation samples.
vil Adv
wel ndvy
FID
Ldistill
FFHQ
Teaaher
GS
10.70
0.116
GAS
7.86
0.127
Figure 4: Incorporating an adversarial loss into the training process enhances generation quality, reducing occurring image artifacts in low NFE regimes. In this setup, the teacher model uses UniPC (3M) solver with NFE=10, while the student models operate with a reduced NFE=4.
LSUN
GS
9.64
0.172
GAS
7.54
0.174

---

Published as a conference paper at ICLR 2026
4.3 METHOD EFFICIENCY
We next show that GAS is efficient in terms of dataset size and training time.
Dataset size By the dataset we mean the set of samples generated from the teacher model to use in the training process. To this end, we measure method's performance on the "full" dataset scenario with 49000 samples and find the smaller dataset size that demonstrates equivalent results. First, we observe that the dataset size of 1400 is enough for training GS without adversarial loss. However, the solver's optimization problem becomes more challenging in low-NFE scenarios with adversarial loss. Here, we expand the dataset from 1400 samples to 5000 and obtain results indistinguishable from the full-dataset scenario in all datasets and settings. Additional information is provided in Appendix C.I
Performance Without adversarial training, GS converges within 1-2.5 hours depending the dataset, which is comparable to the most relevant baselines LD3 and S4S. In case of GAS, training time increases to 2-9 hours, which is larger, but still requires similar order. We refer the reader the Appendix C.2 for the exact comparison of metrics depending on training time and Appendix C.3 for peak-memory usage in the backward pass.
DISCUSSION
In this paper, propose Generalized Adversarial Solver, the novel parameterization and training algorithm for automatic gradient-basee solver optimization. The main novelty is additive theoretical guidance of solver coefficients and combination of distillation loss with adversarial training. We establish that the introduced Generalized Solver parameterization significantly outperforms existing parameterizations. We show that adding the adversarial loss significantly boosts method's perfor- mance and allows tackling the image artifacts present in simple solver distillation. We extensively compare our method with other solver/timestep training approaches and demonstrate its superior performance 6 datasets, ranging from 32 x 32 pixel-space CIFAR1O 10 to 256 x 256 latent-space ImageNet and 512 x 512 MS-COCO with Stable Diffusion.
Linitations Our method relies performing backpropagation through the whole solver inference, which may face scalability issues when applied to larger image sizes and bigger models. We explore the generalizability of our method between different datasets in Section B.4 However, potential concern remains as to whether GS/GAS requires separate training for each preferred inference NFE, We leave the development of lightweight modifications our method for future work.
The weights of the pretrained diffusion model remain frozen during solver training. We compare GS and GAS with distillation-basee methods and show Appendix [B.7 that our approach better preserves the generative capabilities of the original model. However, due to the limited number of trainable parameters at NFE=1 and 2, our method yields lower quality these specific settings; perfomance GS and GAS at low NFE is provided in Appendix B.8
REPRODUCIBILITY STATEMENT
To ensure the clarity and reproducibility of our work, we provide excessive description of all parts of our method. Appendix [D the pseudocode of our algorithm, exactly matching the way appears in implementation; configurations and hyperparameters of all "teacher" generations and "student" training processes, including batch sizes, optimizer choice and other tine-grained details; and expressions for commonly used timestep schedules mentioned in the paper.
Furthermore, our experiments are built upon publicly available datasets (e.g., CIFARIO, FFHQ) and pre-trained model checkpoints to ensure our experimental setups accessible and verifiable.
ACKNOWLEDGMENTS
Aleksandr Oganov wishes to express sincere gratitude to his alma mater, Lomonosov Moscow State University, for providing the foundational education and stimulating environment that made this research possible. Mishan Aliev thanks Yandex Education for supporting him during this research.
10

---

Published as a conference paper at ICLR 2026
We thank Dmitry Baranchuk for fruitful discussions and valuable advice on knowledge distillation techniques. This research was supported in part through computational resources of HPC facilities at HSE University. A special thanks to Maxim Kodryan - his mere existence was contribution enough. The work was supported by the grant for research centers in the field of Al provided by the Ministry of Economic Development of the Russian Federation in accordance with the agreement 000000C31 3925P4E0002 and the agreement with HSE University 1=139-1--2055-- thankful for the ICLR reviewers for their detailed suggestions that led significant improvement of this paper.
We are
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
Hicham Badri and Appu Shaji. Half-quadratic quantization of large machine learning models, November 2023.
www.aagg.ghhhuue www.tutthhhg.. bloq//
Andreas Blattmann, Tim Dockhor,, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231 151127,, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, al. Video generation models as world simulators. 2024. URL https://opeaii
www.eeeceiddd--eeeeeetnnnoo-m-ee wawoddel--as-ooldddssmmaaatooo 2024.
Thibault Castells, Hyoung-Kye Song, Bo-Kyeong Kim, and Shinkook Choi. Ld-pruner: Efficient pruming of latent diffusion models using task-agnostic insights, 2024. URL https: prctps://arxx. /arxiv
org/abs/22 2404. 11936
Defang Chen, Zhenyu Zhou, Can Wang, Chunhua Shen, and Siwei Lyu. On the trajectory regularity of ode-based diffusion sampling. arXiv preprint arXiv:2405. //36, 2024a.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp-
7310-7320, 2024b.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8188-8183 8197, 2020.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprini urXiv:2403.03206 2024.
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models, 2023. URL hittps://arxiv org. wttg//bs/////ooo 2305.1300000 10924
Eric Frankel, Sitan Chen, Jerry Li, Pang Wei Koh, Lillian J Ratliff, and Sewoong Oh. S4s: Solving for a diffusion model solver. arXiv preprint 2502.2500..74233, 2025.
Ian J Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Joshua M Susskind. Boot: Data-free distil- lation of denoising diffusion models with bootstrapping. In ICML 2023 Workshop on Structured Probabilistic Inference ({&) Generative Modeling, 2023.

---

Published as a conference paper at ICLR 2026
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang. Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10696-10706, 2022.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in newral information processing systems, 30, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv: 0rXiv:1503.02530 2015.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12599 2022.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 15684006851,, 2020.
Marlis Hochbruck and Alexander Ostermann. Exponential integrators. Acta Numerica, 19:209-20 286. 2010.
Nick Huang, Aaron Gokaslan, Volodymyr Kuleshov, and James Tompkin. The gan is dead; long live the gan! a modern gan baseline. Advances in Neural Information Processing Systems, 37: 44177-44215 2024.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Alexia Jolicoeur-Martinenn The relativistic discriminator: a key element missing from standard gan. arXiv preprint arXiv: 1807000/07700744 2018
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
Dongjun Kim, Chieh-Hsin Lai, Wei- Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Ue- saka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint 11X0v:2310.02279, 2023.
Sanghwan Kim, Hao Tang, and Fisher Yu. Distilling ode solvers of diffusion models into smaller steps. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9010-9111111 9419, 2024.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Christian Ledig, Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta. Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6681-4690 2017.
Jac Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint 1alp::705.0299. 2017.
2017
Tsung- Yi Lin, Michael Maire, Serge Belongie. Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
www.:xiwivvovvroooooo abs 1405 0312
12

---

Published as a conference paper at ICLR 2026
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022a. URL https //openreview net Eorum?id=P p LKWVdddykKKKWYYYY
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0.000 2022b.
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step enough for high-quality diffusion-based text-to-image generation. In The Twelfth International Conference on Learning Representations, 2023.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX0v:221.001095, 2022b.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- instruct: A universal approach for transferring knowledge from pre-traing pre-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15762-15772, 2024.
Xudong Mao, Qing Li. Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2794-2802, 2017.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. International conference on machine learning, pp. 7176-7185. PMLR, 2020.
Thuan Hoang Nguyen and Anh Tran. Swiftbrush: One-step text-to-image diffusion model with variational score distillation. arXiv preprint arXiv:2312.052333 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112..00741 2021.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: using variational divergence minimization. Advances neural information processing systems, 29, 2016.
Training generative neural samplers
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint 2rXiv:2209.14988 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2rXiv:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 11::211-252. 2015.
Amirmojtaba Sabour, Sanja Fidler, and Karsten Kreis. Align your steps: Optimizing sampling schedules in diffusion models. arXiv preprint 2xXx::2404114500 2024.
13

---

Published as a conference paper at ICLR 2026
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, (1.36479-364999 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022.
Tim Salimans, Thomas Mensink, Jonathan Heek, and Emiel Hoogeboom, Multistep distillation of diffusion models via moment matching, arXiv preprint arXiv:2406.04103 2024.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311. 2023.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-11, 2024.
Jascha Dich-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. pp. 2256-2265. pmlr, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1rXiv:2011.13455: 2020b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint 236,0, 2023.
Ruoyu Sun, Tiantian Fang, and Alexander Schwing. Towards a better global loss landscape of gans. Advances in Neural Information Processing Systems, 101 10186-0011,,,, 2020.
Vinh Tong, Trung-Dung Hoang, Anji Liu, Guy Van den Broeck, and Mathias Niepert. Learning to discretize denoising diffusion odes. arXiv preprint arXiv:2405. 2024.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Compu- fation, 23:1661-1674, 2011. URL https: /tpi. www.seeeaniis.m www.hoittccccooooooo ar. wrrg/orrporo......ooooo
3560643
Fu-Yun Yun Wang, Ling Yang, Zhaoyang Huang, Mengdi Wang, and Hongsheng Li. Rectified diffusion: Straightness is not your need in rectified flow. arXiv preprint arXiv:2410.07303, 2024.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36:8406-8441, 2023.
Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In International Conference Learning Representations, 2021.
Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang. Sam Tsai, Jonas Kohler, al. Cache me if you can: Accelerating diffusion models through block caching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2211-6220. 2024.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.0088 2021.
Shuchen Xue, Zhaoqiang Liu, Fei Chen. Shifeng Zhang, Tianyang Hu, Enze Xie, and Zhenguo Li. Accelerating diffusion sampling with optimized time steps. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8292-8301, 2024.
14

---

Published as a conference paper at ICLR 2026
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311.18828, 2023.
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. arXiv preprint arXiv:2405.14800 2024.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint IrXiv:1506. 03365, 2015.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902, 2022.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: unified predictor- corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statisties. Advances in Neural Information Processing Systems, 36: 55502-55542, 2023.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, march 2024.
URI http::/ggtthu. com/hpcaaircch(ppen-Sor 13):4, 2024.
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang. Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference Machine Learning, 2024a.
Zhenyu Zhou, Defang Chen, Can Wang, and Chun Chen. Fast ode-based sampling for diffusion models in around 5 steps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7777-7786, 2024b.
15

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Among many inference-time acceleration algorithms, solver-based methods treat diffusion models as ODEs with a (partially) black-box velocity function. Specifically, PNDM (Liu et al., 2022a) and iPNDM (Zhang & Chen (022) apply the linear multistep method to the corresponding PF-
ODE. IPM-Solver (Lu et al. 2022a), DEIS (hang & Chen 2022) use the variation of constants
(Equation 4 and approximate the underlying integral. DPM-Solver++ (Lu et al. 2022b) extends this idea to the multi-step version, and UniPC (Zhao et al. 2024) modifies with the predictor-correctt
framework. Besides the solver distillation loss, introduced for optimizing the timesteps in LD3 (Tong et al. 2025). many automatic solver selection methods were proposed. DDSS (Watson et al. 2021 directly
2024) and used for optimizing both timesteps and solver coefficients in S4S (Frankel et al.
optimizes generation quality of the solver. AYS (Sabour et al. 2024) optimizes timesteps to minimize the KL divergence between the backward SDE and the discretization. GITS (Chen al. 2024a)
choose the timesteps by utilizing trajectory structure of the PF-ODE and DMN (Xue et al. 2024 allows for the fast model-free choice of parameters via optimizing an upper-bound bound on the solution error. Some approaches manipulate diffusion-specific properties and utilize redundancies in their computations. Namely, DeepCache (Ma 2004)) and CacheMe (Wimbauer et al. (2044) propose to perform block layer caching and reuse activations from the previous timesteps. The other directions of acceleration include quantization (Gu et al. 2022 Badri
Shaji 2023) and pruning (Fang et al.,
2023 Castells et al. 2024).
In contrast, diffusion distillation techniques aim at compressing a pre-defined diffusion model by training a few-step student. Several methods learn to mimic solution of the PF-ODE. This includes optimizing the regression loss between the outputs (Salimans & Ho 2022) or learning the integrator
between arbitrary timesteps (Gu et al. 2023 Song al.) [023] Kim et al.] 2023). ). Others
diffusion models as a training signal that assesses likelihood of the generated images.
commonly
formalized as optimizing the Integrated KI divergence (Luo et al.] 2024 Yin al. 2023 2024
Nguyen & Tran 2023) by training an additional "fake" diffusion model on the generator's s output distribution. Other methods consider matching scores (Zhou et al. 2024a) or moments (Salimans it al.) (024) of the corresponding distributions. Many distillation methods enhance student generation quality by adding the adversarial training (Kim et al.) 2023] Yin et al. 2024). including discriminator
loss on detector (Sauer et al. 2023) or teacher features (Sauer et al. 2024).
B ADDITIONAL EXPERIMENTS
B.1 THEORETICAL GUIDANCE FOR THE COFFFICIENTS
The integration of theoretical guidance for the solver coefficients is important for the effective training of GS. It provides strong inductive bias by embedding knowledge of the theoretically optimal coefficients directly into GS. Thus, in the optimization process, the trained coefficients do not have to learn complex theoretical dependencies, since they are already embedded in the theoretical coefficients.
To empirically validate the contribution, we conducted the ablation by training a version of GS on the FFHQ dataset both with and without this guidance. In both configurations we use hyperparameters from D.4.I the coefficients were initialized to be equivalent to those of DPM-Solver++(3MI) The results, presented in Table demonstrate that the inclusion of theoretical guidance yields substantial improvement in FID. Furthermore, the flexible signature of GS enables the generalization of various modern theoretical solvers, opening up new avenues for research into different forms of theoretical guidance.
B.2 ADVERSARIAL LOSS
To better understand the impact of adversarial loss, we compared different GAN losses on ImageNet at NFE=4. We compared the standard GAS, which uses the relativistic loss from Equation [0] with one trained using the traditional loss from Equation 9 In both configurations, we used discriminator gradient penalties and the same hyperparameters from Appendix D.4.2
16

---

Published as a conference paper at ICLR 2026
Table GS with and without theoretical guidance on FFHQ dataset in terms of FID. Both setups use same hyperparameters and the coefficients were initialized to be equivalent to those of DPM--olver+++(3M))
Table 6: GAS with traditional and rela- tivistic GAN losses on ImageNet dataset at NFE=4 in terms of FID.
Parameterization
NFE=1
NFF=6
NFF=8
NFF=10
GS
GAS (traditiona)
GAS (relativistic)
w/ theory
10.70
4.49
2.96
2.67
7.87
6.49
5.38
w/o theory 15.23
10.53
5.50
4.69
From the comparison results presented Table we conclude that employing a GAN loss enhances the final output quality. GAS demonstrates strong performance with both traditional and relativistic losses. Although the relativistic loss is optional, it leads to a superior model.
B.3 FID PROGRESSION DURING TRAINING
To better understand the training process, we visualize the dynamics of the FID score during the training process.
When comparing the GS and GAS FID scores for FFHQ, visualized in Figure Sa we observe that incorporating the adversarial objective requires more training iterations for our method to converge. However, it is more important that, as previously reported Table 2a it achieves significantly lower FID score, allowing for better trade-off between generation quality and a slight increase in training time.
Figure 5b demonstrates that although GAS achieves excellent FID scores after 30k iterations, it could potentially yield even better results with further training. This is suggested by the continuing decrease in the FID score for NFE of 4 and 5 with each additional training iteration. Scenarios involving larger number of NFE for model inference do not display this pattern, since they comprise a bigger student's 'ss capacity and lead to casier optimization task and earlier convergence.
FFHQ N
155//4/bedroomm-246
Imagenet-256
G3
0.2
GAS
8:02
5.50
7.05
5.25
00
30
0.02
25
057
4.00
4.2
2000 .002
1002
10000
5000
20000
15000
20000
25000
33000
3000
10000
15002
20020
25300
30000
Training iteration
Taring interatinn
Baring interatiro
(a) FID values during training for the FFHQ dataset, using 4 NFE with the GS and GAS. We evaluate FID every 500 training iterations, computing based on 5000 generated samples.
(b) GAS FID training dynamics for latent space datasets for several NFE scenarios. We generate 50k images for each of 5000, 10k, 20k and iterations checkpoints, and evaluate FID scores based on those datasets.
Figure 5: FID score for several checkpoints during training of GS and GAS for both pixel (FFHQ) and latent space (LSUN, ImageNet) datasets.
B.4 GENERALIZATION ACROSS DATASETS
Regarding generalization across datasets with significantly different dimensionalities (e.g., CIFAR vs. COCO), the optimal schedule for smaller resolution may not be optimal for higher resolutions due to simpler denoising tasks at equivalent noise levels (larger images have greater correlation among nearby pixels). To further demonstrate the method's generalization results, tested solver transfer between closely related diffusion models (FFHQ and AFHQv2), demonstrating practical generalizability. We thus illustrate its generalization in Table 7
17

---

Published as a conference paper at ICLR 2026
Table 7: We evaluate FID score comparison of GS and GAS trained on dataset and applied on another against DPM-Solvert+, LD3 and S4S. We use the GS and GAS checkpoints as in Table 2a
(a) Solvers GS, GAS, trained on FFHQ and applied on AFHQv2 (denoted as GS' and GAS') consistently outperform baseline methods.
(b) Solvers GS, GAS, trained on AFHQv2 and applied on FFHQ (denoted as GS' and GAS') consistently outperform baseline methods.
Method
NFE=
NFE
NFE-8
NFE-10
Method
NFE=4
NFE=6
NFE-8
NFE-10
+
27.82
10.72
4.28
3.19
DPM-Solver++
46.14
14.01
6.18
4.18
Best LD3
3.63
2.63
2.27
Best LD3
17.96
5,97
3.50
S4S Alt
6.52
2.70
2.29
2.18
S4S Alt
10.63
4.62
3.15
2.91
GS (Ours)
5.92
2,87
2.33
2.25
GS (Ours)
10.70
4.49
2.96
2.67
GAS (Ours)
4.48
2.66
2.29
2.31
GAS (Ours)
7.86
3.79
2.87
2.66
GS' (Ours)
3.01
2.41
2.29
GS' (Ours)
16.01
5,91
3.27
GAS' (Ours)
5.15
2.81
2.44
2.32
GAS' (Ours)
9.39
4.21
2.92
2.72
B.5 ADVERSARIAL LOSS WEIGHT
One of the few hyperparameters of GAS is the GAN-weight. Starting from the resolution of 64 x 64, the weight of the adversarial loss was fixed 1.0 for all datasets, Figure demonstrates that GAS insensitive to the GAN-weight selection and achieves similar FID with different weights. This shows that our method achieves strong results without the need for hyperparameter tuning.
FFHQ NFE 4
10.5
10.0
FID
9.5
8.5
8.0
5
10
Eddy weight
Figure 6: FID values for FFHQ dataset with 4 NFE for different adversarial loss weights. The metric remains stable even at large weight values. Setting Ladv 0 for GAS results in absence of adversrrial training, thus is a GS setting.
B.6 MODE COLLAPSE
To demonstrate that GAS maintains diversity, computed precision, recall, density, coverage from Naeem et al. (220)) for GS and GAS, comparing their statistics to those of the teacher model. Evaluation results obtained by comparing 50000 generated samples 50000 teacher images using T4096
et al. (2020). Section 4.2). We report results on the ImageNet dataset with NFE=4 Table g We choose this setup because the incorporation of the adversarial loss was significant in this experiment and could raise the greatest concerns regarding mode-collapse. As shown in Table X
GAS does not suffer from the aforementioned problem and even increases mode coverage and recall compared to GS, which is trained without adversarial loss. Additionally, provide random samples in Appendix
Table 8: Mode collapse ablation of the GAS on ImageNet dataset.
Table 9: Comparison of generative properties of models on CIFARIO dataset.
Method
FID
precisson
recall
density
coverage
Method
CD, NPC-1
CD. NFT:44
GS. NPF-4
GAS, NPP-4
Our teacher
68
7.87
0.90
066
1.18
0.90
Coverage
0.942
0.988
0.963
0.961
0.971
GAS
5.38
0.90
0.76
1.20
0.97
18

---

Published as a conference paper at ICLR 2026
B.7 COMPARISON WITH CONSISTENCY DISTILLATION
We compare GS and GAS with distillation-based methods in terms of their ability to preserve the generative properties of the diffusion model. To evaluate diversity, compare the coverage from Naeem et al. (2020) and FID of our methods (GS and GAS) against the official checkpoint of CD CDnSog al. (2023) on CIFARIO. We assess generation quality against a teacher solver, which achieves FID=2.03 and coverage=0.971. Throughout, when working with CD, we used the official implementation and ternary search for 4 al. (203), Section 3). Coverage was measure between 10000 generated images and the CIFARIO test set using T4096.
The comparison results for FID and coverage are presented in Tablegand show that GS and GAS have higher coverage than CD. In addition, CD demonstrates low coverage compared to the teacher solver, which indicates a deterioration in the generative properties of the diffusion model. GS and GAS, without changing these parameters, provide more flexible and property- preservy- acceleration method. We compare the efficiency of GS and GAS with that iiiillation-baseee approaches in Appendix C.2
B.8 PERFORMANCE LOW NFE
We evaluated the performance of GS and GAS at low NFE. We trained GS and GAS on the FFHQ dataset with NFE=1 and NFE-2. For training, we used the hyperparameters from Appendix 1 For comparison, we used DPM-Solver with order set to NFE when NFE < 4, and order=3 when NFE=4. Table [T shows that GS and GAS improve upon DPM-Solver, but they do not perform well at low NFE as the FID scores are too high.
B.9 COFFFICIENTS PARAMETRIZATION
In addition to the coefficient parameterization ablation in Section 4.2
we compared GS with the $4S LMS and LMS+PC parameterizations under the same hyperparameters and within our codebase (S4S does not provide an official implementation). In T we compare GS with LMS+PC when all methods
trained convergence. Here, we further show that GS outperforms both LMS and LMS+PC on FFHQ under identical training settings from Appendix [.4.]] We initialize LMS and GS so that the corresponding solvers are initially equivalent DPM-Solver++(3M) Table I0 shows that for all NFE values, GS achieves better generation quality.
Table 11: Comparison of GS and GAS with DPM-Solver++ on FFHQ dataset in terms of FID. We denoted DPM as DPM--
Table 10: Comparison of S4S parameterizations with GS on FFHQ dataset in terms of FID. We use same hyperpa- rameters, teachers and codebase.
Parameterization
NFF=4
NFF=6
NFF=8
NFF=10
NFE=1
NFF=2
NFF=4
S4S LMS
17.05
5.93
4.37
3.96
DPM
314.95
134.36
46.14
$4S LMS + PC
45.51
24.04
7.95
4.01
GS
147.54
55.98
10.70
GS
10.70
4.49
2.96
2.67
GAS
193.22
60.69
7.86
C EFFICIENCY OF THE METHOD
C.I TRAINING DATASET SIZE
We conduct experiments to assess the efficiency of the proposed methods with respect to the size of the training dataset. We examine several variations of sizes: 49000 bascline, 5000 and 1400 as the more lightweight alternatives. For GS, we observe that taking 1400 images and performing 10000 training iterations is sufficient for our method converge, regardless of NFE. We note that reaches equivalent or better FID scores compared to a bigger training dataset (see Table (2a).
The same pattern occurs with GAS CIFARIO. The dataset of 1400 images is optimal for its training. However, starting from the higher-dimensione. FFHQ dataset, we observe the typical challenges of adversarial training. As the discriminator used in GAS is trained simultaneously with
19

---

Published as a conference paper at ICLR 2026
the other parameters of the solver, it tends to overfit and demands larger dataset size to alleviate this problem.
Adversarial training has demonstrated its effecctiveness, especially in scenarios with smaller inference steps. We thus illustrate its performance in Table on NFE = 4 and NFE 6. It shows that the training dataset size of 5000 is sufficient for matching performance of the model trained on 49000.
Table 12: Comparison of different dataset sizes with several NFE, where N indicates the number of
samples in the training dataset. In Table T2a the FID score is calculated after 10k and 20k training iterations to show the early convergence of the GS method. Table 12b presents the results of GAS evaluation after 10k iterations of training.
(a) Generalized solver
(b) Generalized Adversarial solver
NFE=4
NFE=10
NFE==
NFE=6
N
10k
20k
10k
20k
1400
3.98
2.44
CIFAR10
1400
4.35
4.35
2.14
2.15
49000
3.98
2.48
CIFARIO
49000
4.39
4.39
2.15
2.15
1400
9.44
4.48
1400
10.70
10.72
2.71
2.71
FFHQ
5000
7.83
3.79
FFHQ
49000
10.79
10.82
2.70
2.71
49000
7.93
3.76
C.2 TRAINING TIME
We further investigate GS/GAS training dynamics by estimating their convergence time and comparing their computational efficiency with other methods.
In Table 13a we demonstrate the training time of Progressive Distillation (PD, (Salimans & Ho
2022) and Consistency Distillation (CD, (Song al. 2023)). Those methods focus training new generator model that can sample images in few-NFF manner. Both require days of training time and are computationally demanding.
We also compare our methods with several approaches that involve training certain parameters of solvers. In pixel space GS requires less than an hour of training time on CIFAR10, which comparable to LD3, $4S and S4S-Alt. Notably, it achieves FID of 2.44 with NFE 6, while S4S-Alt results in FID score 2.52 with NFE = and equivalent training time. Adversarial loss extends training time to up to 2 hours, however, as we report in Table 2a achieves superior results in terms of FID score.
In the latent diffusion setting, compare method with LD3, which reports convergence within an hour training time. We observe that GS and GAS require up to hours; however, this still within the same order (for more details, see Table 13b).
In Table 14 we also provide more details about training time of our methods for both pixel and latent space models.
C.3 MEMORY USAGE
We are investigating the peak-memory GS/GAS required for training iteration depending on NFE.
In Table 15 we demonstrate the peak-memory usage GS/GAS compared LD3. When measuring the memory, used the config further report in Appendix D GS requires the same amount of peak-memory allocated as LD3.
Incorporation of the discriminator loss into the training process of GAS only requires additional less than gigabyte of memory usage, which is a minor overhead, especially considering its efficiency in terms of the final generation quality. This overhead is limited to training inference time, GAS and GS sample at the same speed. Additionally, storing prior states does not incur additional overhead for peak-memory usage.
20

---

Published as a conference paper at ICLR 2026
Table 13: Comparison of different training- boining-bbssdd methods in terms of computational effectiveness across both pixel and latent space selected dataset.
(a) CIFAR10
(b) Imagenet-256
Method
NFE
FID
Time
GPU Type
Method
NFE
FID
Time
GPU Type
CD
2
2.93
8 days
A100
4
9.19
PD
8
2.47
8 days
TPU
5
5,03
LD3
hour
A 100
6
4,46
S4S-Alt
7
2.52
hour
A100
7
4.32
S4S
2.18
hour
A100
2.32
hour
A100
4
7.97
1.5 hours
5
4,94
< 2 hours
GS
6
2.44
hour
H100
GS
6
4.29
< 2 hours
H100
10
2.14
hour
H100
7
4.16
2.5 hours
GAS
4
3.98
hours
H100
GAS
4
6.06
3 hours
H100
Table 14: Approximate training time (in minutes) for 10k iterations scenarios for GS and GAS in both pixel and latent space. For MS-COCO we use 1K iterations scenarios. All the numbers reported computed using one H100 GPU.
(a) Pixel space models
(b) Latent space models
NFE=1
NFE=6
NFE=8
NFE=10
NFE=4
NFE=5
NFE=6
NFE=?
CIFARIO
30m
40m
50m
60m
LSUN
35m
45m
50m
60m
GS
FPHQ
40m
60m
80m
95m
GS
ImageNet
75m
95m
115m
135m
AFHQv2
40m
60m
80m
95m
COCO
70m
CIFARIO
100m
115m
130m
LSUN
125m
140m
150m
165m
GAS
FFHQ
160m
185m
210m
240m
GAS
ImageNet
185m
210m
245m
270m
AFHQv2
160m
185m
210m
240m
MS COCO
60m
75m
90m
105m
Table 15: Peak-memory usage (in gigabyte) for training iteration for GS and GAS in CIFARIO and Imagenet-256. We use LD3 in our implementation. The official implementation uses LPIPS, rather than L1 distance in latent space do, which leads to the use of VAE decoder at each step and incurs additional memory usage.
(a) CIFARIO
Imagenet-256
NFE=4
NFF=6
NFE=8
NFF=10
NEE=4
NFE=5
NFE=6
NFE=7
GS
17GB
23GB
28GB
34GB
GS
37GB
45GB
54GB
62GB
GAS
19GB
25GB
30GB
35GB
GAS
41GB
49GB
57GB
66GB
LD3
17GB
23GB
28GB
34GB
LDD
37GB
45GB
54GB
62GB
21

---

Published as a conference paper at ICLR 2026
C.4 INFERENCE TIME
Inference process of our method requires additional operations performed with all prior states. However, they incomparably computationally simpler than one step of diffusion model (function evaluation). Thus, the wall-clock time of inference for GS is comparable to the solver baselines, which we show Table
Table 16: Inference time in minutes for ImageNet dataset. We obtain the comparison by generating 1,024 images with batch 64 utilizzig single H100 GPU. GAS differs from GS only in the training process; their inference times are identical.
Method
NFE=4
NFE=5
NFE=6
NFE=7
UniPC(3M)
0.36m
0.46m
0,55m
0.64m
GS (Ours)
0.36m
0.45m
0.55m
0.64m
This pattern does not depend on the model and dataset choice; therefore, our method does not introduce any inference time overhead both pixel, latent text-to-image diffusion models.
D EXPERIMENTAL DETAILS
D.1 BASELINE DISCRETIZATION HEURISTICS
In this section, we provide the reader with the common timestep schedules, used in the paper. Polynomial discretization (time-quadratic, time-uniform) defines the timestep schedule via polynomial function of the uniform sequence. Specifically, it defines
t,
T
teps) + teps: i 0.1,..,,NN.
(23)
Here p is often to 1 or (Song al. 2020b Ho et al. 2020 Song al. 2020a) which corresponds to time quadratic and time uniform discretization.
Time loR schedule builds top of the signal-to-noise ratio a² / Specifically, log-SNR the transformation X₂ = -loooor ( and defines
x(4) N
N (AT >eps) + Aeps: i= 0, I.....
(24)
This schedule offers high generation quality with different versions of the DPM-Solver (Lu et al.
2022a b Zheng et al. 2023).
GITS schedule provides optimized sequence of noise levels for diffusion models, targeting very low NFE. Originally proposed in Chen et al. (2024a) for ODE-based diffusion processes
with trajectory regularity constraints. We use optimized timesteps in Stable Diffusion experiments from Tong et al. (2024). Concretely, the timestep schedules are:
NFE- 4
[1, 0.6837, 0.3673, 0.1176, 0.001];
NFE 5:
[1, 0.7669, 0.4839, 0.2341, 0.0676, 0.001]:
NFE 6:
[1, 0.7836, 0.5504, 0.3340, 0.1508, 0.0343, 0.001]:
NFE 7:
[1, 0.8502, 0.6004, 0.4006, 0.2175, 0.0843, 0.0176, 0.001];
NFE 8:
11 0.8502, 0.6504, 0.4672, 0.3007, 0.1675, 0.0676, 0.0176, 0.001].
D.2 TEACHER SOLVER
Data generation For a fair comparison, we follow Tong et al. (024) to generate the teacher dataset.
We choose UniPC with the parameters used in LD3. We utilize class condition of the ImageNet-255 teacher and generate the corresponding dataset with the classifier-free guidance scale of 2.0 and generate 50 images per each of the 1000 classes. We report details in Table 17
22

---

Published as a conference paper at ICLR 2026
Table 17: Detailed description of the UniPC solver parameters used for teacher dataset generation consisting of 50000 images for both pixel and latent space scenarios.
CIFARIO
FFHQ
AFHQv2
LSUN-Bedroom--25 -266
Imagenet-256
Order
3
3
3
3
NFE
20
20
20
20
10
Time schedule
logSNR
logSNR
logSNR
time-uniform
time-quadratic
B(h)
bh1
bhl
bhl
bh2
bh2
leps
le-4
le-4
le-4
le-3
le-3
FID
2.03
2.60
2.16
3.06
4.10
Stable Diffusion details Regarding text-to-image generation with Stable Diffusion, we observe that output image distributions of low-NFE students (NFE 3-5) differ significantly from those of high-NFE teacher (e.g., NFE = 10). Since such students have very few trainable parameters, direct distillation can be inefficient. The same pattern a fair comparison, we follow identical to the LD3 approach teacher generation protocol. We train student NFE n with the teacher at NFE n + This "one-plus" teacher minimizes the gap in noise dynamics and yields smoother, more reliable convergence.
found in [Tong et al. (2024). For such reason and
Moreover, in our experiments, we find that FID loses its correlation with perceived fidelity high NFE, so we treat improvements in that regime with particular caution. Recognizing this unreliability beyond NFE R 8 reinforces our choice of simpler teachers as the most robust path to high-quality samples. Further details on teacher parameters are provided in Table 18
Table 18: Detailed solver parameter settings for Sracher-generale dataset using 30000 MS-COCO prompts.
Student's NFE
NFE=4
NFE=5
NFE=6
NFE=7
Teacher's NFE
5
6
7
Solver
IPNDM(2M)
IPNDM(2M)
IPNDM(2M) GITS
IPNDM(2M)
Time schedule
GITS
GITS
GITS
FID
14.10
12.08
11.80
11.48
D.3
SOLVER COEFFICIENTS PARAMETERIZATION
The detailed description the Generalized Solver step provided in Algorithm T Specifically, when all parameters & are set to zero, the GS reduces exactly to DPM-Solvert+(3MM) (Lu et al. 2022b).
D.4 PRACTICAL IMPLEMENTATION DETAILS
We define W, H, and C as the width, height, and number of channels of image, respectively. Similarly, W", H', and C" represent the corresponding dimensions in the latent space for the Latent Diffusion model (Rombach al. 2022).
Optimizer and trainable parameters We update three primary parameter sets during training: 0 defines the timestep schedule, defines the solver coefficients and & acts as a correction the timesteps that we evaluate the pre-trained model on. We use one optimizer for all parameter groups. We use time-uniform schedule for the initialization of parameters We initialize & and ē (d)) aj (Ø) with zeros. We use the EMA version of the model parameters for evaluation and update the EMA weights after each training iteration.
Evaluation
using the FID score with 50 000 randomly generated samples. For ImageNet, we generate an equal number of samples for each class to ensure balanced FID evaluation. We EMA weights for evaluations. We calculate FID using reference
We evaluate models (Table 2a 2b
23

---

Published as a conference paper at ICLR 2026
Algorithm 1 Generalized solver (GS) with theoretical guidance from DPM-Solver+++(3M) Denote h4 xg X for = I,..., N.
1: 1/1 t ch -1
2: Xx+1 + an,n (tmmmn))))))) + an,n( x, - [[aents @1 + En,o( 0(x, fy En)
3: if tten
4: To + hn
ha-1
5: D1. [o(xxs exnnly ln En) v(xn- 11 1:b,111,1- 1 + fn-1)]]
11 101 1n) D1o
6:
Xn+1 f- -xx++1
2
7:
else if then
8:
To: T1 f- ha-1
hn-2
Am
An
9:
1/ f F1
+
h
4/3
22
h
2
D1g + (v(
xxx in + [n) v(xn-1111 t tn + +
12:
D1 t- 1 [x
n-1 + En-1) v(xn 2, 1n En En-2)
D1 Din totri
D10 D11]
14: D2 + rotmi
D10
D11
Xn+1 X + M 02 + 1n
DD
[atnt] 03 cn 2,(( D2
16: end if
max(n- 1,0)
max(n-3,0)
17: X,+1 f X +
Σ
(o) Xj
+
Σ
(o)( v(x,, t2 +E))
=
statistics and code from Karras al. 2022). For MS-COCO (Table 2c we obtain the FID score on 30 000 images using the same validation captions and FID reference statistics as in LD3 TTng et al.
2024).
D.4.1 PIXEL SPACE DIFFUSION ON CIFAR1O, FFHQ, AND AFHQV2
Pre-trained diffusion model:
EDM (Krrras et 2022);
Teacher:
-UniPC solver, NFE 20, logSNR schedule;
Discriminator R3GAN (Huang et al. 2024):
Pre-trained CIFAR10 checkpoint for CIFARIO;
Pre-trained FFHQ-64 checkpoint for both FFHQ and AFHQv2;
Training in pixel space;
Image resolution:
W H 32, C 3 for CIFARIO;
W = 64, C = 3 for FFHQ and AFHQv2;
Training/validation dataset size:
-CIFAR10: 1400/1000 for GS and GAS;
-FFHQ and AFHQv2: 1400/1000 for GS; 5000/1000 for GAS;
Solver training:
Laistsul is LPIPS;
Ladv with weight 0.1 for CIFAR10 and weight 1.0 for FFHQ and AFHQv2:
EMA decay = 0.999;
Batch size = 24;
Adam optimizer, Ir - 0.001, betas (0.9, 0.999), weight decay 0.0;
Gradients clipped by the norm of 1.0:
24

---

Published as a conference paper at ICLR 2026
Discriminator training:
Batch size = 24;
-Adam optimizer, Ir 0.00001, betas (0.9, 0.999), weight decay 0.0:
Å and 22 in Equation 11 are equal 0.1;
Training duration:
10k iterations for GS/GAS;
D.4.2
LATENT SPACE DIFFUSION ON LSUN- BEDROOM AND IMAGENET
Pre-trained diffusion model:
LDM (Rombach et al. 2022
Teacher:
UniPC solver for both LSUN- Bedrooms and ImageNet;
-NFE = 20 and time-uniform schedule for LSUN;
NFE = 10 and time-quadratic schedule for ImageNet;
Discriminator R3GAN (Huang et al., 2024):
-FFHQ-64 architecture with random initialization;
-Triining in latent space;
Image resolution:
W HI 256, C 3;
W" H' 64, C' 3;
Guidance scale: 2.0 (for ImageNet);
Training/validation dataset size:
-1400 1100 for GS;
-5000/1000 for GAS;
Solver training:
Laistitl is L1 in latent space;
Ladv with weight = 1.0;
EMA decay 0.999;
Batch size 8;
Adam optimizer, Ir = 0.001, betas = (0.9, 0.999), weight decay = 0.0;
Gradients are clipped by the norm of 1.0;
Discriminator training:
Batch size = 8;
Adam optimizer, Ir = 0.00001, betas = (0.9, 0.999), weight decay = 0.0;
Å and X2 in Equation 11 are equal to 0.1;
Training duration:
30k iterations for GS/GAS;
D.4.3 TEXT-TO-IMAGE GENERATION WITH STABLE DIFFUSION
Pre-trained diffusion model:
Stable Diffusion v1.5 (Rombach al. 2022):
Gradient checkpointing at every UNet inference;
Teacher:
-NFE 1, where student NFE;
IPNDM(2M) solver with GITS;
Discriminator R3GAN (Huang al. 2024);
25

---

Published as a conference paper at ICLR 2026
FFHQ-G 64 architecture with random initialization;
First convolution layer modified to accept 4-channel latent inputs;
Training in latent space;
Image resolution:
W x // 512 x 512, C
-W x H' 64 x 64, C'
Guidance scale: 7.5;
Training/validation dataset size:
1400//28 for GS;
-5000/128 for GAS;
Solver training:
disttit LI in latent space;
adv with weight 1.0;
EMA decay = 0.999;
Batch size = 4;
Adam optimizer, Ir = 0.001, betas = (0.9, 0.999), weight decay 0.0;
Gradients clipped by the norm of 1.0;
.
Discriminator training:
Batch size 4;
Adam optimizer, Ir = 0.00001, betas (0.9, 0.999), weight decay = 0.0;
A and X2 in Equation II are equal to 0.1;
Training duration:
1k iterations for GS;
2k iterations for GAS;
E TIMESTEPS VISUALIZATION
To validate the adequacy of the learned decoupled timesteps, we provide plots showing both t and decoupled 1 timestep schedules for GS and GAS across several NFE settings on FFHQ and ImageNet (Figure 7
The decoupled timesteps should remain positive and monotonically decreasing, and observe that they consistently satisfy these common-senss characteristics without any explicit constraints during training. They never become negative and always decrease monotonically; moreover, they are typically slightly lower than the corresponding unconstrained timesteps.
In one experiment, the first timestep was slightly above 1, but this caused no issues: the denoising model uses continuous positional embeddings and is robust to small shifts, so such value is not out of distribution.
F ADDITIONAL SAMPLES
To further demonstrate the method's competitive results, we provide the reader with the additional samples of GS and GAS, compared to the teacher and the baseline UniPC with the same NFE. For all models/datasets except Stable Diffusion, we choose samples corresponding to random seeds (marked "random") ") and samples that are the most distinguishable between GS and GAS in terms of pixel-space L1 distance (marked as "sclected"). We choose the selected sample seeds NFE and report the corresponding samples for all NFE. We report the samples for FFHQ 15). LSUN Bedroom (Figures [6]
(Figures g 9 10 [T]. AFHQv2 (Figures
18
and ImageNet (Figures 20 21
23
Most random samples show only minor fine- fine-grained differences between GS and GAS (which is still important and has positive effect FID,, indicated in Table 2). At the same time, the selected
26

---

Published as a conference paper at ICLR 2026
Leamed timesteps for FFHQ dataset
NFE==4
NFE=6
NFE=8
NFE=10
2.0
0.81
Umalug
0.6
0.2
8-
E 0.0001
MINIC
0.8
Limokag
C45
0.0
MiIIC 3.0001
minE 8 0.0003
mint B 0.0001
MiIIC 8 0.0001
Step number
Stop num ber
Stop number
Stop Tumber
Learned tim limesteps for ImageNet eNet dataset
NFE=
3.0
0.6
025
0.4
mintc 0.0010
mantE 6 0.0010
minCE 0.0010
minic 8
0.0
1.0
685
0.4
minit 0.0010
mint+ 8 0.0010
0.001.
MiniC &
3
2 3
3
Stepnumber
Step number
Step number
Stop number
Figure 7: Visualization of final timestep schedules with and without additive corrections on FFHQ and ImageNet datasets. It is important to note that they never violate any of the common-sense characteristics expected of timestep schedules.
samples fully demonstrate the potential effect of the adversarial loss on the image quality. Most GAS samples at NFE = 4 demonstrate superior image quality compared to GS, while being farther from teacher. This further complements the results demonstrated in Figure 4 At the same time, could tell that the pictures enhanced by adversarial loss differ depending on NFE: pictures from the same random seeds become significantly closer to the teacher starting from NFE 6. This also indicates that the effect of the adversarial loss is the most prominent at low NFE, where is harder for the student to replicate teacher's performance.
Mode collapse It is also worth noting that incorporation of the adversarial loss into the training process does not lead to mode collapse - a common concern in such cases - as we explicitly address this issue using the relativistic GAN loss from Huang et al. (2024). The random samples reported in Figures g 25 show generation diversity, while low resulting FID values indicate both high quality of our images and the absence of mode collapse.
Stable Diffusion For the Stable Diffusion experiments, we generate images from the 250 MS- COCO-val prompts with both the official LD3 implementation and our GAS method, initializing
27

---

Published as a conference paper at ICLR 2026
both with identical random latent noise. From these outputs, select six images at random (marked "random") and six that best highlight the visual differences between GAS and LD3 (marked "selected").
Random prompts:
woman sitting bench and a woman standing waiting for the bus."
"jumbo jet sits the tarmac while another takes off"
"An old green car parked on the side of the street."
"A gas stove next to a stainless steel kitchen sink and countertop."
"A person walking through the rain with an umbrella."
Selected prompts:
"A man in wheelchair and another sitting on bench that is overlooking the water."
"A fireplace with a fire built in it."
"A half caten dessert cake sitting on cake plate."
"an airport with one plane flying away and the other sitting on the runway"
"A dirt bike rider doing a stunt jump air"
The resulting comparisons shown in Figures 25
28

---

Published as a conference paper at ICLR 2026
UniPC
GS
GAS
Teacher
UniPC
GS
GAS
Teacher
Selected
Selected
SAH
Random
Random
Figure 8: Comparison of GS and GAS with the teacher and UniPC on FFHQ with NFE
Figure 9: Comparison of GS and GAS with the teacher and UniPC on FFHQ with NFE = 6.
29

---

Published as a conference paper at ICLR 2026
UniPC
GS
GAS
Teacher
UniPC
GS
GAS
Teacher
Selected
Selected
Random
25
Random
Figure 10: Comparison of GS and GAS with the teacher and UniPC on FFHQ with NFE
Figure 11: Comparison of GS and GAS with the
teacher and UniPC on FFHQ with NFE = 10.
30

---

Published as a conference paper at ICLR 2026
UniPC
GS
GAS
Teacher
UniPC
GS
GAS
Teacher
Selected
Selected
Random
Random
Figure 12: Comparison of GS and GAS with the teacher and UniPC on AFHQy2 with NFE =
Figure 13: Comparison of GS and GAS with the
teacher and UniPC on AFHQy2 with NFE 6.
31

---

Published as a conference paper at ICLR 2026
UniPC
GS
GAS
Teacher
UniPC
GS
GAS
Teacher
Selected
Selected
Random
Random
Figure 14: Comparison of GS and GAS with the teacher and UniPC on AFHQv2 with NFE = 8.
Figure 15: Comparison of GS and GAS with the
teacher and UniPC on AFHQy2 with NFE 10.
32

---

Published as a conference paper at ICLR 2026
UniPC
GS
GAS
Teacher
UniPC
GS
GAS
Teacher
Selected
Selected
Random
Random
Figure 16: Comparison of GS and GAS with the teacher and UniPC on -eUrooodroom with NFE- 4.
Figure 17: Comparison of GS and GAS with the teacher and UniPC on LSUN- with NFE 5.
33

---

Published as a conference paper at ICLR 2026
UniPC
GS
GAS
Teacher
UniPC
GS
GAS
Teacher
Selected
Selected
Random
Random
Figure 18: Comparison of GS and GAS with the teacher and UniPC on -edroood with NFE- 6.
Figure 19: Comparison of GS and GAS with the teacher and UniPC on with NFE
34

---

Published as a conference paper at ICLR 2026
UniPC
GS
GAS
Teacher
UniPC
GS
GAS
Teacher
Selected
Selected
Random
Random
Figure 20: Comparison of GS and GAS with the teacher and UniPC on ImageNet with NFE
Figure 21: Comparison of GS and GAS with the teacher and UniPC on ImageNet with NFE = 5.
35

---

Published as a conference paper at ICLR 2026
UniPC
GS
GAS
Teacher
UniPC
GS
GAS
Teacher
Selected
Selected
Random
Random
Figure 22: Comparison of GS and GAS with the teacher and UniPC on ImageNet with NFE
Figure 23: Comparison of GS and GAS with the teacher and UniPC on ImageNet with NFE 7.
36

---

Published as a conference paper at ICLR 2026
GITS
LD3
GAS
GITS
LD3
GAS
Selected
Selected
muo
Random
Random
Figure 24: Comparison of GAS with LD3 and GITS on MS-COCO with NFE 5.
Figure 25: Comparison of GAS with LD3 and
GITS on MS-COCO with NFE
37


Reviews:
[{"review_id": "UnK1Y3CBLn", "reviewer": "Reviewer_JFWr", "summary": "This work studies the topic of distilling diffusion models into a few-step solver. This line of work focuses on the compute-efficient setting in which the parameters of the *solver*, not the score function/network itself, are learned.\n\nIt proposes a new efficient distilled solver, the generalized adversarial solver (GAS), which puts up state-of-the art results in the 4-10 NFE range on several datasets. \n\nGAS is a combination of two techniques\n- The generalized solver (GS): by parameterizing the next step $x_{n+1}$ as a linear combination of (1) the entire history of step values $\\sum_j^n a_{j, n} x_j$ and (2) the entire history of vector field evaluations $\\sum_j^n c_{j, n}v(x_j, t_j)$. The coefficients $\\{a_{j, n}, c_{j, n}\\}$  are parameterized as additive corrections to theoretically derived coefficients from past work. This contrasts to previous work, which learns only coefficients for the latter.\n- GAS is formulated by adding a relativistic GAN loss term to the standard distance loss (either LPIPS or L1) used to train GS.", "questions": "- In Table 2, you tagged S4S Alt as unfair due to a different teacher FID score. Where did you find the teacher FIDs for S4S? I did not see them in the S4S paper (but may be missing them).\n - Can you explain the advantage of this approach over PEFT-based distillation of a diffusion model? I would think PEFT was more tractable because there's no need to backpropagate through entire 4-10 step solves?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "I think this project has the ingredients to ultimately build a strong paper.\n- The research direction seems logical to me - training the solver might be more efficient than training the model itself.\n- Its two components, GS and the GAN loss, each provide an FID boost over past methods, with the GAN loss's boost being particularly impressive in the 4-6 NFE range. \n\t- The GAN loss in particular is a very natural import from other subfields of diffusion modelling.", "weaknesses": "However, to me the paper reads more like a technical report than a top conference paper. A couple of ideas are proposed and results are shown, but not much beyond that in terms of motivation or generalizable insight and analysis.\n-  Very little motivation is given for the way GS is designed. The motivation given (I believe L225 -> 229) is not honestly not very clear. Are you just looking for ways to add more parameters to the solver or is there something more to it? \n\t- As someone familiar with diffusion but without specific expertise in distilling diffusion solvers, I found the technical part, Section 3.1 tough to read, in large part because there is not much setup and motivation for the exposited ideas.\n\t- (However, I do think the motivation for the adversarial loss is clearer, as it is simpler and more obvious.)\n- Additionally, neither of the two components of the method are really probed or ablated to understand why they work so well. \n\t- For example, the decision to use additive vs. non-additive coefficients in GS could be ablated easily, but it is not. As could the decision to include the past history of $x_j$s (this is similar to but not the same as S4S). With little motivation and not much ablation, it's hard to glean generalizable insights from the work.\n\t- Additionally, the decision to use the relativistic GANs is not ablated over popular alternative GAN losses.\n- No code has been provided.\n\nIn summary, the two individual components of the method - the parameterization of the solver and the adversarial loss - are incremental, and little motivation, analysis, or generalizable insights are provided. Although I believe someone will find it useful, I do not believe it meets the bar for ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X614tJaNcG", "reviewer": "Reviewer_bLXP", "summary": "This paper introduces a novel approach to training solvers for fast sampling by combining a new parameterization with an adversarial loss. The proposed method enhances the alignment between the generated and teacher distributions, outperforming traditional teacher-matching losses.", "questions": "1. **Ensure Fairness in Training Time Comparisons (Table 7)**  \n   Training times should be reported using the same GPU model to ensure fairness, as differences in hardware can significantly affect latency measurements. Additionally, comparing results under the same number of function evaluations (NFE) would provide a more accurate measure of computational efficiency.  \n\n2. **Align Training Iterations Across Methods (Appendix D.4.1)**  \n   In Appendix D.4.1, GS and GAS appear to have been trained for different numbers of iterations (e.g., 1k for GS vs. 2k for GAS in the text-to-image experiment). This inconsistency complicates fair comparison. Both methods should be trained for the same number of iterations to enable a meaningful evaluation.  \n\n3. **Clarify the Choice of Initialization Solver**  \n   The paper initializes coefficients using the DPM solver [2], which belongs to the exponential integrator family which might affect the theoretical soundess of initializing some coefficients with it.  \n\n4. **Absence of Alternating Training Strategy and Its Rationale**  \n   Unlike S4S [1], the proposed method does not employ an alternating training strategy for the timestep scheduler and solver coefficients. It would be valuable for the authors to elaborate on why their approach doesn't require alternating strategy to stabilize training.  \n\n5. **Retrain S4S for Consistent Comparison**  \n   The S4S results presented are taken directly from the original paper, yet S4S performance is highly sensitive to the choice of teacher model. For example, in the LSUN-Bedroom 256 dataset, S4S underperforms relative to other solvers due to the poor teacher performance used in S4S in the original paper. Retraining S4S under the same setup and teacher model would ensure a fair and unbiased comparison with the proposed GAS method.\n\n---\n\n## References  \n[1] **S4S: Solving for a Diffusion Model Solver.**   \n[2] **DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps.**", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Effective Use of Adversarial Loss**  \n   The incorporation of an adversarial loss is both intuitive and impactful, substantially improving distribution alignment between the generated samples and the teacher model.  \n\n2. **Clarity and Organization**  \n   The paper is clearly written and well-structured, making it easy to follow the methodology and results.  \n\n3. **Flexible and Innovative Parameterization**  \n   The proposed parameterization, distinct from that of S4S [1], introduces additional flexibility to the solver. This results in measurable improvements in both performance and generalization.", "weaknesses": "1. **High Training Cost from Adversarial Loss**  \n   While the adversarial loss improves overall quality, it significantly increases the training cost. As shown in Table 8, the training time is more than double that of GS (without adversarial loss), raising questions about computational efficiency.  \n\n2. **Questionable Initialization Strategy**  \n   The solver learns coefficients for a linear multistep method but initializes some parameters from the DPM solver [2], which belongs to the exponential integrator family. Since exponential integrators differ fundamentally from linear multistep methods, this initialization may lead to inconsistent behavior and a theoretically unsound starting point.  \n\n3. **Dependence on a Pretrained GAN**  \n   The method’s reliance on a pretrained GAN introduces unnecessary complexity and limits its generalizability. This dependency may hinder reproducibility and make it difficult to apply the method across different architectures.  \n\n4. **Unfair Comparison with S4S**  \n   The reported comparisons with S4S [1] may be misleading, as solver performance heavily depends on the teacher model used. Variations in teacher configurations can skew the results, weakening the validity of the performance claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qochXePX32", "reviewer": "Reviewer_kJGM", "summary": "The paper proposes an ODE solver distillation method for diffusion models that augments prior work with: an adversarial loss, a better initialization inspired by theory in other ODE solvers, and deliberate over parameterization. The goal is to improve sample quality at low NFE. Experiments and ablations indicate consistent gains over prior work, with the improvements most pronounced at low NFE. Ablations also isolate the utility of the adversarial loss and the theoretically sound initial parameterization. The work also provides useful evidence on cross dataset generalization and argues for faster training relative to consistency model or progressive distillation approaches.", "questions": "- Line 243: Please include the exact formula used for the initialization adapted to your notation. The cited paper does not provide sufficient detail on the higher order 3M variant.\n\n- In Eq. 22, the meaning of p_T Y_T is not specified. Is it to indicate a different noise is sampled from the prior for the teacher.\n\n- By Eqs. 7 and 22 it appears training does not depend on real training data. Then what does Sec. 4.3 \"Dataset size\" vary: number of noise samples, or a teacher generated dataset. Do you first generate a dataset with the teacher and then train on it? If so  are the generated samples used multiple times for training? (appendix suggests yes to both but it would be better to mention these details in main text as well)\n\n- Did you test lower NFEs  than reported (one or two)? If so, what failed or succeeded?\n\n- Do the learned time steps t_n + \\xi_n increase with n? or does training learn a mixed or non monotone schedule?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The work improves the performance of the ODE solver distillation paradigm.\n\n- Thorough ablations demonstrate the contribution of each component, with the adversarial loss boosting performance at low NFE.\n\n- Cross dataset generalization results in the appendix are strong and main text worthy.\n\n- The appendix argument for faster training relative to consistency models or progressive distillation strengthens its practical appeal within the solver distillation family.", "weaknesses": "- The paper should explicitly state whether training is data free. By equations 7 and 22, no dependence on training data is apparent, yet in section 4.3 \"Dataset size\" is used and it is unclear. It seems to refer to the number of samples generated by the teacher, as parts of the appendix suggest.\n\n- Line 475 claims the generalized solver parameterization \"significantly accelerates training\" relative to existing parameterizations. Section 4.2 compares final performance to S4S but does not show convergence speed. Appendix mentions the time required is similar to S4S (line 910).\n\n- There exist methods that do not optimize the ODE solver but improve samples directly and can outperform solver distillation at very low NFE (for example Truncated Consistency Models [1]). The paper should acknowledge this limitation of the solver distillation class more clearly.\n\n- The paper candidly acknowledges two important weaknesses shared by ODE solver distillation: per NFE training and backpropagation through the whole solver inference.\n\n- Line 129 links \"Table 2\" to the table in this paper, but it seems to refer to a table in another work.\n\n- The mode collapse discussion in the appendix line 1238 is hand wavy. Consider a quantitative method like coverage from Reliable Fidelity and Diversity Metrics for Generative Models [2]. (This is a minor issue as the claim is in the appendix and the method only changes the ODE-solver and not the main model)\n\n\n[1] Lee, Sangyun, et al. \"Truncated Consistency Models.\" The Thirteenth International Conference on Learning Representations.\n\n[2] Naeem, Muhammad Ferjad, et al. \"Reliable fidelity and diversity metrics for generative models.\" International conference on machine learning. PMLR, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)