Score-of-Mixture Training: One-Step Generative Model Training Made Simple via Score Estimation of Mixture Distributions
Authors: Tejas Jayashankar
Jongha Jon Ryu
Gregory W. Wornell
Venue: ICML 2025 spotlightposter (2025)
Keywords: one-step generation
skew Jensen-Shannon Divergence
diffusion models
score estimation
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We introduce Score-of-Mixture Training, a simple and stable framework for training one-step generative models from scratch using the α-skew Jensen–Shannon divergence by estimating the scores of mixture distribution across multiple noise levels.

Abstract: We propose *Score-of-Mixture Training* (SMT), a novel framework for training one-step generative models by minimizing a class of divergences called the
$\alpha$-skew Jensen–Shannon divergence. At its core, SMT estimates the score of mixture distributions between real and fake samples across multiple noise levels.
Similar to consistency models, our approach supports both training from scratch (SMT) and distillation using a pretrained diffusion model, which we call *Score-of-Mixture Distillation* (SMD).
It is simple to implement, requires minimal hyperparameter tuning, and ensures stable training. Experiments on CIFAR-10 and ImageNet 64×64 show that SMT/SMD are competitive with and can even outperform existing methods.

Score-of-Mixture Training:
One-Step Generative Model Training Made Simple via Score Estimation of Mixture Distributions
Tejas Jayashankar"
J. Jon Ryu"
Gregory Wornell
Abstract
2021; Salimans & Ho, 2022; Meng et al., 2023; Berthelot et al., 2023) focused on training student model with a lower sampling budget by condensing multiple teacher denoising steps into one. The most recent works on distillation im- prove performance further by leveraging a pretrained model for distribution matching via minimization of the reverse KL divergence (Luo al., 2024a; Yin et al., 2024b;a; Salimans et al., 2024; Xie et al., 2024). While attractive, distillation approaches necessitate pretrained diffusion model which adds significant overhead on the required compute.
We propose Score-of-Mixture Training (SMT), novel framework for training one-step genera- tive models by minimizing class of divergences called the a-skew Jensen-Shannon divergence. At its core, SMT estimates the score of mixture distributions between real and fake samples across multiple noise levels. Similar to consistency mod- els, our approach supports both training from scratch (SMT) and distillation using a pretrained diffusion model, which we call Seore-of-Mixtur Distillation (SMD). It is simple to implement, re- quires minimal hyperparameter tuning, and en- sures stable training. Experiments on CIFAR-10 and ImageNet 64x64 show that SMT/SMD are competitive with and can outperform exist- ing methods.
As yet another alternative, consistency models (Song et al., 2023; Song Dhariwal, 2024b) and their variants (Kim et al., 2024) have been proposed for training few-step gener- ative models from scratch by simulating the trajectories of the induced probability flow ODE (Song et al., 2020) of diffusion process. While consistency models have demon- strated promising results in both distillation and training from scratch, training is sensitive to the choice of noise schedule and distance measure (Geng et al., 2025).
1. Introduction
In this paper, we tackle the problem of training high-quality one-step generative models more directly, i.c., without sim- ulating iterative reverse diffusion process for sampling or leveraging pretrained diffusion model during training. Starting from first principles of statistical divergence mini- mization, we show that a high-quality one-step generative model can be trained from scratch in a stable manner, via the multi-noise-leved denoising score matching (DSM) tech- nique (Vincent, 2011) used in diffusion models. We em- phasize that do not require simulation of the reverse diffusion process in our framework.
Fast and efficient sampling key characteristic sought after in modern generative samplers. For many years, gen- erative adversarial networks (GANs) (Goodfellow et al., 2014) set the benchmark for high-quality one-step genera- tive sampling. However, due to the inherent training insta- bilities associated with discriminator training, attention has recently shifted toward diffusion- generative models (Sohl-Dickstein et al., 2015; Ho al., 2020; Karras al., 2022b). These models trade-off sampling efficiency for more stable training and significantly improved downstream sample quality through iterative sampling.
The proposed framework achieves the best of several worlds: (1) a new, simple statistical divergence minimization frame- work without probability paths of ODE (like GAN), (2) stable training using denoising score matching (like diffu- sion models), (3) training from scratch without a pretrained diffusion model (like consistency models), and (4) near state-of-the-ar one-step image generative performance (like GAN and consistency models). We also demonstrate that the proposed method can be extended to distill from a pretrained diffusion model, and can achieve performance similar to state-of-the-ard methods for the same. See Table I for the overview of comparison.
More recently, the diffusion distillation approach has been studied as an appealing option to significantly reduce the number of sampling steps. Early work (Luhman & Luhman,
'Equal contribution 'Department of Electrical Engineer- ing and Computer Science, Massachusetts Institute of Tech- nology (MIT), Cambridge, Massachusells, USA. Correspon- dence to: Tejas Jayanshankar <tejasj@mit.ddu> Jon Ryu <jongha.ryu@ gmail.com>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Scorc-of-Mixture Training: Training One-Step Generative Models Made Simple
Table Comparison of different generative modeling techniques capable of high-quality sample generation.
Training
Require pretrained model?
Generative models
Training idea
Generation
stability
GAN
minimizing JSD, with discriminator
one-step
unstable
N
Diffusion models
training multi-noise-lew denoisers DSM (mostly) minimizing reverse KLD (in DMD)
multi-step
stable
Diffusion distillation
Jone.fow]-stor
stable
Consistency distillation
stable
Y
simulating trajectories of probability flow ODE
[one,few)
Consistency training
unstable
N
Score-of-Mixture Training (ours)
minimizing (a-ISD).ejon with multi-noise-level
N
Score-of-Mixture Distillation (ours)
training & scores of mixtures via DSM
one-step
stahle
The rest of the paper is organized as follows: Sec. we introduce the necessary background and related works central to proposed method. In Sec. 3 we introduce our novel one-step generative modeling approach and in Sec. 4 we detail how our framework can be modified to perform diffusion distillation. We describe practical implementation details in both the latter sections and present experimental results Sec. We conclude with remarks in Sec. Proofs and training details are deferred Appendix.
imizing the Jensen-Shannon divergence (JSD) with the help of discriminator, via the variational characterization of JSD,
Despite the popularity of GANs, training them is notoriously difficult. Although various techniques have been proposed to regularize the GAN objective-through alternatives to JSD (Nowozin et al., 2016; Arjovsky et al., 2017; Mao et al., 2017), novel regularizers (Miyato et al., 2018), and specialized network architectures (Karras al., 2021; Brock et al., 2019; Sauer et al., 2022)-the discriminator training remains unstable. This has sparked increasing interest in developing new objectives for training generative models which we briefly discuss below.
2. Preliminaries and Related Work
In one-step generative modeling, we wish to align the gen- erated sample distribution (x) := with the true data distribution p(x). Here, ge: + X' is a parametric neural sampler which is also often called an implicit generative model that transforms samples from base measure q()). In this section, we review some popular methods for training generative models, which will serve as preliminaries for our framework. More detailed discussion on the literature is deferred to Appendix B.
[5 8(x (o(z))g dz
Diffusion Models. Diffusion models or score-based gener- ative models (Sohl-Dickstein et al., 2015; Ho et al., 2020) are tfte-off-the-ar generative models that are based on the principles of thermodynamic diffusion. Given forward stochastic differential equation (SDE) process
dx; = fxx,, t) + g(t)dw::
where f(x,, t) is the drift function, g(t) is the diffusion func- tion, and w, represents Brownian noise process, diffusion models simulate the reverse (generative) process, which is also SDE
Generative Adversarial Networks. The most prominent approach in training implicit generative models is the gener- ative adversarial network (GAN) (Goodfellow al., 2014). In its most standard and widely used form, it alternates between the gradient steps of discriminator and generator training, which are
dx; [f(x,,t) -
g(t) (ogpVVxlxlpp///// log logp(x;))dt g(t)dw
An equivalent deterministic probability flow ODE with the same marginals as the SDE can also be used in practice:
min (xxx +E (s((<(())))
min Equ( (s(c))/p((eep) css (x))), (x))],
(2)
dx, = (f(x,,t) -
1 _ log log p(x.) )
respectively, where sp(y) == log(1 + et) denotes the soft- plus function." Here, we will call Ep (x) the discriminator, which is supposed to capture the log density ratio log p(x) 2 This so-called adversarial training can be understood as min-
Thus, to generate samples, diffusion models are trained to learn the score of the data distribution at multiple noise levels or via denoising score matching (DSM) (Vincent, 2011), i.c. by minimizing
go (x)'
'The generator loss (x) (x)) in the second line the so-called non-saturating version, while the original GAN generator loss Gx (pp is referred to as saturating. "Note the one-to-one correspondence between the standard exp(t,(xx)
LUSM (8) Ep(0)(2))1()) [=xtt]lso(xx;tt s(x; [x]]
where p(t) denotes a distribution over different noise lev- els, X, = x OFE,6 NO,,I), and s(x/(x) ==
definition discriminator D& (x) = (x0 E [0, 1].
2

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
(x)). It is easy to show that (x(x;: ()
whose gradient can be approximated by estimating the score of mixture distributions of real and fake distributions (Sec. 3.3), hence the name Score of Mixture Training, We adopt the concept of multi-noise level learning from diffu- sion models and propose multi-divergence minimization for stable training (Sec. 3.2). A practical implementation of our method is described in Sec. 3.4, followed by details of the training procedure in Sec. 3.5.
x, log (x+) using Tweedie's formula (Robbins, 1956).
Sampling can then be achieved by Langevin dynamics (Song
& Ermon, 2019; Song et al., 2020) or via black-box ODE
solvers (Karras et al., 2022b; Lu et al., 2022b;c).
Diffusion Distillation. In practical applications, running a diffusion model for multiple steps to generate single sample can be prohibitively expensive. Distilling few-step generative models from high-quality pretrained diffusion model has thus become popular (Luo et al., 2024a; Yin et al., 2024b;a; Salimans et al., 2024; Xie et al., 2024). To learn the generator's 's parameters, most, if not all, approaches aim to minimize the reverse Kullback- Leibler divergence (KLD) DRI. (4o ()) averaged across multiple noise levels:
3.1. Minimizing a-Skew Jensen-Shannen Divergences
The crux of the new framework lies in minimizing a class of statistical divergences between p(x) and (x(x) defined as
D Dio p) == _ Dgo op ap+ (1 - )))
1
DKi. (ge Ip) := (x (x))(((((()) [log qo (xx) - log p(x:)].
a
+ 1 (p op + (1 a))a)
To update the parameters via gradient descent, the gradient of this divergence is computed as
for some a E (0, which we call the a-skew Jensen- Shannon divergence (a-JSD) (Nielsen, 2010). This diver- gence belongs to f-divergences (Csiszár et al., 2004).
V V&DRE
(3)
Ey(e))(()(e)) [Vago (xx²t) sy(x1; ()) Ix=goiz]]
Interestingly, a-skew JSD naturally interpolates between the forward Kullback-Leibler divergence (KLD) DRL (p || qa) (when - 0), the standard definition of JSD (when and the reverse KL.D (xI p) (when a + 1). In contrast to the forward KLD and reverse KLD, the a-skew JSD with o E 10, is well-defined even when there is a support mismatch in p and qa. which may be the case especially in the beginning of training.
where Sy, and Sp are the noisy scores of the fake and true samples, respectively. In the distillation setup, pretrained diffusion model is plugged in as a close proxy to the true noisy score sy (xx; t), while the fake noisy score Sno (Xxi t) is trained along with the generator to assist the training.
}1)
Consistency Models. Distillation approaches often rely on pretrained score models and may use expensive regularizers to address issues like mode collapse and improve sample quality (Yin al., 2024b; Salimans et al., 2024). contrast, consistency models (Song et al., 2023; Song & Dhariwal, 2024b), which can be trained from scratch, simulate the underlying probability flow ODE and ensure each sample along the trajectory maps to the origin. Con- sistency training, however, can be unstable and is known to sensitive to the noise schedule and distance function (Geng et al., 2025). Additionally, the architecture for consistency models need to be carefully chosen, as the approach relies on single-sample approximation of Tweedie's 's formula, which is only valid when noise levels are closely spaced.
Feature 1: Multi- Duvti-Divergence Training, Hence, we pro- pose to minimize weighted sum of the a-JSD's for differ- ent a's, as divergences with different a's 's exploit different geometries between two distributions. For example, it is known that minimizing the forward and reverse KLD leads to mode-covering and mode-seeking behaviors, respectively, and we can enforce better support matching behavior by considering the entire range of a.
trained to
To minimize this family of divergences in practice, we con- sider its gradient expression:
Proposition 3.1. Suppose that (x) (V Volog [x 0. Then, we have
3. Training from Scratch
V VoD (40, P)
(4)
In this section, we introduce our new framework, Score- of of-Mixture Training (SMT). We describe how to efficiently train one-step generative models from scratch, i.e.. without pretrained diffusion model. In Sec. 4, we explain how the framework can be adapted to leverage a pretrained diffusion model when available, referring to this variant as Score-of- Mixture Distillation (SMD).
1 Ev(e)
(ogo
s0:a (x)((x)) (x))
a
where we define the score of the mixture distribution
(x) := Vx log(ap(x) + +(1 - (x)o
3It is standard assumption in the literature (Hyvärinen, 2005), which holds under mild regularity assumption on the parametric model (x) so that I Vogo(x) dx = f go (x)xxx.
The key ingredient of this framework is distribution match- ing using a new family of statistical divergences (Sec. 3.1),
3

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
This proposition suggests that we can update the generator Bo(z) using this gradient expression, provided that we can estimate the score of the mixture distribution 86 (x).
Proposition 3.2. For any a E [0, 1), the minimizer of the objective function
((v;a) = a Ep(xillss (x; (x,xx)
Feature 2: Amortized Score Model. To implement this idea, in this paper, we propose to use an amortized score model (x, a) + Su (x; approximate the score of mixture (x). Through our experiments we show that learning the scores of mixture over different a's using a single model is effective and helps training. In Sec. 3.3, we explain how we can train the amortized score model (x, a) + (x(xxa a) using samples from p(x) and (x(x).
+(1 aa) Egnti)]lss - Sup (x)
(7)
satisfies sy- (x; a) = seaa (x).
Since we train with multiple noise levels, we are interested in the marginal score of x = x o2C,C ~ NO I) at some noise level ot. We can use denoising score matching (Vincent, 2011) to define an equivalent sample-only objec- tive to learn the score using Tweedie's formula. Namely, to approximate s (x) using the amortized score model Sup (x; az l), we can minimize
3.2. Learning with Multiple Noise Levels
To achieve stable training, we opt to minimize the diver- gence at different noise levels by considering the con- volved distributions, P: := p * (N ID) and qo.: := qe o(, This idea is widely used in the exist- ing distillation methods. We borrow the variance-exploding Gaussian noising process notation from Karras et al. (2022b)
Lucore Ep(a))(t) (Cooce (1;; (xtoopte a, a, t)
where
Cicoro Cacono a,,)
(8)
where of E [miin maxx] As we also integrate over different o's, the final objective becomes
== a Ep(xx)ate) (X e/oull + (1 - a) (x)( (xs; (x;;, a, + +/
(0) Ep(a))(1) [D (10,t P:)]:
(5)
See Proposition A.1 for formal statement. In practice, we parametrize the score model in the form of a denoiser and reconstruct the score from the denoiser output via Tweedie's formula; see Appendix C.I.
where we will prescribe the choice of p(a) in Sec. 3.5.
Similar to Eq. (4), the gradient of the divergence at noise level f can be approximated via the amortized score as
DoD (0o,r, Pc) R (0 ox t) S. (x,x 0,1) - s (xx;,1) == Eg(x))(e) (z)
(6)
Feature 3: Leveraging Real and Fake Samples. We re- mark that our score learning objective seamlessly utilizes both real and fake samples throughout the training, helping the generator better generalize. This is in contrast to some existing diffusion distillation methods, which introduce ex- pensive regularizers to integrate real samples, or backpropa- gate through the pretrained score model (Yin et al., 2024b;a; Salimans et al., 2024).
a
(x)
where the amortized score model sy (x:: a 1), which is con- ditioned on the noise level f, is an estimate of sp (xx) Vx, log(or((.) + (1 ax)qo (x:)). We provide practical implementation of the amortized score model as a small modification of a diffusion model architecture in Sec. 3.4. We remark in passing that this expression can be understood as a generalization of the gradient update of Eq. (3) used in the existing reverse-KLD-basse KLD-based distillation schemes.
3.4. Practical Design of Amortized Score Network
With an additional conditioning scheme to embed auxiliary information about in addition to the noise level o, any existing diffusion model backbone can be used to param- eterize the amortized score network s. (x; a, t). Here, we describe how we can modify the popular bNet-based score architectures (Song et al., 2020; Nichol & Dhariwal, 2021; Karras et al., 2022b) with minimal modifications.
Finally, we can then approximate the generator gradient
VoCgen R Ep(a))(tl7 (,, t)].
Importantly, similar to existing distillation methods, the gradient only involves the output of the score model, but not its gradient. This is beneficial since such extra gradient information requires expensive backpropagation through the score model to the generator (Zhou et al., 2024).
First, drawing from the noise embedding sensitivity analysis by Song & Dhariwal (2024b), opt for Fourier embed- ding C. with a default scale of 16. This choice ensures that the embedding is sufficiently sensitive to fluctuations in particularly during the early stage of training.
3.3. Estimating Score of Mixture Distributions
Estimating the score of the mixture distribution turns out to be as simple as minimizing a mixture of the score matching losses, as stated in the following proposition:
Then, we concatenate the a-embedding with the embedding of other auxiliary information (e.g., t and labels) and apply
4

---

Scorc-of-Mixture Training: Training One-Step Generative Models Made Simple
single SiLU (Elfwing al., 2018) activated linear layer:
This weighting still preserves the limiting forward KLD behavior of the objective as - 0 and simplifies to DMD gradient when of our adaptive weighting term Wa (X,, t) through ablation studies on the CIFAR-10 dataset in Sec. 5.3; see Fig. 2b.
Cout = silu( W nux Caux + W C..)).
1. We empirically show the efficacy
The rationale behind this choice is as follows: as training progresses, the real and fake distributions begin to overlap, making it natural for the amortized score model to become less sensitive to a. Thanks to the additional linear layer W. a after the a-embedding Co. this behavior can be realized when W. ~ O, when necessary.
Regularization with GAN. We empirically found that a GAN-type regularization can accelerate convergence even further in the beginning of training. More concretely, we can train the discriminator & (xx; 1) Rs log pix) by the GAN discriminator training in Eq. (1). In our implementation, we opt to train a discriminator using a variant based on the a-JSD, as described in Appendix D.2. Given a discriminator ly (x,;),, we minimize a non-saturating version of the a- JSD loss (ef. Eq. (2)).
qx(
3.5. Training
Alternating Training. Our training scheme alternates be- tween the score estimation with the score matching objective in Eq. (8), and the generator training with Eq. (6), where we plug-in Sp (x;; Ot 1) in place of $0;0, (Xt). This is sim- ilar in spirit to GAN training, but the DSM technique in our framework in place of the discriminator training natu- rally stabilizes training. The overall training framework is summarized in Fig. 1 and Alg. in Appendix C.
cla.t) GAN (0) = (0)=E))(x))) [sp(-eu (xx:t) log - (11)
The derivation can be found in Appendix D.2. Similar to Yin et al. (2024a), we parameterized the discriminator by a stack of convolution layers, applied on top of an intermediate feature of the amortized score network at a = 1/2.
Initialization. We warm up the generator with standard denoising task as in diffusion models for several steps to better initialize the weights, as we empirically found that initializing the generator with pretrained weights from a de- noiser significantly accelerated convergence. The amortized score network is randomly initialized.
Similar to DMD2 (Yin al., 2024a), we implement a GAN discriminator building on top of the score network, with only few additional MLP layers. This pooe--oddl-ddeendeen design allows the full model to benefit from the training stability provided by denoising score matching, while the GAN discriminator loss only trains the small auxiliary MI.P (For ImageNet, the generator has 296M parameters and the discriminator has 1 18M.) Thus, the discriminator represents small fraction of the overall model size and has negligible impact on training speed. As a result, our use of the GAN regularizer is both efficient and stable.
Choice of p(a). The choice of p(a:) is crucial in our frame- work. To train both the generator and score model, we sample a from a uniform distribution over 1000 equally spaced points in [o. 1). ensuring a dense enough grid to gen- eralize to any a. For score training, we further ensure that 25% of the sampled a's are zero, since this is always used in our gradient update; see Eq. (6).
Adaptive Weighting. In practice we compute the gradient with an adaptive weight w(,,, a) to ensure that the scale of the gradient for each minibatch sample is roughly uniform for different values of a and t. Hence, we modify the generator gradient in Eq. (6) as
4. Distilling from Pretrained Diffusion Model
In our development so far, we do not assume access to a pretrained diffusion model. In this section, we show how practitioner can train one-step generative model lever- aging a pretrained diffusion model, if available, within our framework. The proposed distillation scheme is comparable or even outperforms the state-of-the-ar distillation schemes.
75 a. == Ex(x) VZog(zz)
(9)
{ {w (x(x,, a,
So (Xx; (x;; (x )
&
where the weighting is defined as
4.1. How To Leverage Pretrained Diffusion Model
x,x,,x,a,t) == wn (x,, t)unmn x, t). (10)
In the distillation setup, we treat the pretrained diffusion model as the data score sp (xx; t). and thus training the score of mixture so.a (x;: t) using a single, amortized model may not be the most efficient parameterization. Hence, instead, we consider the following expression
Here WDMD is the adaptive noise weighting introduced by (Yin et al., 2024b) (see Eq. (32) in Appendix B) and
w. (xx, t) is new weighting inspired by the psbudo-Huber norm (Song& & Dhariwal, 2024a; Geng et al., 2025)
| 0,6) sy (x:; 1, 11
C. (x., / || (Xe: 0,6) sy (xx; 0, 1)
So:a (x) = Doza (x)sp(x) + (1 -osa -o:aax))) (x);
5

---

Scorc-of-Mixture Training: Training One-Step Generative Models Made Simple
Generator
(1)
Amortized Score Model
Sy (xeeale, 00
2 (),
x fake go(a)
Take
5y (x) (xx ax
Noise
Fake Image
(0: (...))
Generator Update (Eq. (6))
N(0,
fake
X
t) Bo [x (xt
Real Dataset
x
si (x/(x/ (xtaal aa,))
Real Image
Amortized Score Update (Eq. (8))
(1
Figure 1. Overview of SMT. Top: To update the generator, compute the gradient of the a-JSD noisy fake samples with the frozen amortized score model using Eq. (6). Bottom: The amortized score model is updated by computing the score the mixture distribution on both fake and real noisy samples, and then updating the weights using the gradient in Eq. (8).
where
Corollary 4.1. Let a E [0, 1] be fixed and oN be some fixed noise level. Then, the minimizer of the objective function
ap(xx
Do:a = ap(x) + (1 - ()))(x))
L (00(1) a,t)
(14)
= (10g p(x)
log -
= a Ep(x)ote)llss exp (x;;, + t/ooll?]
(x(
(1 (1-a) a)) E ( c/or11
See Proposition A.2 for formal statement. In words, we can express the score of mixture Soco (x) as a mixture of scores Sp and Sgo where the weight is (Do:a (x), (x)). This suggests that instead of an amortized mod- eling of the score of mixture, we can use an alternative parameterization,
satisfies
sfake (x; Syo (x;;) and (xx; log p(x,) qo (xt)
We remark that this regression objective in Eq. (14) pro- vides new way compute the log density ratio, an alter- native to the GAN training (see Eq. (1)). In Appendix D.3, we establish a connection between this objective for training a discriminator to an existing GAN discriminator objective in the literature.
exP <xp at) = Do (x; (1 Di (x; (x),
where
(=
:
log
Here, we can parameterize the discriminator x ++ (x) in the same way as we do for the GAN discriminator.
With this new, explicit parameterization, we can approxi- mate the gradient expression in Eq. (6) as
We can extend this multiple noise levels easily. Hence, an alternative parameterization for se (xx; t) is
V VoD ISD (g) exp 2J 7 (0; a,;,t)
(15)
(x D& (x15 a, ()s (xei( () + (1 - Do (x:: a, t)) (x(: t),
(12)
Ex(z)
D (x
Sake (xtil) sp(xe; 1)
where
(z)
(x=g)(x)
Do(x; a, := (Lv( + log a
)
(13)
4.2. Implementation and Training
Plugging this explicit score model into Eq. (8), we can learn both the fake score model fake and the discriminator lo at different noise levels.
Model Architectures. We can leverage any existing diffusion model architectures directly for the fake score
6

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
sfake (xx; I). We parametrize the discriminator (x () sim- ilar to the noise-conditional discriminator in our training from scratch setting (see Sec. 3.5). The difference is that we can train the discriminator by minimizing the DSM loss in Eq. (14) naturally, without an additional GAN loss. When training the generator, we plug in this approximate log den- sity ratio into Eq. (11) to regularize the generator updates.
with batch size of 512). Our model also competes favorably with iCT-deep, despite using a generator with half the num- ber of parameters: FID of 3.23 with 296M parameters (ours) vs. 3.25 with 592M parameters (iCT-deep). We observed stable training throughout, without requiring extensive hy- perparameter tuning or special noise schedule adjustments as consistency training, as visualized Fig. 2a. We also suraass the ECT model (Geng et al., 2025) of similar size and training budget that includes several modifications to induce stability in consistency training. Samples generated using our method can be found in Fig. 3 and Appendix E.
Training. We also train in an alternating fashion. Since we have access to pretrained score model, we use this to initialize the weights of both the generator and the fake score model. We utilize the same sampling distribution for a as in our training from scratch setup (see Sec. 3.5). The procedure is summarized in Fig. 4 and Alg. 2 in Appendix C.
In the distillation setting, our model achieves a competitive FID of 1.48, outperforming several baselines. Notably, we outperform consisteney distillation methods, such as mul- tistep consistency distillation (Heek al., 2024), despite using only fraction of the model size (256M parameters against 1200M parameters). Our model also surpasses con- sistency trajectory models (CTM) (Kim et al., 2024), which ensure consistency between random points along the PF ODE trajectory, by simulating the reverse diffusion sam- pler for an arbitrary number of steps per minibatch, thus resulting in high computational cost.
5. Experiments
In this section, we first present results on the ImageNet 64 x 64 dataset. We then demonstrate the competitive- ness of our method on the CIFAR-I0 dataset and con- duct a series of ablation studies. We measure performance through sample quality as measured by the Fréchet Incep- tion Distance (FID) (Heusel et al., 2017). The exact hy- erparameters, training configurations used and additional results, including example training dynamics and la- tent interpolation, can be found in Appendix E. Our imple- mentation can be found at score- of- mixture- training.
We also outperform reverse- KLD methods with similar com- pute or regularizers such as DMD (Yin et al., 2024b) and DMD2 (Yin et al., 2024a) with FIDs of 5.60 and 1.51 respec- tively. We note that on spending significant extra compute, DMD and DMD2 achieved improved results. For example, in the DMD framework without any GAN regularization, the authors simulate the reverse process of a diffusion model and sample several thousand noise-image pairs anchor the generator's outputs. Each noise-image pair requires evaluating the diffusion denoiser 256 times for ImageNet 64x64, which is extremely costly in practice. In contrast in the DMD2 framework the authors adopt a lengthy finetun- ing stage with GAN regularization of 400k steps to further improve results.
wttpt/uu...o///s////aa.... /
5.1. Class-conditional ImageNet 64x64 Generation
Experimental Setup. We trained class-conditional one- step generative models on ImageNet 64 x 64 (Deng et al., 2009), experimenting with both distillation and training from scratch. In both cases, we used the ADM architecture (Nichol & Dhariwal, 2021) as the base score model archi- tecture. and the discriminator (x;;);; t) was implemented as stack of convolution layers operating on the bottleneck feature from the score network, similar to DMD2 (Yin et al., 2024a). For training from scratch, we augmented the score architecture using an a-embedding as described in Sec. 3.4. The total number of parameters of the amortized score model remained unchanged otherwise. As a warmup stage, we pretrained the generator the dataset using standard diffusion denoising objective for 40k steps to ini- tialize the weights. For distillation, we used a pretrained diffusion model from (Karras et al., 2022b).
We did not resort to any of the above techniques and sought to find an approach that worked best with single execution of the training pipeline run for 200k steps.
5.2. Unconditional CIFAR-10 Generation
Experimental Setup. We evaluated our method on the CIFAR- 10 dataset (Krizhevsky et al., 2009) for uncondi- tional one-step generative modeling, considering both train- ing from scratch and distillation. In both cases, we em- ployed a DDPM++ architecture (Song et al., 2020) with EDM preconditioning (Karras et al., 2022b). The discrimi- nator again followed the convolutional stack used in DMD2. For training from scratch, we modified the score model to incorporate the a-embedding (Sec. 3.4) while maintaining a similar network size. To mitigate overfitting due to the dataset's small size, we enabled dropout with p = 0.13,
Results. We evaluated our method against several published baselines for both training from scratch and distillation. As shown in Table 2, when trained from scratch, our gener- ator with 296M parameters outperforms both consistency training and its improved variant (Song et al., 2023; Song & Dhariwal, 2024a), with a much smaller training budget (200k iterations with batch size of 40 vs. 800k iterations
7

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
Table Image generation results on ImageNet 64x64 (class-conditional) and CIFAR-10 10 32x32 (unconditional), The size of the sampler is denoted by the number parameters (ff params), and NFE stands for the Number of Function Evaluations. The best FIDs from each category are highlighted in bold, and our methods SMT and SMD are highlighted with a blue shade.
SMT/SMD on ImageNet 64x64
SMT (scratch)
(distillatio)
ImageNet 64x64
CIFAR-10 32x32
10!
Method
purams NFE
FIDI
params
NFE
FID
Training from scratch: Diffusion models
DDPM (Ho al., 2020)
56M
1000
3.17
ADM (Dhariwal & Nichol, 2021)
296M
250
2.07
EDM (Karras al., 2022b)
296M
512
1.36
56M
35
1.97
5.0
7.5
10.0
17.5
20.0
Training from scrotch: One-step models
training steps
x10*
2-RF+ distill (Liu et al., 2022)
56M
4.85
CT (Song 2023)
296M
56M
8.70
(a) ImageNet 64x64 (seratch and distillation).
iCT (Song& & Dhariwal, 2024a)
296M
4.02
56M
2.83
iCT-ddeep (Song Dhariwal, 2024a)
592M
3.25
112M
2.51
SMT on CIFARIO
BCT (Geng ul., 2025)
280M
5.51
56M
3.60
10a
SMT (ours)
296M
3,23
56M
3.13
random a: DMD weighting
our weighting
Diffusion distillation
weighting
PD (Salimans Ho, 2022)
296M
10.7
60M
9.12
randam our weighting GAN
TRACT (Benhelot 2023)
296M
7.43
56M
3.78
CD (LPIPS) (Song al., 2023)
296M
6.20
56M
4.53
Diff-Instruct (1 Lao al., 20244)
296M
5.57
56M
4.53
MultiStep-CD (Heek al., 2024)
1200M
3.20
10a
DMD w/o (Yin al., 2024b)
296M
5.60
56M
5.58
DMD2 w/ GAN (Yin et al., 2024a)
296M
1.51
56M
2.43
MMD (Salimans et al., 2024)
400M
3.00
SiD (Zhou et al., 2024)
296M
1,52
56M
1.92
2
10
12
training steps
x10
SiM (Luo et al., 2024b)
56M
2.02
2-RF ++ (Lec et al., 2024)
296M
3.07
56M
4.31
(b) CIFAR-I0 with ablation studies (scratch).
SMD (ours)
296M
L488
56M
2.22
w/ expensive regularizer, simudation or finetuning
CTM (Kim et al., 2024)
296M
1.92
56M
1.98
DMD w/ reg (Yin et al., 2024b)
296M
2.62
56M
2.66
Figure 2. FID evolution with training.
DMD2 (finctuned) (Yin et al., 2024a)
296M
1.23
as EDM. In the distillation setting, we initialized the generator with a pretrained unconditional diffusion model from (Karras et al., 2022b), using the same UNet backbone and weights. Distillation performed well without dropout,
Choice of Adaptive Gradient Weighting, Starting with our base objective without the GAN regularizer, we tested our (a, ad-adive weighting in Eq. (10). Fig. 2b demonstrates the benefits of weighting scheme, compared to the DMD weight function that only depends on t.
The last three columns in Table 2 highlight the per- formance of our method on CIFAR- 10 compared to various baselines. In our training from scratch setting, despite utiliz- ing a lower training budget (150k steps with a batch size of 40) than many methods, our approach remains highly com- petitive. In terms of training budget, the most comparable baseline is ECT, which requiring excessive design considerations and hyperparame- ter tuning. Our distillation results are also competitive. In particular, we outperform Diff-Instruct and DMD2, which are only based on minimizing the reverse KLD. This cor- roborates the benefit of our multi-divergence mininization approach. Image samples can be found in Appendix E.5.
Results.
Learning with Single vs. Multiple a's. The a-JSD re- duces to the reverse KI.D of DMD and other distillation methods, when a = 1. To test the efficacy with multi-a learning, we implemented an amortized variant, training the score model only with E 10, Results show that conditioning
able outperform without
range of o-values not only minimizes multiple divergences but also strengthens the embedding as a conditioning signal thereby facilitating more accurate divergence minimization.
Accelerated Convergence with GAN Regularizer. We finally verify the benefits of our novel GAN-type regularizer for a-JSD minimization. As demonstrated by the second and fourth curves in Fig. 2b, the GAN regularizer helps ac- celerate convergence especially in the beginning of training.
5.3. Ablation Studies
We use the CIFAR-10 dataset to study the effectiveness of the design choices that we have proposed; see Fig. 2b.
8

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
learning datasets, but we recognize the importance of ensur- ing generative models are secure and peeeecyprrs democratize this technology responsibly.
Acknowledgements
This work was supported in part by the MIT-IBM Watson Al Lab under Agreement No. W1771646, by MIT Lincoln Lab- oratory, by ONR under Grant No. N000014-23-1-2888 and by the Department of the Air Force Artificial Intelligence Accelerator under Cooperative Agreement No. FA8750- 19-2-1000. The views and conclusions contained in this document are those of the authors and should not be inter- preted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Govermennt. The U.S. Government is authorized to re- produce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
Figure 3. Samples from SMT on ImageNet 64x64, Each row represents a unique class. Additional samples can found in Appendix E.5.
6. Concluding Remarks
References
In this paper, we show that high-quality one-step generative models can be trained from scratch and in a stable manner, without simulating the reverse diffusion process or prob- ability flow ODE as in diffusion models and consistency models. The key distinctive idea in our framework a new multi-divergence minimization paradigm implemented by estimating the score of mixture distributions. For stable training, we borrow multi-level noise learning and denoising score matching techniques from the diffusion literature. Our empirical results show that accurate score estimation facili- tates stable minimization of statistical divergences. We hope this work offers a fresh perspective on generative modeling and inspires further research in the field.
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein GAN. In Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2017.
Berthelot, D., Autef, A., Lin, J., Yap. D. A., Zhai, S., Hu, S., Zheng, D., Talbott, W., and Gu, E. TRACT: De- noising Diffusion Models with Transitive Closure Time- Distillation. arXiv Preprint arXiv:2303.0422 2023.
Brock, A., Donahue, J., and Simonyan, K. Large Scale GAN training for High Fidelity Natural Image Synthesis. In Int. Conf. Learn. Repr., 2019.
Limitations and Future Work. While SMT/SMD achieve strong empirical performance, there is still room for im- provement in both architecture and training strategies. De- spite achieving highly competitive FID scores for one- step generation from scratch, models capable of few-step generation such as consistency models-mine might further improve FID with additional iterations. Finalll given the generality of our framework, we believe these ideas could extend to other complex modalities, including speech and audio synthesis. We leave such directions for future work.
Che, T., Zhang, R., Sohl-Dickstein, J., Larochelle, H., Paull, L., Cao, Y., and Bengio, Y. Your GAN is Secretly an Energy-Based Model and you should use Discriminator Driven Latent Sampling. In Adv. Neural Inf. Proc. Syst., volume 33, pp. 12275-12287, 2020.
Csiszár, I., Shields, P. C., et al. Information Theory and Statistics: A Tutorial. Found. Trends Commun. Inf. The- ory, 1(4):417-528, 2004.
Deng. J., Dong, W., Socher, Li, Li, K., and Fei- Fei, L. ImageNet: Large-Scale Hierarchical Image Database. In Proc. IEEE Comput. Soc. Conf. Comput. Vis. Puttern Recognit. pp. 248-255, 2009. doi: 10.1 CVPR.2009.52068.44
Impact Statement
We introduce Ccore-of-Mixtus Training, a simple yet effec- tive one-step generative modeling framework that requires minimal design effort and hyperparameter tuning. We hope its ease of implementation will drive further research into efficient, saateoof-the-an neural sampling. However, we acknowledge the potential risks of misuse, including the generation of fake, biased, or misleading content. Our work focuses on fundamental research using standard machine
Dhariwal, and Nichol, A. Diffusion Models Beat GAns On Image Synthesis. In Adv. Neural Inf. Proc. Syst., volume 34, pp. 8780-8794, 2021.
Elfwwng. S., Uchibe, E., and Doya, K. Sigmoid-weighter Linear Units for Neural Network Function Approxima-

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
tion in Reinforcement Learning. Neural Networks, 107: 3-11, 2018.
Kingma, D. and Gao, R. Understanding Diffusion Objec- tives the ELBO with simple Data Augmentation. In Adv. Neural Inf. Proc. Syst. volume 36, 2024.
Geng, Z., Pokle, A., Luo, W. Lin, J., and Kolter, J. 7. Consistency Models Made Easy. In Int. Conf. Learn. Repr, 2025.
Kingma, D., Salimans, T., Poole, B., and Ho, Varia- tional Diffusion Models. In Adv. Neural Inf. Proc. Syst., volume 34, pp. 21696-21707, 2021.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley D., Ozair, S., Courville, A., and Bengio, Y. Generative Adversarial Nets. In Adv. Neural Inf. Proc. Syst, volume 27, 2014.
Kingma, D. P. Auto-encoding Variational Bayes. In Int. Conf. Learn. Repr. 2014.
Kong, X., Brekelmans, R., and Steeg, G. V. Information- theoretic Diffusion. In Int. Conf. Learn. Repr., 2023.
Heek, J., Hoogeboom, E., and Salimans, T. Multistep Con- sistency Models, 2024.
Krizhevsky, A., Hinton, G., et al. Learning Multiple Lay- ers of Features from Tiny Images. Technical report, U. Toronto, 2009.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs Trained by Two Time-Scale Update Rule Converge Local Nash Equilibrium. In Adv. Neural Inf. Proc. Syst., volume 30, 2017.
Le Cam, L. Asymptotic methods in statistical decision theory. Springer Science & Business Media, 2012.
Ho, J., Jain, A., and Abbeel, P. Denoising Diffusion Proba- bilistic Models. In Adv. Neural Inf. Proc. Syst., volume 33, 2020.
Lee, S., Lin, Z., and Fanti, G. Improving the Training of Rec- tified Flows. In Adv. Neural Inf. Proc. Syst., volume 37, pp. 63082-63109, 2024.
pp.
Hoogeboom, E., Heek, J., and Salimans, T. Simple Dif- fusion: End-to-end Diffusion for High Resolution Im- ages. In Proc. Inl. Conf. Mach. Learn., pp. 13213-13232. PMLR, 2023.
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow Matching for Generative Modeling. In Int. Conf. Learn. Repr, 2023.
Liu, X., Gong, C., and Liu, Q. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. arXiv preprint arXiv:2209.0300 2022.
Huang, T., Zhang, Y., Zheng, M., You, S., Wang, F., Qian, C., and Xu, C. Knowledge Diffusion for Distillation. In Adv. Neural Inf. Proc. Syst., volume 36, pp. 65299-65316, 2023.
Liu, X., Gong, C., and Liu, Q. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. In Int. Conf. Learn. Repr. 2023.
Hyvärinen, A. Estimation of Non-Normalized Statistical Models by Score Matching. J. Mach. Learn. Res., 6(4), 2005.
Lu, C., Zhou, Y., Bao, F,, Chen, J., Li, C.,, and Zhu, J. DPM- solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling Around 10 Steps. Adv. Neural Inf. Proc. Syst., volume 35, pp. 5775-5787, 2022a.
Karras, T., Laine, S., and Aila, A Style-Based Generator Architecture for Generative Adversarial Networks. JEEE Trans. Pattern Anal. Mach. Intell., 43(12):4117-4228, Dec 2021. doi: 10.119//PAMM2220022999999
Lu, C., Zhou, Y., Bao, F., Chen, J., Li. C., and Zhu, J. DPM- Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. In Adv. Neural Inf. Proc. Syst., volume 35, pp. 5775-5787, 2022b.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the Design Space of Diffusion- Generative Models. In Adv. Neural Inf. Proc. Syst., volume 35, pp. 26565- 26577, 2022a.
Lu, C., Zhou, Y., Bao, F,, Chen, J., Li, C.,, and Zhu, J. DPM- Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models. arXiv preprint arXiv:221 1.01095. 2022c.
Karras. T., Aittala, M., Aila, T., and Laine. S. Elucidating the Design Space of Diffusion- based Generative Models. In Adv. Neural Inf. Proc. Syst., volume 35, pp. 26565- 26577, 2022b.
Luhman, E. and Luhman, T. Knowledge Distillation in Iter- ative Generative Models for Improved Sampling Speed. arXiv preprint arXiv:2101.02388, 2021.
Kim, D., Lai, C.-H., Liao, W.-H., Murata, N., Takida, Y., Uesaka, T., He, Y., Mitsufuji, Y., and Ermon, S. Con- sistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion. In Int. Conf. Learn. Repr., 2024.
Luo, W., Hu, T., Zhang, S., Sun, J., Li, Z., and Zhang Z. Diff-Instruc:: A Universal Approach for Transferring Knowledge from Pre- Pre-Trained Diffusion Models. In Adv. Neural Inf. Proc. Syst, volume 36, 2024a.
10

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
Luo, W., Huang, Z., Geng. Z., Kolter, Z.,, and Qi, G.-J. One-step Diffusion Distillation through Score Implicit Matching, In Adv. Neural Inf. Proc. Syst., volume 37, pp. 115377-115409 2024b.
Song, and Dhariwal, Improved Techniques for Training Consisteney Models. In Int. Conf. Learn. Repr. 2024a.
Song, Y. and Dhariwal, P. Improved Techniques for Training Consisteney Models, Int. Conf. Learn. Repr., 2024b.
Mao, X., Li, Q. Xie, H., Lau, R. Y., Wang, Z., and Paul Smolley, S. Least Squares Generative Adversar- ial Networks, In Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 2794 2802, 2017.
Song, Y. and Ermon, S. Generative Modeling by Estimating Gradients of the Data Distribution. In Adv. Neural Inf. Proc. Syst., volume 32, 2019.
Song, Y., Garg, S., Shi, J., and Ermon, S. Sliced Score Matching: A Scalable Approach to Density and Score Estimation. In Proc. Conf. Uncertainty Artif. Intell., pp. 574-584 PMLR, 2020.
Meng, C., Rombach, R., Gao, R., Kingma, D., Ermon, S., Ho. J., and Salimans, T. On Distillation of Guided Diffusion Models. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 14297-14306, 2023.
Song, Y. Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based Generative Modeling through Stochastic Differential Equations. In Int. Conf. Learn. Repr., 2021b.
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spec- tral Normalization for Generative Adversarial Networks. In Int. Conf. Learn, Repr., 2018.
Nichol, A. Q. and Dhariwal, P. Improved Denoising Dif- fusion Probabilistic Models. In Proc. Int. Conf. Mach. Learn., pp.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency Models. In Proc. Int. Conf. Mach. Learn., volume 202, pp. 32211-32252 PMLR, 23-29 Jul 2023.
PMLR, 2021.
Nielsen, F. A Family of Statistical Symmetric Diver- gences based on Jensen's Inequality. arXiv preprint 1009 2010.
Vincent, P. Connection Between Score Matching and Denoising Autoencoders. Neural Comput., 23(7):1661- 1674, 2011.
Nowozin, S., Cseke, B., and Tomioka, R. f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization. In Adv. Neural Inf. Proc. Syst., volume 29, 2016.
Wang. Z., Zheng, H., He, P., Chen, W., and Zhou, M. Diffusion-GAN: Training GANs with Diffusion. In Int. Conf. Learn. Repr., 2023.
Xie, S., Xiao, Z., Kingma, D. P., Hou, T., Wu, Y. N.N., Murphy, P., Salimans, T., Poole, B., and Gao, EM Distil- lation for One-Step Diffusion Models. arXiv Preprint arXiv:2405.1693 2024.
Polyanskiy, Y. and Wu, Y. Lecture notes on information the- ory, 2019. URL http:///wwwwtttt. yale edu/-yw562/ teaching/itlectures,
Robbins, H. F. An Empirical Bayes Approach to Statistics. Proc. Berkeley Symp. Math. Stat. Probab., 1956.
Yin, T., Gharbi, M., Park, T., Zhang, R., Shechtman, E., Durand, F,, and Freeman, W. T. Improved Distribution Matching Distillation for Fast Image Synthesis. In Adv. Neural Inf. Proc. Syst., 2024a.
Salimans, T. and Ho, J. Progressive Distillation for Fast Sampling of Diffusion Models. In Int. Conf. Learn. Repr., 2022.
Yin, T., Gharbi, M., Zhang, R., Shechtman, E.,, Durand, F,, Freeman, W. T., and Park, T. One-Step Diffusion with Dis- tribution Matching Distillation. In Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit, 2024b.
Salimans, T., Mensink, T., Heek, J., and Hoogeboom, E. Multistep Distillation of Diffusion Models via Moment Matching. In Adv. Neural Inf. Proc. Syst., 2024.
Zhou, M., Zheng, H., Wang, 7.,, Yin, M., and Huang, H. Score Identity Distillation: Exponentially Fast Distilla- tion of Pretrained Diffusion Models for One-Step Gener- ation. In Proc. Inf. Conf. Mach. Learn., 2024.
Sauer, A., Schwarz, K., and Geiger, A. Stylegan-Xii XL: Scal- ing Stylegan to Large Diverse Datasets. In ACM SIG- GRAPH Conf. Proc., number 49, pp. 1-10, 2022.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep Unsupervised Learning using Nonequi- librium Thermodynamics. In Proc. Int. Conf. Mach. Learn., pp. 2256.2265. PMLR, 2015.
Song. J., Meng, C., and Ermon, S. Denoising Diffusson Implicit Models. Int. Conf. Learn. Repr. 2021a.

---

Scorc-of-Mixture Training: Training One-Step Generative Models Made Simple
Appendix
Deferred Statements and Proofs
12
A.I Proof of Proposition 3.1
12
A.2 Proof of Proposition 3.2
13
A.3 Deferred Statements.
13
B Detailed Discussions on Related Work
14
B.1 Diffusion Models
14
B.2 Diffusion Distillation
16
B.3
Consistency Models
16
Detailed Description of Score-of-Mixture Training and Distillation
17
C.1 Amortized Denoiser..
17
C.2 Score-of-Mixture Training
17
C.3 Score-of-Mixture Distillation
18
On GAN Training
20
D.I On the Non-Saturating Generative Loss
20
D.2
GAN-Type Regularization with a-JSD
21
D.3 On Discriminator Training with Mixture Score Matching Loss
21
E
More on Experiments and Additional Results
22
E.1
Training Configuration
22
E.2
Toy Swiss Roll
22
E.3
Image Interpolation
23
E,4
Additional Training Curves
25
E.5
Samples
A. Deferred Statements and Proofs
A.1. Proof of Proposition 3.1
Proof of Proposition 3.1. We can simplify the gradient of each term separately as follows:
(x(x)
(oD lap (1 a)go) = (x) Vo log
+ Eg(xs)
(ogo())(500(()) (x,) x=)=(x)
ap(x) (1 a (x)
VoDKL (plaap (1 a))o)) Ep(x) Vo log(ap(( (1 a)qo (x))]-
Here, note that the first expression, we invoke the chain rule: for some function fo: X + R, we have
Vofo(go(z)) = aVafo (x)) + (ogo(z)(V
Combining these two terms with the weights, we get the gradient the a-skew JSD:
V VoD (go.)) V DeDKa + (1 - a)ge) + 1 a
V (xlllaa (1 ox)o)
a
a Eq(x)
(= - soja (x)) (x)
Eoptx)+(-oooo Vo log(ap(x) + +1 Volg(appxx (x))
a(1 a)
1
+ E Ex((x)
Vo log
a
Ex(z)
Vogo (z).0
so;a (x)) x=))
Here, we use the assumption that E (x) Vo log qo (x)) 0.
12

---

Scorc-of-Mixture Training: Training One-Step Generative Models Made Simple
A.2. Proof of Proposition 3.2
Proof of Proposition 3.2. We can write the objective ((15a))
((bba))= = / (ap((x) + +1 (1- (x)) ([==)))())]ssssss==]]] (2(o))2((((((()))))) (x) (x; (x; dx +
(op(x)) (x)=aa)(x)) 50 ap(x)sp(x) + (1 (1-0))00 qo JS (x)
dx + C²
ap(x) (1 2. (x)
Hence, it is clear that the global minimizer should be
Sy (x: ap(x)ss(()) + (1 a (x (x) (x)
ap(x) + (1 a 20 (x)
aVx((x) + (1 - a:) (x
ap(x) + (1 (x)
Vx(ap(x) +(1 aa (x))
ap(x) + - ex)o(())
Vxlog(ap(x) + (1 -x)))(xx):
A.3. Deferred Statements
Proposition A.1. Let a E [0, 1] be fixed and o, be some fixed noise level. Then, the minimizer of the objective function
Sacore (1:: a,,) = Ep(x)((e) (asst)))llssssx (x; + e/o, + (1 -a) Exooxittelllssxxxxx c/o,
(16)
satisfies
Sgx (x,; = So,n,(((x))
Proof.
We can write the objective C(1; a, as
Licore (v: a,
(x,;,,)) (x;;;)) // (op(x,) + (1
a. _ dx dx
This is standard minimum mean square estimation (MMSE) problem for which the global minimizer is the conditional mean.
8.- ax,
E Eep.+ [<|x;]
0,
Eope+(1-a)go, [xt - x x
07
1
1
xx
+
Eapct(1-a)quu. [x|x]]
of
Vx, log(ap((x)) + ax)o
Here we use that x4 = x + o&E and make the connection to the marginal score in the last line using Tweedie's 's formula (Rob- bins, 1956).
Proposition A.2. Let [0, 1), s,(x) be the data score, Syo (x) be the score of the generated samples. Then, the score of the mixture distribution can be expressed
$g.a (x) Do:a (x)s,(x)) + (1 -osc Dosa(()))s (x),
(17)
where
Do (x) = o (10g [x(x)
p(x)
log
(18)
1 a
13

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
Proof. The amortized score can be expressed as
vx log(ap(x) + (1 - (x)) = Vx(ap(x) + (1 (x))
ap(x) + (1 - (1-=o))00x))
ap(x)
(1 a))o(x)
ap(x) + (x) Vx logp(xx) + ap(x) + (1 ())
Vx log (x())
= = + (1 D(x; ((x:a)) log (x)).
We now simplify the scaling factor
D(x; a) ap(x) + (1 - (x) = (log p(x)
op(xx
gx(x) + log a
1-a²
B. Detailed Discussions on Related Work
B.1. Diffusion Models
Prior Work. Sohl-Dickstein et al. (2015) first introduced diffusion prohabilistic models (DPMs) as deep variational autoencoders (Kingma, 2014) based on the principles of thermodynamic diffuusinn with a Markov-chain variational posterior that maximizes the evidence lower bound (ELBO). Several years later, Ho et (2020) re-introduced DPMs (DDPMs) with modern neural network architectures and a simplified loss function that set new state-of-the-arl in image generation. Since then, numerous connections to existing literature in statistics, information theory and stochastic differential equations (SDEs) have helped bolster the quality of these models. For example, Song & Ermon (2019) illustrate the equivalence between DDPMs and DSM at multiple noise levels, thus bridging the areas of diffusion-basee models. Subsequently, Song et al. (2021b) showed that in continuous time, DPMs can be appropriately interpreted as solving for the reverse of a noising process that evolves as an SDE while Kingma et al. (2021) demonstrated that continuous -imee DPMs can interpreted as VAFs and that the variational lower bound is invariant to the noise schedule except for endpoints, thus bolstering its density estimation capabilities. Following the latter discovery, Kong et al. (2023) show that DPMs can in-fact be used for exact likelihood computation by leveraging techniques from information theory. To further improve DPMs, extensive research has gone into the choice of noise schedules, network architectures and loss functions (Nichol & Dhariwal, 2021; Hoogeboom et al., 2023; Karras et al., 2022a; Kingma & Gao, 2024). Many tangentially discovered frameworks such as rectified flows (Liu et al., 2023) and conditional normalizing flows trained with Gaussian conditional flow matching (Lipman et al., 2023), are also particular instances of (Gaussian) diffusion models with specialized noise schedules and weighted loss functions, as show in (Kingma & Gao, 2024).
models and score- base-
Formulation. We take the following unified view in our definition of DPMs inspired by (Kingma & Gao, 2024) and (Karras al., 2022a). Let p(x) be the data distribution and let X()) define variance exploding noise schedule with distribution p(t) where ~ (0,0 1). Under this noise schedule we can define a noisy version of x at noise level or
X = x + OrE where e~ (O.
(19)
Given noisy samples of data, the diffusion objective can be reduced a weighted denoising objective,
(DPM(EE) Ep(e)(())(()) [w(t)]le ea (x (x]:
(20)
where w() is positive scalar-valued weighting function. Note that for the forward process defined in Eq. (19), the conditional score is s(x:/x) = -e/o,. Thus, Eq. (20) can be interpreted as weighted denoising score matching loss (Vincent, 2011) over multiple noise levels,
(DPM(EE) 1 Ep(0p(x)p(xx)))) wd (o(c0)((xx.)
Eo(x(it) Ot
(21)
where (() oiw(() and the marginal score estimator is Su (x () ;= Ex 1)/or.
Sampling. It is often beneficial to view DPMs as SDEs (Song et al., 2021b) where the forward process can be expressed as dx; = f(x,, t)dt + g())dw,
14

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
where is standard Wiener process and Xo x. The time reversal of this process (i.e., the generative process) is known to follow the reverse SDE,
dx, = t) (t) log px:)] dt + 9(t)dw.
Note that in practice Vx, p(p) would be estimated by the score function sy (xx;t) t) from a variant of DSM in Eq. (21).
Sampling can be simulated through techniques such as annealed Langevin dynamics or ancestral sampling (Song et al., 2021b). While the above reverse SDE is stochastic in nature, there also exists deterministic process known as the probability flow ODE that satisfies the same intermediate marginal distributions,
dx, F(xx,)) 1 (1) p(x)) dt.
(22)
The benefit of the ODE formulation is that it can discretized more coarsely and hence sampling can done in fewer timesteps. Furthermore, sampling is possible by plugging in the updates from Eq. (22) into black-box ODE solvers, e.g., the Heun 2nd order solver (Karras et al., 2022a). Sampling can be sped even further if Eq. (22) can be solved exactly. Lu et al. (2022a) show that the exact solution to Eq. (22) at timestep / given an initial value at timestep 8 < l is,
/ Cu (Xui do..
x = x +
(23)
Various samplers can be derived by approximating the exponentially weighted integral in different ways. For example, the widely used DDIM sampler (Song al., 2021a) is an example of first-order Taylor expansion of the integral term. At the core of all these algorithms is a score estimator/denoiser,, which if learned accurately could improve the quality of samples produced.
EDM Diffusion Architecture. The EDM preconditioning diffusion model utilizes a base DDPM++ architecture from (Song et al., 2021b) for CIFAR-I0 and the ADM architecture (Nichol & Dhariwal, 2021) for ImageNet 64 x 64. The EDM model uses noise schedule that defined as
log or (M(-1.2,1122))
(24)
Rather than regressing against the unscaled additive noise as in DSM, EDM regresses against the original sample expressed in the following form,
Tx ddaia g:
X2
(25)
of
Vo + data
data
where data = 0.5. To this end, EDM is parametrized with denoising neural network,
data (x = of + o
Ot Odata
Xx +
Vo? + o Ba (Xei 1),
(26)
which is trained by minimizing
min Extx))(ccn(e) fo(x #
where
0+ Ty
WEDM( (1)
(27)
Vo? + o
This is equivalent to estimating g by minimizing the objective,
(EDM(go) = Ep(x)((c)p() [llg go (xx:t) [12]
(28)
Using Eq. (24) and Eq. (25) we can show that,
Vož + J
o data
g
X,
(29)
σ oiV of
0 data
Vot
o data
Øt
c +
X1-
(30)
ddtta
v of + data ddata
Therefore, in terms of Eq. (20) the EDM objective boils down to the unified diffuusion objective with weighting function,
w(t) = of data
(31)
0 data
15

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
B.2. Diffusion Distillation
Achieving state-of-the-art generation results on CIFAR-10 and ImageNet 64 x 64 using Heun 2nd order sampler with the EDM architecture requires 35 and 512 function evaluations (FEs) respectively. The goal of diffusion distillation is to distill teacher model into a student model that can achieve high quality signal generation with few FEs.
The earliest works on distillation such as progressive distillation (Salimans & Ho, 2022) and knowledge distillation (Huang et al., 2023) train a student diffusion model with drastically reduced sampling budget to match the performance of a teacher model that is simulated in reverse. For example, given a teacher diffusion model parametrized as a denoiser fo and a noisy (k) is constructed by running the teacher model for l steps in reverse. The student denoiser fo
sample X,, "clean" target x is then optimized by minimizing the loss,
(8) [|']. (()) (())((x)p(() (ttpecaappppp(((((ll((oC (x(( -X X
Knowledge distillation on the other hand conditions the student model on intermediate features from the teacher diffusion model so as to regularize the learned weights more effectively and retain knowledge from the teacher model. These methods are expensive as it requires either simulating multiple steps of a teacher diffusion model or additionally probing it for feature extraction.
More recently& a class of new diffusion distillation techniques grounded in reverse KL divergence minimization have gained popularity as discussed in Sec. 2. Diff-Instruct (Luo et al., 2024a), DMD (Yin al., 2024b) and DMD2 (Yin et al., 2024a) all train one-step generator go mapping noise z _N I) to generated samples by updating the generator in the direction of minimizing the reverse KLD,
V V WE (9e (p) = (ooe)n (x:) - (xxx)xx((()))))) (x)) (4o
where (x,) Vx, pxz) and Syo (x,) Vx log (x Assuming that the score model was learned using a parametrization similar to EDM, DMD scales the gradient and uses Tweedie's formula (Robbins, 1956) express it in terms of pretrained denoiser fo and denoiser for the fake samples fup
V VDLD (0) = E (g()))((((e))
(w [WDMD x, () (z)g(z) (L) (x;;)) (xx; (2))
where an adaptive weight is used to ensure that the scale of the gradient roughly uniform across noise levels,
WaMD (x,,
(32)
XX (x
To mitigate mode collapse and enhance sample diversity, DMD employs an ODE-based regularizer by simulating the pretrained diffusion model in reverse. This process generates noise-image pairs, which are then used to further supervise the generator's 's training. However, collecting this dataset becomes prohibitively expensive for high-dimensional samples. To address this limitation, DMD2 introduces GAN-based regularizer, which effectively minimizes the Jensen Jensen-Shannon divergence alongside the reverse KLD, or variant of the forward KLD when implemented in non-saturating manner. For further details on GAN training, refer to Appendix D.
Several methods build upon the divergence minimization framework by introducing regularizers based on alternative statistical distance measures. For instance, Moment Matching Distillation (MMD) (Salimans et al., 2024), Score Identity Distillation (SiD) (Zhou et al., 2024), and Score Implicit Matching (SiM) (Luo al., 2024b) align the fake score model with the pretrained score model using a variant of the Fisher divergence:
(v)) = (x)((()())) (wo' (1) Go (x:: sg (xx: ))
Here sg stands for the stop gradient operator. Additionally, both SiD and SiM extend this approach to generator training by minimizing the Fisher divergence, which requires a computationally expensive gradient calculation through the entire score model. To address this, they employ statistical approximations to make these gradient computations more practical.
B.3. Consistency Models
Consistency models are a new class of generative models introduced by Song et al. (2023) that learn a consistency function between all points along the trajectory of the probability flow ODE of a reverse diffusion sampler. Concisely, given points
16

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
along one such trajectory, Xx, E (e, 1J, where X1
-N( I), the consistency function satisfies,
[x
i =6
f(x,,t)=
f(xx, s)
8 SE [c.1]
Given the boundary condition at the origin, the consistency function can be parametrized using neural network similar to EDM,
oam X,
(or -0e) datt
Lo (x(x,, 1) (0: o.)² data
go go(x;; ().
Vo + o
Given a noisy sample X = x + orE, E (0, I), first a single step of the probability flow ODE is simulated using the Euler sampler by running one step of sampling using Eq. (23),
X, = X + (t s)(Vx, logp(x;)
This can be computed using either a pretrained score model or via a single sample Monte-Carie estimate. In the latter setting, it is important that the timesteps and are very close to each other for the approximation to hold. consisteney distillation a pretrained score model s is available and single sampling step along the PF-ODE is simulated
x x
s)ts>(x(;)) 8).
Then the consistency function is learned by minimizing
(f) Ep(x)((e)ttt) (w(t)d(fo( (), sg(o At)]),
where d is some distance measure, wtt) is some positive weighting function and = - At, with At some fixed timestep difference. Song et al. (2023) initially proposed using the LPIPS distance but subsequent works (Song & Dhariwal, 2024a; Geng et al., 2025) have shown that similar performance can be achieved by using the &g distance or pseudo- Huber norm.
Unlike distillation techniques, consistency models can also be trained from scratch. Assume that 8 = L - ōt, 51 + 0. Then, the sampling step can be approximated using Tweedie's formula (Robbins, 1956),
Xg ~ X + (t 8) X X,
t
x + 8c.
Thus, the consistency function can now be learned by minimizing,
(0) = Eptx)g(e)pte)(u
+ t); + (f - -5t));t----- - -t)))},
Consistency distillation still lags behind distillation methods based on reverse KI. minimization, but consistency training often demonstrates more impressive results. However, consistency training is still inherently unstable and requires careful design of both the noise schedule due to limiting nature of ôt and distance measure (Song & Dhariwal, 2024a; Geng et al., 2025). Stabilizing and making this objective simpler is the focus of lot of current research in the area.
C. Detailed Description of Score-of-Mixture Training and Distillation
C.1. Amortized Denoiser
Modern diffusion architectures such as the EDM architecture (Karras et al., 2022b) are specially designed for denoising purposes (see Appendix B). Hence, in practice we choose to train an amortized demoiser, f,y (x;; 1) R Eopc+(1-a)go.. [x xt). upon which the amortized score can be recovered using Tweedie's s formula (Robbins, 1956),
1
Suy (x:; a, -. (x::
The mixture score matching loss in Eq. (8) can be expressed with this denoiser as
denoise (1,; = a E El
x|| + (1 (1-a)) ax) (x x|12).
C.2. Score-of-Mixture Training
Here, we present a pseudocode for Score-of-Mixtuee Training (SMT). See Algorithm 1.
17

---

Scorc-of-Mixture Training: Training One-Step Generative Models Made Simple
Algorithm Score-of-Mixture Training
Inputs: Randomly initialized generator go, amortized model Sy, discriminator Ey., real dataset D, score training sub-iterations 5, learning rates (7gan Misore), GAN regularizer weights (score μ., gen A)
Pretraining: Train ge with DSM using D for each pretraining iteration do DD and add noise x, x 6 (0,, Compute DSM loss (0) (see Sec. 2)
Sample mini-batch
Update parameters: 0 < 0 7/SM V (0)
end for
Training: Alternating updates of ge and Sy for each training iteration do
Generator Training: Freeze S.
Sample mini-batch of fake samples xake g)) (O,
Sample p(t) and a as described in Sec. 3.5
Compute weighted generator gradient 7 (0; az, 1) from Eq. (9)
Compute GAN regularizer loss cla from Eq. (11)
Update parameters:
- 0
Tlgen (pta)pto([(xx? fake aa,t) (0;aa, + AV 2o.00 (0)
Amortized Score Training: Freeze go
for each sub-iteration do
Sample mini-batch of real samples real D
Sample ~ p(() and a
Compute score matching loss (yo a, t) from Eq. (8)
Compute non-saturated discriminator loss Liise(bb) (see Appendix D) Update parameters:
1 x- 2/- Tsccore V (Ep(o)))))) (euore (1/: a, t)] + (LCiss(())
end for end for
Return: Trained model parameters 8, +b
C.3. Score-of-Mixture Distillation
We present overview figure and pseudocode of Score-of-Mixtum Distillation in Fig. 4 and Alg. 2. We highlight the central differences in the distillation training from the training from scratch in Alg. 1.
The pretrained score model on the real data is available during distillation. Thus, rather than defining an amortized score model that takes conditioning variable for a, we show that we only need learn score model on the fake samples make the objective simpler (see Proposition A.2). This is more effective as learning the score of multiple interpolated distributions is generally more challenging task.
We can initialize the fake score and generator with the weights from the pretrained model and don't need run separate pretraining phase in training from scratch.
Notice that in amortized score training in the distillation setting there is no explicit GAN loss to learn the discriminator in Algorithm Learning the discriminator implicit to our proposed parametrization and once learned we can use this to minimize the skewed divergence via nottratua GAN regularizer for generator training.
18

---

Scorc-of-Mixture Training: Training One-Step Generative Models Made Simple
Amortized Score Model
Generator
W. ~ (00,
Fake
go(z)
Eq. (12)
Noise
Sp
oz at:
Generator Update (Eq. (15))
W, ~ (I)
x
S (xcfote cr 8) (xe 3 a,
Real Dataset
tcal
Real Image
Ve (dia.
Score Model Update (Eq. (14))
Figure 4. Overview of ofoore-of-Mixture Distillation. Top: To update the generator weights, the fake image is diffused at noise level t and then used compute the gradient æ-skew divergence with explicitly parametrized amortized score model using Eq. (15). Bottom: Amortized score model training involves computing the score of the mixture distribution on both fake and real samples diffused with noise level and updating weights using gradient Eq. (14).
Algorithm 2 Score-of Mixtrre Distillation
Inputs: Randomly initialized generator ga, fake score model ffake, discriminator tys pretrained score model Sp, real dataset D, score training sub-iterations 5, learning rates (Ngen Mscore), GAN generator regularizer weight
Initialization Initialize sti and go with weights from Sp
Training: Alternating updates of go and Sg. for each training iteration do
Generator Training: Freeze sfet and bx
Sample mini-batch of fake samples fake
I0,
Sample p(() and a as described in Sec. 3.5
Compute generator gradient (0; a, t) from Eq. (15)
Compute GAN regularizer loss (a.t) from Eq. (11)
Update parameters:
- 0 Tgen Ep(a) he (0:aa) + XV
Vocla ((,t)
(0)
Amortized Score Training: Freeze go
for each sub-iteraa -ebbiteration do Sample mini-batch of real samples real D
Sample p(t) and
Compute score matching loss using explicit parametrization Cext (v); ar, f) from Eq. (14)
Update parameters:
4b <- 1/ - Tuucore V Ep(a))(e) Lext (v); a. t)]
end for end for
Return: Trained model parameters 0, x
19

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
D. On GAN Training
The vanilla GAN
min max(Epto) flog (x)( + (log(1) De (x))]} w
Breaking down, the discriminator training is
min Ep(x) log Dx(x)) (x)))))og(( (x)));
and the generator training
min E (x)))))lg(( - D=(x))]-
Note that
(x)
(x(() 1 (x)
(x)
Further, the sigmoid logit (x)x), ie., ( = (x)x( is log (x). Note that the optimal discriminator for each (*)x)
[xx r* (x)
p(x)
p(x)
(x(x)*
The non-saturating versions is training generator based
min Equ (x)) [log (x = Ego(x) log (x(x)) p(x) + 1)
The StyleGAN uses the non-saturating loss:
min Ep(x) logre (x)) + (sp(o))s(((g
min (x) (ss(=ss(oppp log (x))).
Note that sp(y) = log(1 + e²). Hence, note that
sp( lp(-log (x)) = log + (x))
(x)) = - log DΦ (x), sp(logry(xx) (x)) log(1 (x)) = - -log(1 (x)).
D.1. On the Non-Saturating Generative Loss
The original, saturating version of the generator objective is
min Equ(x) ro (x)) Ex)( log(1 + (x))],,
whose gradient is
r (go(z))
Ve + re (g)(2)) Eg(x)
[ 1 (go (ogo(z)
Note that the plug-in reverse KL-divergence loss is
min Ego (x) log (x)).
Compared to this, the non saturatii loss has the additional :
min E (x) [sp( log (x)) = E (l) + rx(xx)
@
This seems to help prevent vanishing gradients. Consider
min (x) log(r + (=(x) -1)] = (lgg(T + (g)
20

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
If it boils down the plug-in reverse KL divergence, and gradient with respect 0, get
recovers the non-saturating loss. If consider
(z)( V Ex))) llog(T tp (p(g)()))) En(e) ro (go(z)) + Tr (go(2)) Vogo(()
p(x) is supposed to be small for generated samples x = =g((z). Therefore, the plug-in loss with Here, recall that (x(x) R [x)x]
0 inherently prone to vanishing gradient.
D.2. GAN-Type Regularization with a-JSD
To train discriminator for our GAN-type regularizaion,,,, opt to modified GAN discriminator objective defined as a)] (1 a) (x))) - D a))l
min -a Ep(x) (x;
Similar to the vanilla GAN, the optimal discriminator for each 0 and in this case is
ap(xx
(x(x;)) ap((x) + (1 (x)
Then,the a-JSD can be approximated as
D& (x)
1 D (x; a)
D ja) ISD p)
R
Ep(x)
log
+ Exu(x)
log
1
1
Hence, with this approximation, the generator update can be done via
1 E (x))
log - (x
min
In practice, we can use weighted non non- saturattig version of the loss well.
log Dx a)
= min (x)) sp (-lo(x) log 1 a
a =2)
) log
min Eg (x)
where (x) = log p(x] (x))
D.3. On Discriminator Training with Mixture Score Matching Loss
In Sec. 4, we plugged in the explicit parameterization
exp (x; a)) = (x; + (1 Dr as)
to the mixture regression loss in Eq. (7), to train the fake score and the discriminator simultaneously. If we consider an ideal scenario where we have the perfect score models for both p and y, then all we need to train the discriminator andthe mixture regression objective can be interpreted discriminator objective. Here we reveal its connection an instance of f-GAN discriminator objective.
Let (x) and (x) be the underlying score functions for p and g, respectively. Then, the explicit parameterization becomes
(x: a) = (x; + (1 - (x(xx)))s,
and the mixture regression objective becomes only a function of the discriminator, i.c.,
L(v: = a [l(x) (x:
(x)) a) [1o (x: a) [x)
= a [( - (x; (1s, - sxx) + (1 a) Eg a)) [Is)(()) -s (x)]
= /(op()) -appx)(11DDxxxx))) (x(x:: (xx - -x((x)1
21

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
Here, we note that the term [xs,(x) sy (x)] is common in both expectation, and be safely dropped to train the discriminator, which leads simplified objective
C (1r; a) = a Ep(x) - (x(x a) a) Ex(x)
= [app(x)(1 (x(xx))) ) + - (x) a) dx.
We note that this is equivalent to the discriminator objective induced by the following f fdivergennce
pxx)g(x) (= x (1-agg(x) dx : DD-LC(P q).
where (r) = (1=a) a1-(1-a) is a convex function over x)) for E (0, 1). For a = 2. this divergence becomes symmetric in and and is known as the Le Cam distance (Le Cam, 2012, 47) in the literature (Polyanskiy & Wu, 2019). We thus call the general divergence for E (0, 1) the a-Le Cam distance. In the GAN literature, this is known as the LSGAN objective (Mao et al., 2017).
As we revealed, our discriminator training distillation can also be done separately using the a-Le Cam-distance-base objective. However, we conjecture that our score- seareeregresson-beee based end-to-end objective may have benefit, as our primary goal of discriminator training is it in the generator update in the form of approximate score of mixture. We leave the further exploration of such alternative methods future work.
E. More on Experiments and Additional Results
We present some additional experiments and results in this section. We first provide a more detailed training configuration for our experiments Sec. 5 and then evaluate our proposed method on synthetic swiss-roll dataset in Appendix F.2. Finally, we present some samples generated from Score-of-Mixture Training and Score-of-Mixture Distillation Figs. 9-12.
E.1. Training Configuration
We summarize the detailed training configuration in Table 3.
Table 3. Hyperparameters used for training one-step generators with Sfore-of-Mixtune Training and Distillation.
Hyperparameter
CIFAR-10
ImageNet 64
64
Scratch
Distillation
Scratch
Distillation
Generator learning
le-4
Se-5
Se-6
2e-6
Score learning rate
5c-4
5c-5
5e-5
2c-6
Score lcarning rate decay
cosine
None
cosine
None
Batch size
280
280
280
280
Diffusion pretraining steps
15k
N/A
40k
N/A
Training iterations
150k
150k
200k
200k
Score dropout probability
0.13
0.00
0.00
0.00
Number of GPUs
2 x A100
4x A100
7x A100
7x A100
E.2. Toy Swiss Roll
We tested our proposed framework and ablated various design choices on a synthetic swiss roll dataset. We followed the dataset setup by Che et (2020). We trained models with SMT and SMD and compared this against an amortized version of reverse KL minimization with DMD weighting (a {0, 1}) similar to the ablations in Sec. 5.3. Additionally we compared against non-score-based baselines including the vanilla GAN and Diffusion-Gaai (Wang et al., 2023).
Across all experiments, we use the same generator architecture a two-layer MI. with a hidden dimension of 128 and leaky ReLU nonlinearity. We train all models for 200k steps on a single NVIDIA 3090 GPU with a batch size of 256. All score-based methods leverage learning rate of le-5 for the generator and le-4 for the amortized score (and discriminator when applicable) whereas the GAN-based methods use a learning rate of 1c-4 for both generator and discriminator. We use the AdamW W optimizer without any learning rate schedulers.
22

---

Scorc-of-Mixture Training: Training One-Step Generative Models Made Simple
(a) Ground Truth
(b) GAN
Diffusion-GAA
(d) SMT (with GAN)
(c) SMT (No GAN)
(I SMD
(g) Amortized Reverse KLD
Figure 5. Samples produced by generators trained using different methods. All figures are created using 10,000 samples from the respective generator.
The samples produced are shown in Fig. 5. Notice how the GAN is unable perfectly cover the entire continuous mode of the swiss roll. The Diffusion-GAN of low density. We found the latter to be sensitive to the chosen noise levels in comparison to the methods based on updating the generator using the score.
multi-noise level extension of the GAN, covers the mode but also samples from areas
Our results for training from scratch and distillation are presented in Fig. 5d-f. All three methods successfully capture the modes of the underlying distribution. While the impact of the GAN regularizer is less pronounced than in our high- dimensional experiments, we observe that enabling it (as in Fig. 5d) reduces the number of samples in low-density regions compared to Fig. 5e. The distillation results in Fig. 5f appear slightly noisy, likely due the quality of the pre-trained score model. This highlights the advantage of training from scratch, as it avoids amplifying existing estimation errors in the pre-trained model.
In Fig. 5g, we present an ablation result of the s-mmpreer, by using only a = 1. This corresponds to minimizing reverse KLD as DMD and DMD2. Unlike in the eigh-dimensional setting presented in Fig. 2b, we observe that using the single produces visually plausible samples in this low-dimensional synthetic example. However, our method in Fig. 5d (i.e.. SMT with GAN regularizer) produces fewer spurious samples compared to Fig. 5g, suggesting the benefit of multiple values.
E.3. Image Interpolation
The one-step generator is mapping between the representation space, which in this case is the space of standard multivariate Gaussian variables, and the space of images. Thus, to study the representation space followed the approached in (Song et al., 2023) and spherically interpolated between two randomly chosen noise instances Zo and Z1,
sin 3)bb)
(inh) sin
23
2o +
21,
(in(
where 3 E [0, and ei = arccos
x
After interpolating between these two points, the interpolated image can be
[Rolla R 22
obtained as X = go (2,e) as shown in Figures 6 and 7.
23

---

Scorc-of-Mixture Training: Training One-Step Generative Models Made Simple
Figure 6. Visualizing the latent space of the one-step generator trained on ImageNet 64 x 64 by interpolating between two noise inputs. The leftmost and rightmost image each row correspond to synthesized images with the same class and different noises 2p and 21. All intermediate images are obtained by applying the generator on spherical interpolation between these noise instances, demonstrating the interpretable learned latent space the generator.
Figure 7. Visualizing the latent space of the one-step generator trained on the CIFAR-10 10 dataset by interpolating between two noise inputs. The leftmost and rightmost image in each row correspond to different noises 2u and Zı. All intermediate images are obtained by applying the generator on spherical interpolation between these noise instances, demonstrating the interpretable learned latent space of the generator.
24

---

Score-of-Mixture Training: Training One-Step Generative Models Made Simple
Amortized Score Estimation Loss
Pre-autograd Generator Loss
0.3
Loss
Loss
0.5
0.2
0.4
0.1
0.3
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
training steps
x10*
training steps
x10*
(a)
(b)
Amortized Score Grad Norm
Generator Grad Norm
10
Norm
10
Norm
10-1
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
training steps
x10*
training steps
x10*
(c)
(d)
Figure SMT training curves on ImageNet 64x64. Plotted dark blue the running average trajectory the different metrics. Both the loss curves in (a) and (b) are smooth and do not explode. This is further supported by the curves of the respective gradient norms in (c) clear underlying optimization stable and no gradient explosion occurs. actual generator loss involved the
(d), where gradient of the generator multiplied by the difference of scores. During implementation we let autograd take care of this gradient and the loss that calculated pre-aum pre-autograd shown (b).
E.4. Additional Training Curves
To further demonstrate the stable training dynamics of SMT, we provide additional training curves in Figure 8 showing consistent decrease in both loss and gradient norm. Notably, at no point do we observe any spikes or instability in either metric. Since we compute the generator's s gradient directly, the plotted loss in Figure 8b corresponds to the proxy objective prior to applying automatic differentiation.
E.5. Samples
We present some image samples generated by SMT and SMD in Figs. 9-12.
25

---

Scorc-of-Mixture Training: Training One-Step Generative Models Made Simple
Figure 9, One-step generated samples from SMT on CIFAR- 10 (unconditional).
26

---

Scorc-of-Mixture Training: Training One-Step Generative Models Made Simple
2
Figure 10. One-step generated samples from SMD on CIFAR- (unconditional).
27

---

Scorc-of-Mixture Training: Training One-Step Generative Models Made Simple
Figure II. One-step generated samples from SMT on ImageNet 64x64 (conditional..
28

---

Scorc-of-Mixture Training: Training One-Step Generative Models Made Simple
Figure 12, One-step generated samples from SMD on ImageNet 64x64 (conditional).
29


Reviews:
[{"review_id": "MuVLAsip7q", "reviewer": "Reviewer_vjzy", "summary": "The paper presents the ScoreMix, a new type of one-step generative model, trained using the $\\alpha$-JSD from $f$-divergence. ScoreMix can be trained from scratch and used for distillation. It achieves SOTA performance in the 1-NFE regime. The paper grounds the theoretical approach and performs extensive experiments/analyses to showcase its significance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the effort in reviewing our manuscript and providing constructive comments. We will incorporate all the feedback in our revision to improve the manuscript.\n\n### On Weaknesses\n\n* `Missing reference and baselines`: We thank the reviewer for pointing out the missing references and baselines (rectified flow models and their variants). We will certainly add and discuss these in our revision to better clarify and contextualize our contribution.  \n* `Efficiency comparisons`: We thank for the suggestion. We will include GPU hours for our experiments in the revision to highlight the efficiency of our framework. To obtain the best FID result for ImageNet reported in Table 2, for example, it took 80 hours using 7 × A100 GPUs (200k iterations with an overall batch size of 280). As a comparison, both consistency training (CT) and improved consistency training (iCT) used 800k iterations with a much larger batch size of 2048. They do not provide the number of GPU hours, making a direct comparison difficult. We do note that ECT is able to *finetune* a pre-trained diffusion model on 4 × H100 GPUs within 8.5 hours. However, this consistency model is initialized with a pre-trained EDM2 diffusion model that was trained on 32 × A100 GPUs with a batch size of 2048 for a total of 2500 million images. Again, the EDM2 paper does not report the number of GPU hours, but its predecessor, EDM, reports a total training time of 2 weeks with a similar computational budget on ImageNet 64×64 — which alone exceeds our entire training-from-scratch effort. This shows that our proposed method can train a one-step model from scratch with a far smaller computational budget in terms of number of GPUs, training iterations, and overall batch size. Moreover, the two-model update is not more expensive than existing SOTA methods. \n* `Code release`: As promised, we will release our codebase and model weights upon acceptance for reproducibility.  \n* `Typo`: We appreciate the reviewer’s careful reading of our paper. We will correct the typos and thoroughly check the manuscript.\n\n### Answers to Questions\n\n* `Why α=0 for 25% of the time during the score training?`: This is due to the nature of the gradient of the skew JSD in Eq. (4). In that expression, the score of the mixture distribution with $\\alpha=0$ is *always* used, which implies its particular importance and the need for accurate estimation. We will clarify this point in our revision to avoid any confusion.  \n* `Extension to multi-step inference?`: This is a great question, and we agree that having a multi-step refinement feature would be of great interest. One potential direction is to develop a hybrid method that combines consistency models with our approach. We leave this for future work.  \n* `Invertibility?`: Our current manuscript focuses on developing a high-quality, efficient sampling scheme. Whether our model can be inverted is indeed an interesting question from a representation learning perspective, and we leave this for future investigation.  \n* `Linear interpolation?`: Thanks for the helpful suggestion. We conducted the interpolation experiment for CIFAR10 and ImageNet and uploaded the results at this [anonymized link](https://docs.google.com/document/d/e/2PACX-1vTw0koYOvxXorKuyTBw3C4FPnB_tX6IE6sYZRUnuFhQfy2ixqVt5fDk5GjnDFlcInYUBy83hNKe10gT/pub); see Figures B and C. Similar to GANs and consistency models, we found that “spherical interpolation” leads to natural interpolations.", "claims_and_evidence": "All the claims made in this paper are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Image generation is a testbed for evaluating generative models (especially for diffusion/flow models). The authors follow standard evaluation criteria that evaluate the method on the community-defined datasets (ImageNet64 & CIFAR10).", "theoretical_claims": "The reviewer has not carefully verified derivations line-by-line from the appendix. However, the reviewer believes the idea is straightforward and intuitively makes sense.", "experimental_designs_or_analyses": "Yes, the soundness of the following experimental designs by the author is verified and looks good.", "relation_to_broader_scientific_literature": "This paper presents the new type of generative model that is 1NFE and seems easy to train and could be of interest to various communities such as audio, applied, and AI4Science.", "essential_references_not_discussed": "Rectified flow models and their variations also fall under the 1NFE requirement. Hence, for a better comprehensive understanding, they should be included in Table 2. \n\n[1] \"Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow\"\n[2] \"Improving the Training of Rectified Flows\"\n[3] \"One Step Diffusion via Shortcut Models\"", "other_strengths_and_weaknesses": "Strengths:\n- The paper has been written concisely and clearly.\n- ScoreMix achieves the SOTA performance on the ImageNet64 and CIFAR10 that can be trained from scratch and contains the ablation on $\\alpha$ parameter that clearly showcases the advantage of the design choices made for ScoreMix.\n\nWeaknesses:\n\nOverall, the paper is in good shape, except for some minor improvements that could be made.\n- **Missing baselines:** Rectified Flow models and their variants, such as RF++ and ShortCut models, could be included as part of the main results in Table 2.\n- **Efficiency comparisons:** Although it is claimed that the training budget is smaller for ScoreMix, it is important to share the total GPU hours for each experiment and how much it does for at least some baselines. Otherwise, training the two models seems a slow and expensive process.\n- **Code release**: Authors claim to release the code to the public after the reviewing process; however, they have not even shared it as a part of supplementary material for reviewers. Hence, it is hard to verify the actual reproducibility. That said, I trust the authors will release it; hence, my current rating is also conditionally.", "other_comments_or_suggestions": "- Instead of Fig. 2b, many sentences (Lines 256, 387, 392, 409) contain the typo and mention Fig. 1a.", "questions_for_authors": "- Could authors elaborate more on why $\\alpha=0$ for 25% of the time during the score training (Lines 261-263)?\n- ScoreMix currently supports only the 1NFE approach. Can authors share some insights (if any) on how this could be extended to multi-step inference, as many downstream tasks such as editing and inverse problems might depend on it?\n- Is it possible to perform inversion at all?\n- What does the linear interpolation between two random input noises result in? Take two noises, then get interpolated noises in-between, and create a gif or image collage of corresponding generations. This could shed some light on the latent space of ScoreMix.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ncyXvXr57C", "reviewer": "Reviewer_nETY", "summary": "This paper proposes a generalization of the KL-minimization procedure for learning one-step generators from score-based models. The authors introduce an \"$\\alpha$-skew Jensen–Shannon divergence\", which interpolates between the KL divergence and the reversed-KL divergence. They propose two settings: one training from scratch, another one training with a pre-trained diffusion model. They add a GAN-based regularization strategy. Finally, they demonstrate with several experiments on image generation the interest of their method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s effort in reviewing our paper and constructive comments. We will address the raised concerns in our revision as follows.\n\n* `On FID evaluation and adversarial loss`: We appreciate the reviewer for the thoughtful comment on the limitation of FID evaluation and its interaction with adversarial losses (Stein et al., 2023). We will clearly explain these points in our revision. Here, we wish to clarify our standpoint.  \n  * While we acknowledge that FID is not a perfect metric as argued in the reference, it remains the most popular benchmark for measuring perceptual quality. Our ablation study shows that ScoreMix achieves stable and competitive performance even without the GAN regularization, indicating that the performance improvement is not solely due to adversarial losses.  \n  * Regarding fairness in comparison, we highlight that some recent distillation methods in Table 2 such as DMD2 use adversarial regularizer. Similar to DMD2, our GAN discriminator is built on top of the score network, with a few layers of MLP. This score-model-dependent design allows the entire model to enjoy training stability driven by denoising score matching, while the GAN discriminator loss only trains the small additional MLP. (For ImageNet, the generator has 296M params and the discriminator has 18M params.) However, for other distillation or training-from-scratch methods such as consistency training or distillation, it is nontrivial to implement the GAN discriminator assuring training stability. For example, the notable example of the consistency model framework differs conceptually from traditional distribution matching approaches. While a GAN regularizer fits within this framework, its implementation requires more than a small discriminator like ours. Designing a full discriminator network (similar to those in StyleGAN or its XL variants) would be necessary, and this could involve substantial hyperparameter tuning and careful initialization. Moreover, due to the inherent instability of the consistency training framework, introducing an adversarial loss could exacerbate these instabilities, given the well-known challenges of GAN training. Hence, investigating GAN regularization to an existing method such as consistency training is an interesting research direction, but it is beyond our current scope.  \n* `Comparison of the proposed GAN regularization with standard GAN regularization`: This will be an informative ablation study that can clarify the role of skew divergence in GAN regularizer. We will add an additional result for CIFAR10 with standard GAN regularization, i.e., only using $\\\\alpha=\\\\frac{1}{2}$ in the regularization.  \n* `Additional ablation with standard KL or reversed KL divergence minimization`: We appreciate the reviewer’s suggestion. We will add a result only using the reverse KL divergence for CIFAR10 in the revision. In the meantime, we performed an additional experiment with a toy dataset, as shared in the global response. \n* `Toy experiment on Swiss roll dataset`: We ran a toy experiment on 2D Swiss roll dataset, to demonstrate the effectiveness of our ScoreMix framework compared to the existing schemes for a simpler setting; see Figure A in this [anonymized link](https://docs.google.com/document/d/e/2PACX-1vTw0koYOvxXorKuyTBw3C4FPnB_tX6IE6sYZRUnuFhQfy2ixqVt5fDk5GjnDFlcInYUBy83hNKe10gT/pub). In particular, our results for training from scratch and distillation are presented in Figure A (d, f, g). All three methods successfully capture the modes of the underlying distribution. While the impact of the GAN regularizer is less pronounced than in our high-dimensional experiments, we observe that enabling it reduces the number of samples in low-density regions in (g). The distillation results in (d) appear slightly noisy, likely due to the quality of the pre-trained score model. This highlights the advantage of training from scratch, as it avoids amplifying existing estimation errors in the pre-trained model. We will add this experiment to Appendix in our revision.\n\nWe thank the reviewer again for their suggestions and comments, and will certainly incorporate all the points in our revision including additional ablation study results with CIFAR10. If these clarifications satisfactorily address the reviewer's concerns, we kindly ask if the reviewer would consider updating the score to reflect what we believe is a paper with noteworthy contributions to the community.", "claims_and_evidence": "The paper mainly claims that minimizing the proposed $\\alpha$-skew Jensen–Shannon divergence leads to better one-step generative models than minimizing the KL-divergence, which is a current practice in generative modelling. This is mainly evaluated in an ablation study on CIFAR-10 (Figure 1b), where the authors compare training with $\\alpha=\\{0,1\\}$ and with random $\\alpha$. I would also appreciate an ablation with standard KL or reversed KL divergence minimization in this Ablation study.", "methods_and_evaluation_criteria": "Method: The method is sound, well-formulated, and elegant. \n\nEvaluation criteria: The evaluation criteria is the FID on image generation benchmarks. The authors could consider relying on other type of metrics. Mainly, my biggest concern is the interaction between GAN losses and the FID. It has been shown that the FID is biased by adversarial losses (see \"Exposing flaws of generative model evaluation metrics\nand their unfair treatment of diffusion models\", Stein et al., 2023). However, the ablation study shows that the improvement in FID does not only come from adversarial losses.", "theoretical_claims": "The theoretical claims mostly extend known results to $\\alpha$-skew JSD. I did not check the proofs in detail.", "experimental_designs_or_analyses": "I checked the experimental design and analyses. As mentioned above, my biggest concern is with the use of the GAN regularization. Indeed, GAN regularization is not used in most other methods. It is thus a bit unfair to compare ScoreMix with other methods, since the other methods would also benefit from an adversarial loss. It could be interesting to include models trained without GAN regularization in Table 2. This would allow to assess the superiority of ScoreMix over other methods for one-step generator training.  \n\nMoreover, it would be interesting to compare the proposed GAN regularization with a standard GAN regularization to assess its interest. \n\nProvided improvements on these crucial points, I would be willing to raise my score to Accept.", "relation_to_broader_scientific_literature": "The proposed paper is extending a widely used method for training one-step generators (KL-divergence minimization with score-based models). It proposes the minimization of $\\alpha$-skew JSD, which interpolates between KL divergence and reversed KL divergence. This is new and original to me.", "essential_references_not_discussed": "Not that I know.", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No", "questions_for_authors": "See weaknesses raised above (in Claim and Evidence, and in Evaluation Criteria).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IUEdW4vjBf", "reviewer": "Reviewer_cyoT", "summary": "This paper proposes a framework for training one-step generative models, called ScoreMix. The proposed method is derived by minimizing the $\\alpha$-skew Jensen-Shannon Divergence ($\\alpha$-JSD) between the generated distribution $q_{\\theta}$ from an implicit generative model and the data distribution $p$ (or the generated distribution from the pretraining diffusion model in the distillation setting). The gradient of $\\alpha$-JSD can be computed from the score of the mixture distribution of $q_{\\theta}$ and $p$. Hence, training ScoreMix includes this mixture score network $s_{\\psi}$ training. ScoreMix demonstrates competitive performance on  ImageNet 64x64 and CIFAR-10.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the effort in reviewing our work and the helpful suggestions for improving the readability of our paper. \nBelow, we provide clarifications on the identified weaknesses and responses to the questions.\n\n### Clarifications on Weaknesses\n* `On stability of ScoreMix training`: We appreciate the reviewer’s suggestion to include training curves to further demonstrate the claimed stability of ScoreMix. While we included the FID curve over training iterations in Figure 2, we agree that plotting additional statistics would better illustrate training stability. In Figure D of this [anonymized link](https://docs.google.com/document/d/e/2PACX-1vTw0koYOvxXorKuyTBw3C4FPnB_tX6IE6sYZRUnuFhQfy2ixqVt5fDk5GjnDFlcInYUBy83hNKe10gT/pub), we have plotted an example training trajectory from our best ImageNet model, showing training losses and gradient norms. This further illustrates the stability during training. We will also include results for different parameter settings in the revision.\n* `On the role of the GAN regularizer`: We acknowledge that the GAN regularizer significantly helps accelerate convergence and improve FID. However, we note that we did not train the version without the GAN regularizer long enough to reach full convergence. To clarify whether the GAN regularizer is essential for achieving SOTA FID or primarily helps speed up convergence, we will conduct a training run on CIFAR10 without the GAN regularizer until convergence and report the result.\n* `On the stability and complexity of the GAN regularizer`: Similar to DMD2, our GAN discriminator is built on top of the score network, with only a few additional MLP layers. This score-model-dependent design allows the full model to benefit from the training stability provided by denoising score matching, while the GAN discriminator loss only trains the small auxiliary MLP. (For ImageNet, the generator has 296M parameters and the discriminator has 18M.) Thus, the discriminator represents a small fraction of the overall model size and has a negligible impact on training speed. As a result, our use of the GAN regularizer is both efficient and stable.\n\n### Answers to Questions\n* `Additional regularization using consistency between scores of mixtures?`: We note that the score of a mixture distribution can be expressed as a weighted sum of the scores of the true and fake distributions, as shown in Eq. (12), where the weight is determined by the density ratio and $\\alpha$. We leverage this relation in our distillation scenario, which is in the same spirit as the reviewer’s suggestion regarding consistency. However, for training from scratch, the relation between the scores for different $\\alpha$ values is more implicit. It would be very interesting if a similar consistency regularization could be achieved in this context, and we agree that this is a promising direction for future work.\n* `On expensive regularizers`: We apologize for any confusion caused by our insufficient explanation of the term \"expensive regularizers.\" We provide a clarification below and will address this point clearly in the revision.  \n  * One example of an expensive regularizer is *the regression loss* used in the DMD paper. To address mode collapse in the DMD framework without any regularization, the authors simulate the reverse process of a diffusion model and sample several thousand noise-image pairs to anchor the generator’s outputs. Each noise-image pair requires evaluating the diffusion denoiser 256 times for ImageNet 64×64, which is extremely costly in practice. Moreover, the cost of collecting this regression dataset scales poorly with dimensionality.  \n  * Another example of costly training is the approach in the CTM paper. To ensure consistency between random points along the PF ODE trajectory, the reverse diffusion sampler must be run for an arbitrary number of steps per minibatch, which also results in high computational cost. Here we note that we mistakenly referred to this expensive procedure as a \"regularization\" technique, and we will correct this terminology in the final revision.  \n  * In contrast, our method does not require sampling from a diffusion model. It relies only on score estimation of mixture distributions, making it significantly more efficient.\n\n---\n\nWe thank the reviewer again for their insightful questions and helpful feedback. We will incorporate all of the above points in our revision, along with the proposed additional ablation studies.", "claims_and_evidence": "The claims regarding the methodology are generally well-supported, such as Prop 3.1, 3.2, and Cor 4.1", "methods_and_evaluation_criteria": "One claimed advantage of ScoreMix in the manuscript is the stable training. However, this claim is not explicitly evaluated through empirical experimental results. Currently, it is supported only by assertions, such as the adoption of Multiple Noise Levels (Line 172) and denoising score matching techniques (Line 422)), without direct evidence. To substantiate this claim, the authors could consider various options, such as reporting the variance of scores or loss in Table 2 or Fig 2, or evaluating ScoreMix across diverse hyperparameters.", "theoretical_claims": "I reviewed the proofs of Prop 3.1, 3.2, A.1, and A.2 presented in Appendix A.", "experimental_designs_or_analyses": "I checked the validity of any experimental designs.", "relation_to_broader_scientific_literature": "**Contributions**\n\n- Novel training framework based on $\\alpha$-JSD\n- (Probably) stable training scheme motivated by DSM\n- Supports both scratch training and distillation from a pretrained diffusion model\n- Demonstrates competitive performance\n- In Prop 3.2, estimating the mixture score using the mixture for the score matching loss is interesting.\n- In Cor 4.1, the proposed method leads to a new way of training the discriminator in GAN.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strength**\n- This paper suggests a novel method for both training a one-step generator from scratch and distilling a pretrained diffusion model.\n\n- The suggested approach is well-motivated by theoretical results.\n\n- In Prop 3.2, estimating the mixture score using the mixture for the score matching loss is interesting.\n\n- In Cor 4.1, the proposed method leads to a new way of training the discriminator in GAN.\n\n**Weakness**\n- Without the GAN regularizer, the performance of ScoreMIX is not competitive in Fig 2(b), which limits the novelty of the proposed method in Sec 3.1-3.4.\n\n- The proposed method relies on GAN regularization in scratch training (Eq. 11), which might incur training instability as in GANs. Also, the distillation method requires three networks, i.e., generator, score network, and discriminator, which increases complexity.\n\n- The proposed method requires initialization, such as warm-up training.", "other_comments_or_suggestions": "- Can we derive the relationship between the score of the mixture distribution $s_{\\psi}$ between different values of $\\alpha$ and utilize it as an additional regularizer?\n- Could you clarify the 'expensive regularizers' in Line 189?\n\n**Typo**\n- Fig 1a in Line 381\n- Fig 1b in Line 392.", "questions_for_authors": "Questions are included in other sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for training one-step generative models based on the $\alpha$-skew Jensen-Shannon divergence. Reviewers unanimously praised the writing, novelty, and empirical results of the paper. The method is sound and the empirical results are strong, and I believe this paper might be impactful in the area. I thus recommend acceptance.
Decision: Accept (spotlight poster)