Unifying GANs and Score-Based Diffusion as Generative Particle Models
Authors: Jean-Yves Franceschi
Mike Gartrell
Ludovic Dos Santos
Thibaut Issenhuth
Emmanuel de Bezenac
Mickael Chen
Alain Rakotomamonjy
Venue: NeurIPS 2023 poster (2023)
Keywords: deep learning
generative models
GANs
generative adversarial networks
diffusion
score-based
gradient flows
TL;DR: We present a framework that unifies GANs and score-based diffusion models, allowing us to train a diffusion model that integrates a generator, and a GAN without a generator.

Abstract: Particle-based deep generative models, such as gradient flows and score-based diffusion models, have recently gained traction thanks to their striking performance. Their principle of displacing particle distributions using differential equations is conventionally seen as opposed to the previously widespread generative adversarial networks (GANs), which involve training a pushforward generator network. In this paper we challenge this interpretation, and propose a novel framework that unifies particle and adversarial generative models by framing generator training as a generalization of particle models. This suggests that a generator is an optional addition to any such generative model. Consequently, integrating a generator into a score-based diffusion model and training a GAN without a generator naturally emerge from our framework. We empirically test the viability of these original models as proofs of concepts of potential applications of our framework.

Unifying GANs and Score-Based Diffusion as Generative Particle Models
Jean-Yvee Franceschi Criteo AI Lab, Paris, France rreje..francechh@@eritoo.. com
Mike Gartrell Criteo AI Lab, Paris, France mike, wakke..atreelll. gartrell@acm org
Ludovic Dos Santos"
Criteo AI Lab, Paris, France
dossantos@critee com
Thibaut Issenhuth"
Emmanuel de Bézenac"
Criteo AI Lab, Paris, France LIGM, Ecole des Ponts, Univ Gustave Eiffel, France
Seminar for Applied Mathematics, D-MATH, ETH Zürich, Rämistrasse 101, Zürich-8092, Switzerland
CNRS, t. c.issenuuthcrriteo com
dibeanall.deeeeeacc@@am math, ethz. ch
Mickaël Chen"
Alain Rakotomamonjy" Criteo Lab, Paris, France a. rrakotomamojjy@ciic..o com
Valeo.ai, Paris, France mic mickael. www.cheeeee...oom com
Abstract
Particle-based deep generative models, such as gradient flows and score-based diffusion models, have recently gained traction thanks to their striking performance. Their principle of displacing particle distributions using differential equations is conventionally seen as opposed the previously widespread generative adversarial networks (GANs), which involve training pushforward generator network. In this paper we challenge this interpretation, and propose a novel framework that unifies particle and adversarial generative models by framing generator training a generalization of particle models. This suggests that generator is an optional addition to any such generative model. Consequently, integrating generator into a score-based diffusion model and training GAN without generator naturally emerge from our framework. We empirically test the viability of these original models as proofs of concepts potential applications our framework.
Introduction
Score-based diffusion models (Song al., 2021) have recently received significant attention within the machine learning community, due to their striking performance on generative tasks (Rombach et al., 2022: Ho al., 2022). Similarly gradient flows, these models involve systems of particles, where the displacement of the particle distribution is described by differential equation parameterized by gradient vector field. Such particle-based deep generative models are typically seen as opposed to generative adversarial networks (GANs, Goodfellow et al., 2014), as the latter involves adversarial training of generator network (Dhariwal & Nichol, 2021: Song. 2021; Xiao al., 2022).
In this paper, we challenge the conventional view that particle and adversarial generative models are opposed to each other. We make the following contributions.
unified framework. We present a novel framework that unifies both classes of models, showing that they are based on similar particle evolution equations. Particle models follow gradient vector
Authors listed in randomly chosen order.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1: Taxonomy of particle models, including our proposed hybrid models: Score GANs and Discriminator Flows.
Model
Generator
Flow type Vhp
Wasserstein gradient flows Stein gradient flows
X
Wasserstein gradient F(p)
Score-based diffusion models
a4 V log Pdata * kg(0)) (t)
3.V log
Score GANs
/
Discriminator Flows
((cofal
GANs
where Sp, is a discriminator between Pt and pdata
field during inference; in similar fashion, the generator's 's' outputs can be seen as following the same gradient field during training, up to a specific smoothing due to the generator. Building upon prior scattered literature, we propose new framework that encompasses variety of methods, which are listed Table together with their respective flow type.
Decoupling generators and flows. By uncovering the role of the generator as a smoothing operator on vector fields, we suggest that the existence of generator and the flow that particles follow can be decoupled. We deduce that it is possible to train generator with score-based gradients which replace adversarial training (which we call a Score GAN); and that a GAN can be trained without a generator, using only the discriminator to synthesize samples (which we call a Discriminator Flow); cf. Table 1. We introduce these new models as proofs concept, which we empirically assess to support the validity of our framework and illustrate the new perspectives it opens up in this active field of research.
Throughout the paper, we call out some specifics of the contributions of our framework as Definitions and Findings (italicized in the main text).
Outline. We begin with a discussion of particle models without generators in Section 2, covering Wasserstein gradient flows and score-based diffusion models. In Section 3 we discuss how generator training can be framed as a particle model with a generator, including GANs and Stein gradient flows. Section 4 then highlights how our framework allows to decouple the generator and flow components of particle models, leading to the aforementioned new models, Score GANs and Discriminator Flows. Finally, we discuss the implications of our findings and conclude the paper in Section 5.
We consider the evolution of generated distributions Pt over R w.r.t. time t E R4+, where is the inference time for particle models without generators, the training time for generator training, respectively. This evolution until some finite or infinite end time T E TR, U (+x) then yields final generated distribution Pg = Pr, which ideally approaches the data distribution Pdata-
Notations.
2 Particle Models without Generators: Non-interacting Particles
In this section we formally introduce the notion of generative particle models that do not use a generator, and present two standard instances: Wasserstein gradient flows and score-based diffusion models. They involve the manipulation of particles 24 Pt following differential equation parameterized by some vector field. We characterize these models by noticing that their particles actually optimize, independently from one another, a loss that depends on the current particle distribution.
Definition 1 (Particle Models, PMs). PMs model particles Pt. starting from prior po
Zi T== Po:
dx, (xt) dt,
(1)
where hp. RR E. is a functional that depends on the current particle distribution Pt. Time corresponds to generation iinference time from Po to the final distribution Ps -pr-
Finding 1. In PMs. the evolution of Equation (1) makes each generated particle xt individually follow a gradient ascent path the objective h., (zz).
In prior works, the prior π is conventionally chosen be easy to sample from, such a Gaussian. hpp is usually defined in theoretical manner so that the flow of Equation (1) conveys good convergence

---

properties for towards Pdata when - T. Because analytically computing hpp is often intractable, is empirically estimated and replaced by neural network in practice.
2.1 Wasserstein Gradient Flows
Wasserstein gradient flow is a generalization of gradient descent on a functional the space of probability measures. More formally, it is absolute continuous curve of probability distributions in a Wasserstein metric space P2 over IR' that satisfies a continuity equation (Santambrogio, 2017), and equivalently an evolution of particles 2, Pt under mild hypotheses (Jordan et al., 1998):
Oep: V (p: Vw F(pi)) = 0,
dæ
F(p))
(2)
where P: R is the functional to minimize in the Wasserstein space.
This definition involves the Wasserstein gradient of the functional F(p:) - similar to the gradient of a functional defined over a Euclidean space which for some functionals can be obtained in closed form by computing the first
variation of the functional F (Santambrogio. 2017, Section 4.3):
Op:. Vw = (.((pt)
RR + +RP
(3)
Finding 2. Wasserstein gradient flows are PMs with hpp (F((:))
+ ost and T = +00.
0p,
The partial differential equation governing the
particle evolution, as well as its convergence properties toward Pdata= strongly depends the functional F. We detail the standard ex-
Table 2:
Gradient flows for standard objectives F.
Objective J(p)
hp
amples of the forward Kullback-Leibler (KL) divergence (Kullback & Leibler, 1951), f-
Forward KI.
B. log P/Pdato
Ing P/Pato (1/vaara)
f-divergence
Branto (P/vausu)
divergences (Rényi, 1961), the squared Maxi- mum Mean Discrepancy (MMD, Gretton et al., 2012; Arbel et al., 2019), and entropy regu- larization in Table 2. They can be additively combined for variety of objectives F. More
A (xxxz")
Squared MMD
s.oimp v.y PPaata
Ey [<(x.)) -Kompp M(xr)))
w.L. kernel
Entropy
Ep log
-log p
examples exist in the literature (Liutkus et al., 2019; Mroueh et al., 2019; Glaser et al., 2021).
Several methods have been explored in the literature to solve Equation (2) in practice, either using input-convex neural networks (Mokrov et al., 2021; Alvarez-Melin et al., 2022) to discretize the continuous flow, parameterizing hh by neural network (Gao et al., 2019; Fan et al., 2022; Heng et al., 2023). In all cases these methods fit within the class of PMs as framed in Definition 1.
2.2 Score-Based Diffusion Models
Early score-based models (Noise Conditional Score Networks [NCSN], Song & Ermon, 2019) rely on Langevin dynamics as described in the following stochastic differential equation, converging towards Pdata when t + oc:
dx; log (araa((xe) + dWW
Several methods that neural networks estimate the score function of the data distribution V log Paata (Hyvärinen, 2005), coupled with the use of Langevin dynamics, can work in practice even for high-dimensional distributions. Nonetheless, because of ill-definition and estimation issues of the score for discrete data on manifolds, Gaussian perturbation of the data distribution is introduced stabilize the dynamics. Thus, Pdata in Equation (4) is replaced by the distribution Plata of ae, where I ~ Pagta and & ~ (O, ID). Denoting as the convolution of a probability distribution by a kernel k: R R D we notice that Plata is actually the convolution of Pdata by a Gaussian kernel:
Piata Pdata
KRBP,
k 4
dp(z).
K
2
(5)
2T
NCSN then follows this equation, estimating the score with denoising score matching (Vincent, 2011) and repeating the process for a decreasing sequence of as:
dæ,
V log[pdata KR (xt) dt + v2dw.
(6)
3

---

Building on Song et al. (2021), newer score models (Karras et al., 2022) make a continuous function of time (t), decreasing towards finite time to improve convergence. Many of these approaches share the following generation equation, corresponding to the reverse of noising process for Pdata (Elucidating the Design Space of Diffusion- Based Generative Models [EDM), Karras al., 2022):
dr. = 20' (t)o(t)) log Pdata * KREE (xr) dt + /2 (W,,,
(7)
where o'(() is the derivative of o()). These approaches admit an equivalent deterministic flow yielding the same probability path pr:
dx. = o' (t) log Pdata
KRBE (xr) dt.
(8)
However, we notice that this only holds under the implicit assumption that Equation (7) perfectly (.((() reverses the initial noising process, i.c., Pt. Pdata RBBB- This assumption can be broken in practice when the score is not well estimated or for coarse time discretization, In the general case for both NCSN and EDM, by using the Fokker-Planck equation (Jordan et al., 1998), which allows to substitute the stochastic component dW, by the negative of the score the generated distribution, obtain, respectively, the following equivalent exact probability flows:
dæ dt
Pdata KRBE
dæ d((oo((t)V log
(o(t) 27 (psaaa * KRBF) (1r)).
log
(Iz),
(9)
Pt
dt
Finding 3. Score-based diffusion models are PMs:
NCSN (Song & Ermon, 2019) with hp. = log [Paata * RBF] log Pt and T +00;
EDDM (Karras et al., 2022) with hp. = (t) 2o Pdata
po(t) RBF
log and T < +00.
NCSN actually implements through Langevin dynamics a forward KL gradient flow, which is common knowledge in the related literature (Jordan et al., 1998; Yi et al., 2023).
3 Particle Models with Generators: Training of Interacting Particles
In the previous section we presented framework for particle models (PMs) that, in the absence of generator, individually manipulate particles in the data space, and optimize a distribution-depeedent objective hpp via differential equation. In this section of PMs involving direct interaction between particles. We show, supported by prior literature, that our framework applies the case of GANs and Stein gradient flows.
frame generator training generalization
3.1 Generator Training as Modified Particle Model
We begin with the training of neural generator go: Rd + RDD parameterized by Associated with prior distribution on its latent space Ps. ge produces generated distribution pe the pushforward of through ge: Py = getp., by which we seek to imitate Pdata Unlike PMs which progressively construct synthesized samples directly in the data space D generators enable models like GANs to generate samples starting from different latent space. When d < D, this latent space allows the resulting distribution be naturally embedded into a owverddimesiional manifold, thereby integrating the manifold hypothesis (Bengio et al., 2013). The parameters 0 evolve during training, making the generated distribution move accordingly: Pt ps..
We characterized in Section 2, Finding 1, PMs as models that make free generated particles optimize objective h,: D - R.. that conveys desirable convergence properties. We leverage this observation to show that generator training can be framed as PM as well. We see that generators involve generated particles x, Pt as generator outputs I; ge, (z), with ~ Pa. which move during training. We proceed by making the generator optimize the same objective hpp as in PMs, that is, the generator parameters are trained to minimize each optimization step:
Leen (0) = -Eewp [np (9o(3))
(10)

---

In the optimization of Equation (10), we intentionally ignore the dependency of p on 8, i... in practice p = StopGradient(got22 This allows us to mimic PMs where generated particles zys optimize the objective hpp. without taking into account that P: is actually a mixture of all the r,s.
By optimizing Or via gradient descent for the loss of Equation (10) with learning rate n, idealized in the continuous training time setting, we obtain using the chain rule:
d0
17 V,, Eewpa [hpp. (ge, (z))
7/uaaps Vo.go, V2) (go, (2))
(1I)
As consequence, using the chain rule again, each generated particle 2t go. (2) ~ Pt evolves as: (g),
d0, nB Kigu, (2,2')(4pp hp. g0. (z')
(z)
(12)
dt
where kigo 2, Va.go Vo,go, (z') is the matrix Neural Tangent Kernel (NTK, Jacot et al., 2018) of the generator. Equation (12) describes the dynamics of the generated particles as a modified version of Equation (1) for PMs. We formalize this as follows.
Definition 2 (Interacting Particle Models, Int-PMs). Int-PMs model particles resulting from the pushforward go. of a generator 9o, applied to prior Pa, with the following training dynamics:
dgo, (z) n [o, (z)] (Vhp.) dt,
(13)
where h pm: RRP - R is a functional that depends the current distribution pt, time is training time, and Ao, (z) is a linear operator operating on vector fields (Sriperumbudur al., 2010), defined as:
(=) (V) A EE Kkgn (z,z") go. (())) ligo, (z,z²) A Vo.90. (2) T Vo.90. (z')- (14)
Similarly to PMs, the vector field Vhp, in Int-PMs indicates which direction each generated particle will follow get closer to the data distribution. However, Ae, smooths this gradient field using the generator's s NTK, and generated particles thus interact with each other. Indeed, moving one particle makes its neighbors move accordingly because of their underlying parameterization by the generator. Notably, Int-PMs generalize PMs: in the degenerate case where 6(2,=') = 82-2'Ip. with the Dirac delta function centered 0, i.e., when particles can move freely with a sufficiently powerful generator, the effect of parameterization disappears with [Ao, (V) V (go(2)), and therefore Equation (13) reduces to Equation (1).
Finding 4. Int-PMs generalize PMs. Each Int-PM is therefore defined by two components: the objective function hp, and the choice of generator architecture go-
We will show in the remainder of this section that Int-PMs encompass both GANs and Stein gradient flows, borrowing Franceschi et al. (2022)'s 's results which the previous reasoning generalizes.
3.2 GANs as Interacting Particle Models
In GANs, each generator go is accompanied by a discriminator Sp that depends on the generated distribution. fo is optimized as a neural network via gradient ascent (GA) to maximize an objective of the following form:
fo fo GA Calfi Paata) Epla fl Ep.... o fl + R(f: Pdaats)
(15)
for some functions a, b: + R (e.g. for the WGAN of Arjovsky et al. (2017), a = h = id) and regularization R (e.g., the gradient penalty of Guirajani al. (2017)). In this work, remain oblivious to how the discriminator is trained practice as a single network alongside the generator. Nonetheless, we note that this GA is usually stopped early and not run until convergence, because the discriminator is trained only for a few steps between generator updates.
This discriminator is then used train the generator, usually framed in min-max optimization setting. However, several works (Metz et al., 2017; Franceschi et al., 2022; Yi et al., 2023) showed that generator optimization deviates from min-max optimization, because alternating updates between the generator and the discriminator make the generator minimize a loss function of the form (Franceschi al., 2022):
LGAN(go) = Ezzp, (co Se) (co(=))}
(16)
for some I- + I. (c.g., for WGAN, c= ii). Using Definition 2, we deduce the following.
5

---

Finding 5. GANs are Int-PMs with ho - -co f.p where fp is the current discriminator.
Under some assumptions on the outcome of discriminator training in Equation (15), the resulting Vh, for GANs has been proven to implement a Wasserstein gradient F(p). Two notable examples are (see also Section 2.1), 2) f-divergence GANs (Nowozin et al., 2016), which are linked to the forward KL divergence gradient flow (Yi et al., 2023) and therefore to diffusion models; and Integral Probability Metrics (IPM) GANs, which are linked to the squared MMD gradient flow w.r.t. the NTK of the discriminator (Franceschi et al., 2022). However, these links have been made under strong simplifying assumptions, and the GAN formulation as an Int-PM in this paper is far more general.
Finding 6. f-divergence GANs as MMs generalize the forward KL gradient flow and Langevin diffusion models, and IPM GANs generalize MMD gradient flows.
3.3 Stein Gradient Flows as Int-PMs
Int-PMs as framed in Definition 2 are similar to Stein gradient flows (Liu & Wang, 2016; Liu, 2017; Duncan et al., 2023). The latter are generalization of Wasserstein gradient flows in another geometry shaped by a matrix kernel k: RDD x I + DDD defined in the data space:
dx. E (1,, x/)) w F(pe)p)) di.
(17)
Prior works showed strong links between such flows and GAN optimization, which help us to see that GANs are Int-PMs. In the following, will see that Stein Gradient flows serve an example of functional generator-based counterpart of a generator-less PM.
We begin by generalizing the reasonings of Chu et al. (2020), Durr al. (2022) and Franceschi et al. (2022). which initially applied only to the case of GANs. We assume that for Int-PMs in Equations (13) and (14), the generator's 's NTK is constant throughout training, i.e. kgo, = kg. like for many networks with infinite width (Jacot et al., 2018; Liu et al., 2020). Then, when Vh, V w F(pt), e.g., for
gradient flows as in Finding 2 or for GANs following Finding 5, we obtain an equation similar to Equation (17):
dgo, (2) = -IEE'' rup kig Flp)) go. (00((())) (+ dt
(18)
This is special case of Equation (17) with an invertible generator (Chu et al., 2020). However, in the
general case, the kernel ky: Rdd x Rd -> R. DxD acts on the latent space, which makes Equation (18) define a generalized latent-driven Stein flow that was uncovered by Franceschi et al. (2022).
Finding 7. Stein gradient flows are Int-PMs with invertible generator in the NTK regime and ()(())) + cst. They can be generalized non- non-invertible generators with Equation (18).
hps
8p.
4 Decoupling the Generator and the Flow in Particle Models
We saw in the last section that Int-PMs, such as GANs, generalize PMs, such as diffusion and Wasserstein gradient flows, by applying their generative equations to generator training. This implies that many generative models can be defined by their PM flow, which allows an optional generator to be trained. From Section 3.3, this is the case for gradient flows, which can either define PM an Int-PM with the same Vh as a Wasserstein gradient. As per Finding 6, this also holds for GANs, which share common particle movement with gradient flows and diffusion models.
Consequently, our framework suggests that a functional hp used in an Int-PM may equivalently be used in a PM, and vice versa. This leads to formulate the following claim.
Claim 1. A generator can be trained using the gradient flow of score-based diffusion model instead of adversarial training, and it is possible to remove the generator in a GAN by synthesizing samples with the discriminator only.
We confirm this hypothesis this section by introducing corresponding new hybrid models, which we call respectively Score GANs and Discriminator Flows (see Table 1), and by empirically demon- strating their viability. Note that we introduce these models as proofs of concepts of the applications of our framework. Since they challenge many assumptions and standard practices of generative modeling, they do not benefit from the same wealth of accumulated knowledge that classic models have access to, and thus are harder to tune than standard models.

---

Flow BDM
6
6
Floww GAN GAN CDM
9
0
5
8
9
9
9
9
9
9 9
9
9
Score
/
9
6
9
7
3
8
9
2
Figure 1: From left right, generation process on MNIST of EDM and Discriminator Flow for every 8 evaluations of Vhp. The last row shows the first 7 steps of Discriminator Flow.
Discr:
2
7
8
8
Figure
Uncurated samples of studied models on CelebA and MNIST.
Training iteration of Score GANs; all operations can be performed in parallel for batching. See Appendix for details on the practical implementation of lines 3 and 5.
Algorithm 1:
Input: Noise distribution Po. number of intermediate score training steps K, learning rates
X, E R4, previous generator ge IR - R. previous p score model s R R -R
pretrained Pdata score model
" s RD x IR + R.D.
Output: Updated generator Go and score model sy
for -1to K do
// Updates of s" by denoising score matching Pz. go(2), Po. x" ~ N (x (x ID);
- Φ AV
(2 .0)
// Score matching with generator, Equations (11) and (19)
Pa, 0 Po. & NNO,
- + Voga(z)
Pdata (go(z) + oe,o) - (g0())) + oe,
Experimental setting. We conduct experiments on the unconditional generation task for two standard datasets composed images: MNIST (LeCun et al.,
Table 3:
Test FID of studied models.
1998) and 64 x 64 CelebA (Liu et al., 2015). We consider two reference baselines, EDM (the score- based diffusion model of Karras et al. (2022)) and GANs, and use the Fréchet Inception Distance (FID, Heusel et al., 2017) to test generative performance in Table 3. Training details are given in Appendix D; our open- goen-e code is available at https: //github. com/White- Link/ /gpm. We refer to Appendix C and the code for more experimental results and samples for each baseline.
PMs (no generator)
(generator)
Datasct
EDM
Diser. Flow
GAN
Score GAN
MNIST CelebA
3
4
3
15
41
35
4.1 Training Generators with Score-Based Diffusion: Score GANs
We propose training a generator with the score-based diffusion flow of NCSN, Equation (9), left. This involves applying Equation (13) with h,, log Paata KRBF] log Pt. where represents the training time of the generator. To so, we directly use the generator weight update formula of Equation (11), as this avoids the problem of estimating lpp and only requires its gradient Vhp, V log Pdata kij kRpp]] - log Pt. Composed of the scores of, respectively, the noised data distribution and the generated distribution, Vhp, can be efficiently estimated via score matching techniques.
In practice, we use a score network Paaat pretrained with the latest denoising score matching techniques (Karras et al., 2022) estimate the static term V Voog [Patt kg k Moreover, as v log is dynamic and needs to be continuously estimated, we leverage GAN discriminator practices and train network & by alternating with generator updates to estimate this score.
However, our proposed solution remains impractical for two reasons. First, since the dynamics would match Pdata * kep and Pt, we would need to schedule as during training, similar to what Song & Ermon (2019) do during inference. Second, while V log Pt be estimated using sliced score
7

---

Algorithm 2:
Training iteration of Discr. Flows. Cr. batching and discretization in Appendix B.
Input: Initial distribution Po. gradient strength !) R2,, learning rate +. previous discriminator fo: R.D x - IR.
Output: Updated discriminator fø. ~ Pdatas n, 2( 1);
// Initialization, random sampling time
Ft t 20 S Vx, [co (z) ds; // Partial generation, Equation (20)
Φ - + AV La (Jo(: 8zy (2)))))))))))
8xx)
// Train So at time cf. Equation (15)
matching (Song et al., 2020), this approach is less performant than denoising score matching and leads to estimation issues when Pt lies a manifold (Song & Ermon, 2019), in our case with pushforward generator. Both of these problems can be solved by instead matching pdata * RBP and Pr kg KRF for range of Po. using Equations (12) and (13):
pp: log/Pasta
RBF
log(Pt RBF.
(19)
This approach allows us to leverage denoising score matching to train sy and to avoid scheduling os by instead sampling them during training and noising the generated distribution with the chosen noise levels. Overall, obtain the algorithm for Score GANs described in Algorithm
Finding 8. Score GANs Int-PMs with hp Ergrrp
log
Pdsta K
and T +00.
Park
Like in GANs, the generated score network sf, is trained for small number of steps K in between two generator updates. Accordingly, as is the case for discriminator, K is an important parameter for Score GANs. We study its impact of the method's performance in Appendix C.4, where we ensure that small K suffices accurately estimate the score of the generated distribution.
4.2 Removing the Generator from GANs: Discriminator Flows
Based on Findings 4 and 5, we see that removing the generator from GANs to make them PMs, as in Definition simply requires that define hp. cco fp., where fp. is the discriminator between Ps and Pdata at sampling time t in Equation (1):
dx, (c fpx) (2,) dt.
(20)
This makes Equation (20) the equivalent of GAN training, but as PM without a generator. In other words, Discriminator Flows make individual particles follow the gradient of the generator loss of Equation (16), defined through the discriminator, without the generator smoothing of Equation (13). Finding 9. Discriminator Flows are PMs with hp o fp. and T +
Such a model would a priori require us to successively train a neural discriminator fo, = Sp. per time step 1. This results, however, alternative, we train a single time-dependent neural discriminator fa: R. x R + R, that takes as input both sample x, ~ Pt and its corresponding time E. on all time steps at once. Each 3; Pt must then be computed both in training and inference from To ~ = π using Equation (20). This results in the training procedure of Algorithm for Diseriminator Flows. For practical convenience, we restrict, without loss of generality, & E [0, 1].
prohibitively slow and heavy training procedure. As more scalable
Compared to diffusion models, for which the score can be freely estimated at each because Pe
assumed to equal Piata KRAP. is more challenging to elucidate the particle movement of Equation (20). Prior studies on GAN optimization help answer this problem under simplifying assumptions. Indeed, if a, b, and c (cf. Section 3.2) are chosen to implement an f-divergence GAN loss, then Discriminator Flows will implement forward KL divergence gradient flow (Yi et al., 2023) If they instead correspond to an IPM GAN loss, then Discriminator Flows will implement a squared MMD gradient flow (Franceschi et al., 2022).
In the general case, the generation process of Diseriminator Flows is not known in advance. In fact, must simulate the entire process during training, making each training iteration slower. However,
8

---

our approach has the advantage of generalizing the diffusion process and allowing the discriminator to learn another path from the initial distribution Pa to Pdata- We believe that, when properly tuned, Discriminator Flows could provide faster sampling times than diffusion models because they can learn shorter paths towards Pdata-
4.3 Experimental results
We see uncurated samples from our proposed Score GAN and Discriminator Flow models in Figure 2, as well as the baseline EDM and GAN models. In Table we report the FID scores for all models. We observe that both of our introduced models produce reasonable results, which experimentally confirms our initial claim. Nonetheless, these models exhibit worse (higher) FID scores than our baselines, although Discriminator Flows provide good performance on the MNIST dataset. We attribute these performance results to the fact that our proposed models are novel and do not benefit from the accumulated knowledge regarding training best practices that the standard models possess.
Interestingly, the proposed models exhibit typical properties of generator- and generator-based models. Since Score GANs use generator, in GANs, only one function evaluation is required to draw a sample, making it orders of magnitude faster to sample from than EDM. Like GANs, Score GANs may produce mode collapse and benefit from smooth latent space interpolations, as shown in Appendices C.2 and C.3. Of course, Discriminator Flows are slower at both training and inference time than such generator-based models, but this comes with the additional flexibility of operating directly in the data space (Voleti et al., 2022; Couairon et al., 2023). As a consequence, like diffusion models, the latent space interpolation (i.e., interpolating the initial noise no in Equation (1)) capabilities of Discriminator Flows remain below those of generator-based models - cf. Appendix C.3.
However, as shown in Figure and in the numerical results of Appendix C.5, Discriminator Flows have the expected advantage of converging Pdata faster than the state-of-the-are diffusion model EDM. In particular, we notice that images generated by Discriminator Flows are well-formed early in the generation process, hinting at potential temporal cost reduction by stopping the process early. Unfortunately, this has not yet yielded a better time efficiency than EDM; we nevertheless believe can be achieved with additional architectural and model tuning of Discriminator Flows. We further discuss the time efficiency of our introduced methods in Appendices A.4, C.5 and D.3.
4.4 Relationship with Prior Work
Score GANs. Conceptually, Score GANs implement for each forward KL gradient flow, similarly to Yi et al. (2023), who theoretically proved that some GAN models approximate such a flow without noising, under optimality assumptions on the discriminator. However, Score GANs differ from traditional GANs as they do not involve discriminator, but rather split the flow Vhp. into two parts: one part that can be estimated before generator training as it depends only on the data distribution, and another part that must be continuously estimated during training as it depends on the
generated distribution. While hp, could be estimated by adding noise to the inputs of a discriminator (Wang et al., 2022), Score GANs instead only need to estimate the score of the generated distribution, which is no longer adversarial.
Discriminator Flows. As general concept, Discriminator Flows provide encompassing frame- work that helps to reveal the connections of various approaches to GAN training.
Recently, Heng et al. (2023) introduced deep generative Wasserstein gradient flows (DGGF), method that relies on f-divergence gradient flows approximated by estimating the ratio Pr/pant. with a neural network. Examined under our framework, DGGF's training objectives correspond to that of discriminator in fdiivergence GANs, allowing us to frame DGGF as a special case of Discriminator Flows. Nonetheless, we stress that Discriminator Flows have larger scope than DGGF since we can handle all types of GAN objectives; all our experiments were performed with WGAN objectives. Moreover, unlike DGGF, which in practice removes the time dependency in its estimation without theoretical justification, our method does handle time as input to the discriminator.
Discriminator Flows also relate to, and generalize, methods that finetune GAN outputs with gradient flows (Tanaka, 2019; Che al., 2020; Ansari et al., 2021). The latter use discriminator gradients to approximate such flows, making them naturally expressible Discriminator Flows in our framework.

---

In practice, they only apply their sampling procedures in the latent space of the generator, as applying them in pixel space leads to artifacts. We resolve this issue with a principled training procedure for the discriminator conditioned on sampling time.
We note that previous works, such as Xiao et al. (2022) and Mnlicceur-Martinennl et al. (2021), also combine score-based models and GANs. Xiao et al. (2022) train several GANs to successively denoise an image, mimicking reverse diffusion process.
Seficeeur-Martnnss et al. (2021) augment the denoising objective of score-based diffusion models with an adversarial objective to improve the denoising image quality. However, while these works do draw links between both models, they do not aim to unify score-bee score-based models and GANs.
Finally, we provide intuition on the sampling efficiency of Discriminator Flows observed in Figure 1. We observe that diffusion models smooth the data distribution with a Gaussian kernel by Equation (5), while discriminators were shown smooth the data distribution with their NTK (Franceschi et al., 2022). This observation brings diffusion models and GANs closer to each other, while explaining the fast convergence speed of Discriminator Flows thanks the properties of NTKs for generative modeling. Indeed, Franceschi et al. (2022) presented empirical evidence that using NTKs of standard discriminators as kernels in squared MMD gradient flows (which some GAN models implement, cf. Finding 6), instead of Gaussian kernels, accelerates the convergence of Equation (1) towards Paata by several orders of magnitude. More generally, we hypothesize that it is the natural effectiveness of neural networks in high dimensions that endows Discriminator Flows with a faster convergence speed than diffusion models, for which the particles' paths are chosen explicitly.
5 Conclusion
In this paper we have unified score-based diffusion models, GANs, and gradient flows under single framework based on particle models which can be complemented with a generator. Since this framework unifies models that have been customarily opposed in the literature, this work paves the way for new perspectives in generative modeling. As an example of potential applications, we have shown that our framework naturally leads to two novel generative models: generator that follows score-based gradients, and a generator-free GAN that uses a discriminator-guinn guided generation process.
Of course, generator-less and generator- models each retain their unique attributes. On the one hand, generator training provides a simple and efficient sampling procedure and endows the generative model with a -in-dimeniional structured latent space, at the cost of potential instability and mode collapse. On the other hand, generator-less models, despite their slow sampling, may be easier to train, since the generator component has been removed from their flow, and are more flexible as they rely on a continuous-time process directly defined in the data space. We believe that our framework, by revealing the close relationship between these models, can help them to improve upon one another, or can even help create other new hybrid models.
Beyond potential applications, our study could be enhanced and expanded in many ways for future work. On the theoretical side, we would like to tackle the challenging task (Hsieh et al., 2021) of taking into account the fact that the discriminator in GANs is actually continuously trained with the generator. would also be interesting to generalize our framework to second-order and stochastic particle movement in generator- based models, and to study the impact of discretization on the studied differential equations, as hinted Appendices A. to A.3, On the practical side, while we have proposed new models that function reasonably well, we would be interested in refining them further for state-of-the-an generative performance. Furthermore, Score GANs could serve as a distillation method for ccsre-based diffusion models (Salimans & Ho, 2022), while Discriminator Flows could outperform diffusion models for generation efficiency.
Acknowledgments and Disclosure of F Funding
We would like to thank Lorenzo Croissant and Ugo Tanielian for helpful discussions and comments on this paper, as well as Edouard Delasalles for inspiring the architecture of our code.
This work was granted access to the HPC/AI resources of IDRIS under the allocation 2023- ADOI 1013503RI made by GENCI (Grand Equipement National de Calcul Intensif). Emmanuel de Bézenac is financially supported by the ETH Foundations of Data Science.
10

---

References
Alvarez-Melis, D., Schiff, Y., and Mroueh, Y. Optimizing functionals on the space of probabilities with imput convex neural networks. Transactions on Machine Learning Research, 2022.
Ansari, A. F., Ang, M. L.,, and Soh, H. Refining deep generative models via discriminator gradient flow. International Conference on Learning Representations, 2021.
Arbel, M., Korba, A., Salim, A., and Gretton, Maximum mean diserepancy gradient flow. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alché Buc, F., Fox, E., and Garnett, R. (eds.), Advances Neural Information Processing Systems, volume 32, pp. 6484-6494 6494, Curran Associates, Inc., 2019.
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein generative adversarial networks. In Precup. D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. PMLR, August 2017.
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE Transactions on Puttern Analysis and Machine Intelligence, 1598) August 2013.
Bernstein, J., Wang. Y.-X., Azizzadenesheli, K., and Anandkumar, A. signSGD: Compressed optimisation for non non-convex problems. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 560-569. PMLR, July 2018.
Brock, A., Donahue, J., and Simonyan, K. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Che, T., Zhang, R., Sohl-Dickstein, J., Larochelle, H., Paull, L., Cao. Y., and Bengio, Y. Your GAN is secretly an energy-basee model and you should use discriminator driven latent sampling. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.-F, and Lin, H.-T. (eds.), Advances Neural Information Processing Systems, volume 33, pp. 12275-12287. Curran Associates, Inc., 2020.
Chu, C., Minami, K., and Fukumizu, K. The equivalence between Stein variational gradient descent and black-box variational inference. arXiv preprint wxXiv2204.008222 2020.
Couairon, G., Verbeek, J., Schwenk, H., and Cord, M. DiffEdit: Diffusion- based semantic image editing with mask guidance. In International Conference on Learning Representations, 2023.
Dhariwal, P. and Nichol, A. Diffusion models beat GANs on image synthesis. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 8780-8794. Curran Associates, Inc., 2021.
Duncan, A., Nüsken, N., and Szpruch, L. On the geometry of Steinl heurrrr variational gradient descent. Journal of Machine Learning Research, 24(56):1 2023.
Durr, S., Mroueh, Y., Tu, Y., and Wang. S. Effective dynamics of generative adversarial networks. arXiv preprint wtx.x:2222.048800 2022.
Fallis, D. The epistemic threat of deepfakes. Philosophy & Technology, 14:623-643, 2021.
Fan, J., Zhang, Q., Taghvaei, A., and Chen, Variational Wasserstein gradient flow. In Chaudhuri, K., Jegelka, S., Song, I.., Szepesvári, C., Niu, G., and Sabato, S. (eds.), P Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 6185-6215, PMLR, July 2022.
Franceschi, J.-Y., De Bézenac, E., Ayed, I., Chen, M., Lamprier, S., and Gallinari, P. A neural tangent kernel perspective of GANs. Chaudhuri, K., Jegelka, S., Song, I.,, Szepesvári, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 Proceedings of Machine Learning Research, pp. 6660-6704. PMLR, July 2022.
Gao, Y., Jiao, Y., Wang, Y., Wang, Y., Yang, C., and Zhang, S. Deep generative learning via variational gradient flow. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Leaniing, volume 97 of Proceedings of Machine Learning Research, pp- 2003- PMLR. June 2019.
11

---

Glaser, P., Arbel, M., and Gretton, A. KALE flow: A relaxed KL gradient flow for probabilities with disjoint support. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan. J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 8018-8031. Curran Associates, Inc., 2021.
Goodfellow, NIPS 2016 tutorial: Generative adversarial networks. arXiv preprinl 2016.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Farde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D., and Weinberger, K. Q. (eds.), Advances Neural Information Processing Systems, volume 27, pp. 2672-2680. Curran Associates, Inc., 2014.
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. kernel two-sample test. Journal of Machine Learning Research, 13(25):723-773, 2012.
Guirajani, I. Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. Improved training of Wasser- stein GANs, In Guyon, I., von Luxburg, U.. Bengio, S., Wallach, H., Fergus, R., Vishwanathan, V. N., and Garnett, R. (eds.). Advances in Neural Information Processing Systems, volume 30, pp. 5769-5779. Curran Associates, Inc., 2017.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (GeLUs). arXiv preprint arXiv:1606.08415, 2016.
Heng, A., Ansari, A. F,, and Soh, H. Deep generative Wasserstein gradient flows, 2023. URL
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 6629 6629. Curran Associates, Inc., 2017.
Ho, J., Chan, W., Saharia, C., Whang, J., Gao. R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., and Salimans, T. Imagen Video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.023000 2022.
Hsieh, Y.P.., Mertikopoulos, P., and Cevher, V. The limits of min-max optimization algorithms: Convergence to spurious non-critical sets. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4337 4344. PMLR, July 2021.
Hyvärinen, A. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, ((24):695-709) 2005.
loffe. S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 448 456, Lille, France, July 2015. PMLR.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31, pp. 8580-8589. Curran Associates, Inc., 2018
Jelassi, S., Dobre, D., Mensch, A., Li, Y., and Gidel, G. Dissecting adaptive methods in GANs. arXiv preprint arXiv:2210.043/1 2022.
Jolicoeur-Martienne A., Fiché-Taillefee R., Mitliagkas, I., and Tachet des Combes, R. Adversarial score matching and improved sampling for image generation. In International Conference on Learning Representations, 2021.
Jordan, R., Kinderlehrer, D., and Otto, The variational formulation of the Fokker-Planne equation. SIAM Journal on Mathematical Analysis, 29(1):1-17, 1998.
12

---

Kang, M., Shin, J., and Park, J. StudioGAN: A taxonomy and benchmark of GANs for image synthesis. arXiv preprint arXiv:2206.094 09479, 2022.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances Neural Information Processing Systems, volume 35, pp. 26565-26577 Curran Associates, Inc., 2022.
Kingma, D. P. and Ba. J. Adam: A method for stochastic optimization. In International Conference on Learning Representations. 2015.
Klebaner, F. C. Introduction Stochastic Calculus with Applications. Imperial College Press, 3rd edition, 2012.
Kloeden, P. E. and Platen, E. Introduction to Stochastic Time Discrete Approximation, pp. 305-337. Applications of Mathematics. Springer Berlin Heidelberg, Berlin Heidelberg, Germany, 1992.
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, April 2009.
Kullback, and Leibler, R. A. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79-86, 1951.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2222 November 1998.
Lesniak, D.,. Sieradzki, I., and Podolak, I. Distribution-interplaaiio trade off in generative models. In International Conference on Learning Representations, 2018.
Li, X., Thickstun, J., Guirajani, I., Liang, P., and Hashimoto, T. B. Diffusion-LAN improves control- lable text generation. In Koyejo. S., Mohamed, S.. Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.). Advances in Neural Information Processing Systems, volume 35, pp. 1328--433, Curran Associates, Inc., 2022.
Lim, H. and Ye, C. Geometric GAN. arXiv preprint arXiv:1705.02888 2017.
Liu, B., Zhu, Y., Song, K., and Elgammal, A. Towards faster and stabilized GAN training for high-fidelity few-shot image synthesis. In International Conference on Learning Representations, 2021.
Liu, C., Zhu, L., and Belkin, M. On the linearity of large non-linear models: when and why the tangent kernel is constant. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.-F, and Lin, H.-T. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 15954-15964. Curran Associates, Inc., 2020.
Liu, L., Lu, Y., Yang, M., Qu, Q.. Zhu, J., and Li, H. Generative adversarial network for abstractive text summarization. In Proceedings of the Thirty-econn AAAI Conference on Artificial Intelligence, AAAI'18, pp. 8109-8110 10. AAAI Press, 2018
Liu, Q. Stein variational gradient descent as gradient flow. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan. S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 3118 3118-1111221 -3126. Curran Associates, Inc., 2017.
Liu, Q. and Wang, D. Stein variational gradient descent: A general purpose Bayesian inference algorithm. In Lee, D. D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29, pp. 2378-2386. Curran Associates, Inc., 2016.
Liu, Z.. Luo, P., Wang. X., and Tang. X. Deep learning face attributes in the wild. In IEEE International Conference on Computer Vision (ICCV), pp. 3730-3738, December 2015.
Liutkus, A., Simsekli, U., Majewski, S., Durmus, A., and Stöter, F.-.. iesse--aasersttein flows: Nonparametric generative modeling via optimal transport and diffusions. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4104-4113, PMLR, June 2019.
13

---

Lucy, L. and Bamman, D. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pp. 48- 48-55, Virtual, June 2021. Association for Computational Linguistics.
Metz, L., Poole, B., Pfau, D., and Sohl-Dickstein, J. Unrolled generative adversarial networks. In International Conference on Learning Representations, 2017.
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Mokrov, P., Korotin, A., Li, L.,, Genevay, A., Solomon, J., and Burnaev, E. Large-scale Wasserstein gradient flows. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 15243-15256. Curran Associates, Inc., 2021.
Mroueh, Y., Sercu, T., and Raj, Sobolev descent. In Chaudhuri, K. and Sugiyama, M. (eds.), Proceedings of the Twenty-Secont International Conference Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 2976-2985. PMLR, April 2019.
Müller, A. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429 443, 1997.
Nichol, A. Q.. Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B.. Sutskever, I,, and Chen, M. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16784-16804 PMLR, July 2022.
Nowozin, S., Cseke, B., and Tomioka, R. f-GAN: Training generative neural samplers using variational divergence minimization. In Lee, D. D., Sugiyama, M., von Luxburg. U., Guyon, I., and Gamett, R. (eds.), Advances in Neural Information Processing Systems, volume 29, pp. 271-279. Curran Associates, Inc., 2016.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A.. Yang. E., Vito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An imperative style, high-performance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alché Buc, F,, Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32, pp.
Curran Associates, Inc., 2019.
Radford, A., Metz, L., and Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Representations, 2016.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. (0684-10695, June 2022.
Ronneberger, O.,, Fischer, P., and Brox, T. U-Net: Convolutional networks for biomedical image segmentation. In Navab, N., Hornegger, J., Wells, W. M., and Frangi, A. F. (eds.), Medical Image Computing and Computer-Assister Intervention - MICCAI 2015, pp. 234-241 Cham, Switzerland, 2015. Springer International Publishing.
Rényi, A. On measures of entropy and information. In Neyman, (ed.), Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pp. 547-561. University of California Press, 1961.
Salimans, T. and Ho, J. Progressive distillation for fast sampling diffusion models. International Conference on Learning Representations, 2022.
Santambrogio, F. (Euclidean, metric, and Wasserstein) gradient flows: an overview. Bulletin of Mathematical Sciences, 7:87-154, 2017.
Seymour, J. and Tully, Generative models for spear phishing posts on social media. arXiv preprint 10X.008802.05166, 2018.
14

---

Skafte Detlefsen, N., Borovec, J., Schock, J.. Harsh, A., Koker, T., Di Liello, L.,, Stanel, D., Quan, C., Grechkin, M., and Falcon, W. TorchMetrics measuring reproducibility PyTorch, February 2022. URL. htgpt///...tpeg//gt.t...ooooggga wwwp:gg-AAAttttttttttgree
Song, Y. Generative modeling by estimating gradients of the data distribution. https://yang-aa /tlpb/lggg///////////eeeeee
2021. Accessed: 2023-05-17.
Song. Y. and Ermon, Generative modeling by estimating gradients of the data distribution. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alché Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 11918-11930. Curran Associates, Inc., 2019.
Song, Y., Garg, S., Shi, J., and Ermon, S. Sliced score matching: A scalable approach to density and score estimation. In Adams, R. P. and Gogate, V. (eds.), ), Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pp. 574-584, PMLR, July 2020.
Song. Y. Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021.
Sriperumbudur, B. K., Gretton, A., Fukumizu, K., Schölkopf. B., and Lanckriet, G. R. G. Hilbert space embeddings and metrics probability measures. Journal of Machine Learning Research, 11(50):1517-156) 2010.
Tanaka, A. Discriminator optimal transport. In Wallach, H., Larochelle, H., Beygelzimer, A., d' A'chéé Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 6616-6826. Curran Associates, Inc., 2019.
Tanielian, U.. Issenhuth, T., Dohmatob, E.,, and Mary, J. Learning disconnected manifolds: no GAN's land. In Daumé, III, H. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9418-9427. PMLR, July 2020.
Vincent, P. connection between score matching and denoising autoencoders. Neural Computation, 23(7):661116774 July 2011.
Voleti, V., Malicoeur-Martinelll A., and Pal, C. MCVD masked conditional video diffusion for prediction, generation, and interpolation. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 23371-23385. Curran Associates, Inc., 2022.
Wang. D., Li, C., Wen, S., Nepal, S., and Xiang, Y. Man-in-the-middu: attacks against machine learning classifiers via malicious generative models. IEEE Transactions on Dependable and Secure Computing, 18(5):1941 1901-0 September 2021.
Wang, Z., Zheng, H., He, P., Chen, W., and Zhou, M. Diffusion-Gar Training GANs with diffusion. arXiv preprint ar arXiv:2206.0220
2022.
Wibisono, A. and Wilson, A. C. On accelerated methods in optimization. arXiv preprint orXivv1509.036.66 2015.
Xiao, Z., Kreis, K., and Vahdat, A. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference on Learning Representations, 2022.
Yi, M., Zhu, Z., and Liu, S. MonoFlow: Rethinking divergence GANs via the perspective of W Wasserstein gradient flows. In Krause, A., Brunskill, E.. Cho. K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 39984 40000. PMLR, July 2023.
Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A. Self-attention generative adversarial networks. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Larning, volume 97 of Proceedings of Machine Learning Research, pp. 7354-7363. PMLR. June 2019.
15

---

A Further Discussion
In this section we develop a number of elements of our framework and models to provide more context and perspective on our contributions.
A.1 Extension to Stochastic Particle Models
We focus in the main paper on deterministic particle movement. However, some PMs like diffusion models also have equivalent stochastic particle movements as described Equations (4) and (7). We provide some evidence in this section that stochastic particle movement can also be integrated to Int-PMs, thereby strengthening the generality of our approach.
Generally, using the Pokker-Planck equation, the following equation shares the same probability path as both Equations (7) and (9), for any [0, 1]:
dW dæ; = 20' (t) log Pdata * KRBF (xx) dt - ad (t) ( de + 22a0
(21)
This corresponds to an interpolation between Equations (7) and (9) that trades Brownian noise with its deterministic equivalent.
This stochastic component can then be integrated into the formulation of interacting particle models, via Equation (13). This latter equation takes the directions followed by the particles in the particle model, and transforms them via the operator Ag. (z). Since this operator linear, is possible to integrate a stochastic component into the equation (Klebaner, 2012), allowing to take into account stochastic particle models:
dge, (=) = [Ao, (=)] (Vhp, dt + (r)(((WW)
(22)
where 7(t) is a scalar function of time.
This makes it possible integrate the stochastic component of diffusion models into Score GANs by interpolating between Gaussian noise and the score of the generated distribution in step 5 of Algorithm 1, similarly to the previous equation using a. Nonetheless, this comes with no guarantee on experimental performance, as we found in preliminary experiments that adding such a stochastic component is often detrimental to the resulting FID. Indeed, to succeed, the chosen gradient vector field to follow with the generator must be compatible with the generator architecture (i.e., compatible with the generator preconditioning Ao, (z)), which may not be the case with white noise of high variance.
A.2 Discretizing Continuous- Time Equations
Studying how discretizing the considered continuous-time phenomena could affect our formulations an interesting perspective for future work. We initiate a discussion on this topic below.
Choosing the best discretization method for diffusion models is challenging (Karras et al., 2022); since this depends the chosen (t), its efficiency is assessed w.r.t. the number of score queries instead of the number of discretization steps like numerical methods, and the final purpose of discretization (generating realistic data) differs from its initial purpose (approximating solution to a differential equation). Therefore, standard approaches like EDM rely on empirical discretization grids and custom solvers, tailored to the generation task. Our framework, by identifying the true probability flow in Equation (9), may help diffusion models cope with discretization errors through the score of the generated distribution.
The previous discussion, however, only holds for score-based diffusion models, for which the probability path is known in advance. This is not possible for other particle models, and studying the convergence properties of their discretizations, like Arbel et al. (2019) do for MMD, is non-trivial.
Adding generator is an alternative way to solve the underlying particle model differential equation. By generalizing the parallel between Wasserstein and Stein gradient flows in Section 3.3, generators can be seen in our framework as preconditioning over the particle model differential equation via the linear operator Ag, (z) in Equation (13). well-chosen architecture, adapted to the particle flow hh, may speed up and simplify the dynamics towards the data distribution.
16

---

A.3 Second-Order Methods and Adam
The framework introduce relies on first-order solvers for PMs and first-order optimization for Int-PMs. Yet, in practice, we use Adam (Kingma & Ba, 2015), second-order momentum based optimizer, train generators cf. Appendix D.2. Theoretically, is entirely possible formulate continuous-time equations for Adam, paving the way for a generalization of our results to second- order setting.
By fixing the values of (31, 12) and allowing the time step to approach zero, we can recover the continuous version of SignSGD. A relevant example of SignSGD's study within non-convex context was presented by Bernstein et al. (2018), Furthermore, exploring the scenario where non-interaction present (when Ag, disappears from Equation (13) as discussed Section 3.1) reveals a particle gradient flow with a renormalized gradient. This yields intriguing connections with continuous acceleration, as demonstrated by Wibisono Wilson (2015). We consider this avenue be promising direction for future investigation.
A.4 Time Efficiency of Score GAN and Diseriminator Flow
The design of Score GAN and Discriminator Flow induces computational constraints that make each of their training iterations slower than those baseline models. Score GAN requires pretraining a score network as specified in the paper, and its score-based update remains computationally more demanding than a discriminator-base update like GANs since the score function takes values in the data space, while the discriminator output is scalar. Training Discriminator Flow requires sampling at every step from the generating differential equation of Equation (20). This makes its training iterations slower than both diffusion models (which do not require resampling through the differential equation) and GANs (which have fast sampling).
Besides the cost of individual training iterations, the total temporal cost of training also depends on the number of iterations, which we specify Appendix D.3.
B Algorithmic Details
We detail in this section some aspects of the Score GAN and Discriminator Flow algorithms that were described at high level in Section 4.
B.1 Score GANS
Algorithm I involves two major steps: line performs denoising score matching for the gener- ated distribution, and line 5 updates the generator parameters using the resulting gradient flow in Equation (19). We describe some implementation tricks for these steps in the following subsections.
B.1.1 Denoising Score Matching
Denoising score matching was described in line 3 of Algorithm as originally used by Song Ernon (2019). Following best practices later introduced by Karras et al. (2022), instead of directly training a network se estimate the score, we instantiate and train a denoising network de using the following update:
AV de .0) =
(23)
where de is implemented U-Net (Ronneberger et al., 2015), with additional input- skip connections for better preconditioning (Karras et al., 2022). We then use de to compute the estimated score:
d% (I, σ)
{(,,, o)
(24)
We use the same tricks to pretrain the score model of the data distribution, Priata
B.1.2 Generator Training
Line 5 of Algorithm indicates how update the generator parameters, following Equation (11). In order to facilitate its implementation in deep learning frameworks, instead use the equivalent
17

---

Algorithm 3: Training iteration for Discriminator Flow (detailed). Input: Batch size B E N", number of steps N No, initial distribution Po. gradient strength E +. previous discriminator fa: R x -> RR.
Output: Updated discriminator fø.
for b = to B do
// In parallel // Initialization
x² ~ Pdata: 10 म.
// Partial generation do
for fori=00
// Solve Equation (20)
fo) #)]:
241N -
in ~ (10, N 1]): 1/ Train discriminator fs at the chosen random steps, ef. Equation (15)
// Select random step
t GA La fo fo(:;)) (fat//))) )e( u (f=")een..1))
weight update:
t 0 vo T StopGradient (paaan (go(2) te, <6(90(2) (00(:)))e,00)3
(25)
that is, optimize the loss (=) StopGradient (p + oe, - (g((((2) + to, oe,
We also adapt the weight update as follows. While keeping the same h1 as in Equation (19), we multiply in the weight update by (which amounts to changing the noise level sampling distribution Po accordingly). When implemented with a denoiser network as described in Appendix B.I.1, the score estimation (x, o)) can be numerically unstable for small as when using Equation (24). This explains why Karras al. (2022) consistently multiply scores by a, which is choice that follow for Score GANs. Furthermore, we can generalize Score GANs to other functionals hp encompassing the EDM formulation of Equation (9), right:
hp. = Ppaata log[Paata * kRBE] 1p loglp: * KRBF);
(26)
where Mpaata and μP constant hyperparameters (e.g., for EDM, Maasa and μ4p 1). Overall, in practice we implement the generator weight update of line from Algorithm as:
0 0 0 v σ go(2) " 'StopGradient (Mpancn & Pauta (go(z) + OE,0) (p0((10(2)) + +
(27)
B.2 Diseriminator Flows
In Algorithm 3 we provide detailed implementation of the high-level Algorithm 2 for Discriminator Flows, including how batching and differential equation discretization are handled. For the latter, we use the Euler method with uniform temporal grid, with N steps, in [0, 1]. While higher-order solvers and more optimal temporal grid choice could have been used, as EDM (Karras et al., 2022), we avoid using any refinement of time discretization for Discriminator Flows for the sake of simplicity. For eatch-paralle for all batch samples, and then select random time for each sample to compute the discriminator loss and update its parameters.
exccution on GPUs, we solve the differential equation of Equation (20) over [0, 1]
C Further Experiments
In this section we present additional experiments that will help the reader develop better intuition on the behavior of both Score GANs and Discriminator Flows in practice.
18

---

EDM
4
2
5
5
7
8
7
0
Score
Flw GAN GAN
9
2
0
4
1
9
7
O
9
2
1
2
2
6
5
Discr.
d
9
3
9
0
8
5
Figure 3:
Uncurated samples from studied models trained on CclebA and MNIST.
Figure 4: Uncurated samples of a Score GAN variant, which shows strong mode collapse on MNIST. Note that the Score GAN parameters were intentionally chosen obtain this behavior.
C.1 Additional Samples
Figure 3 shows additional samples for all four tested models. Furthermore, for better visualization pur- poses, our public repository https: / /gttttbt....tt/.g//g/ com wwwp:/hhtttttteteeeeeeeeeee gpm includes animated images illustrating the generation process for EDM and Discriminator Flows, as illustrated in Figure 1.
C.2 Mode Collapse on Score GANs
widely known issue for GANs, identified soon after their introduction, is known mode collapse (Goodfellow, 2016), which occurs when the generator only covers a fraction of the generated distribution. Interestingly,
observe the same phenomenon in Score GANs. We illustrate this with an extreme example in Figure 4, where we intentionally change parameters in our original model in order to induce the mode collapse issue. We induce mode collapse by choosing Mpaatta 2 and Mp = 1 in Equation (27) (i.e., EDM parameters instead of NCSN parameters in the original model),
Since mode collapse is absent from the generator-less, particle models that tested (the models on which Score GANs are based, and also Discriminator Flows), this observation suggests that mode collapse is primarily caused by the generator. This is in line with previous theoretical and empirical findings identifying the generator as the cause of mode collapse (Tanielian et al., 2020; Durr et al., 2022).
C.3 Latent Interpolations
We provide examples interpolations on MNIST for all four tested models in Figure We tested four interpolation methods on Gaussian priors considered by Leéniak et al. (2018): linear, spherical, Cauchy-linear and spherical Cauchy-linear. We reached same conclusion for each of these methods and thus only show the result of the most visually appealing one, spherical Cauchy-linear:
We notice that the generator-based models, Score GANs and GANs, show smoother transitions between generated images than the particle models EDM and Discriminator Flows, for which abrupt changes of digit identity and shape can be seen between consecutive interpolation steps. This confirms that interacting particle models (generator-based) can perform feature learning via their smaller latent space, allowing for smoother generation, while particle models operating the data space are less prone to such phenomenon. In this regard, Score GANs and Discriminator Flows are no different than their parent method in the same model category.
C.4 Alternating Updates of Generator and Score in Score GANs
There are K steps of score updates per generator update Score GANs, similarly to discriminators in GANs. Accordingly, K is important parameter in Score GANs. First of all, like in GANs, the tuning of K heavily depends the ratio between the learning rates of the score network and
19

---

7
7
7
7
7
7
7
7
7
2
7
9
1
4
4
4
7
9
7
7
3
5
5
5
5
5
7
7
7
5
9
9
7
7
7
7
7
9
8
9
2
6
6
6
6
6
6
4
4
/
8
8
8
3
3
3
3
3
7
8
8
3
6
6
5
5
5
5
5
4
4
4
0
(a)
EDM.
(b)
Discriminator Flow.
&
C
D
O
O
O
/
/
/
4
4
4
4
\
\
4
4
4
4
4
4
4
5
5
5
5
5
3
3
3
8
8
7
1
7
7
7
7
7
7
7
3
3
3
8
0
0
6
4
4
4
9
9
9
9
9
9
4
17
t7
0
0
0
0
9
9
9
S
(c) Score GAN.
(d) GAN.
Figure 5:
Interpolations (row-wise) in the latent space of the studied models (uncurated samples) at
regular intervals.
42.5
*triining loss
26
training loee
40.0
37.5
20
40
60
80
100
20
40
60
80
100
training iterations
training iterations
(a) At the 50 generator training iteration.
(b) At the 3200 generator training iteration.
Figure 6: Evolution on MNIST of the training loss in-between two generator updates the score of the generated distribution sg, for K 100 and equal score/ generator learning rates
17.
the generator (Jelassi al., 2022) cf. Algorithm 1. A higher ratio may allow to decrease the necessary number of steps K.
In our experiments on image data, we use r 2 2 (Table 7). However, to gain more intuition empirically, we performed set of experiments on MNIST with À such that r by making the number of steps K vary from 1 to 10. We obtained similar results across this range of values for K, close to the values reported in Table This indicates that even low values of can provide sufficient approximation of the score of the generated distribution.
To observe this gualitatively, plot in Figure 6 the evolution of the generator updates for K = 100. At the beginning of training, s% needs around 20 updates to converge. Yet, after a small number generator updates, it is already close to the optimum before its first update.
training loss in between
This confirms that the continuous update of the score of the generated distribution se. like a discrimi- nator, coupled with a learning rate ratio > 1, makes small number of score updates K between generator updates sufficient for adequate performance.
C.5 Time Efficiency of Discriminator Flows
The qualitative experimental results Section suggest that Discriminator Flows, by learning path towards the data distribution with discriminator, may converge faster towards Pdata than diffusion
20

---

10a
Discr. Flow (early stopped)
FID
EDM (Hem,, carly stopped)
EDM (Euler, early stopped)
10*
10
40
NFE
(a) Stopping the generative process of Equation (20) at an earlier time T" than in training (0 < T' < =
Discriminator Flow (early stopped)
Discriminator Flow (discretized)
EDM (Heun solver)
EDM (Euler solvez)
10a
10
20
30
40
50
60
NFE
Time efficiency comparison between alternative samplers of Discriminator Flows and EDM.
Figure 7: FID performance versus NFE (neural function evaluations, i.e., the number of times Vh, is queried produce a single image) for EDM and Discriminator Flow MNIST. We test different NFE modulation methods for the Discriminator Flow, order to evaluate the time efficiency of each method. We consider two standard differential equation solvers for EDM: Buler and Heun.
models, for which the path towards the data distribution is determined by the diffusion equation, which must be followed until the final time T'.
We quantitatively confirm this observation in Figure 7(a), which displays the performance of EDM and Discriminator Flow on MNIST for intermediate generated distributions Pt. < T, i.e., early stopping of the generation process. As expected given that intermediate generations from diffusion models are noisy, Discriminator Flows converge faster towards the data distribution. This raises the question of whether this faster convergence yield better efficiency for Discriminator Flows than for diffusion models. We investigate this claim the following.
Time efficiency in diffusion models is usually measured, as done by Karras et al. (2022), in terms of generative performance versus number of neural function evaluations (NFEs): the most costly operation is the repeated evaluation of the study the efficiency of EDM and Discriminator Flow using this metric.
network throughout the diffusion process. We then
Since Discriminator Flows are based on a discretized differential equation, we can measure the time efficiency of this model by decreasing the number of neural function evaluations (NFEs) in two ways: by using a larger time discretization of Equation (20) by an early stop of the generative process at time T" as already described above. For the first alternative, we train the evaluated model with 128 generative steps and 1; for the second one, we train the evaluated model with 64 generative steps and 1 = 2, i.e., with half as many generative steps as the first one but with a doubled velocity, c.f. Algorithm 3. We compare both alternatives against two standard inference methods for EDM
21

---

based on the (first-order) Euler solver (Kloeden & Platen, 1992). and the (second-order) Heun /EMM sampler (Karras et al., 2022). We show the results obtained for MNIST in Figure 7(b).
Overall, the Discriminator Flow model is comparable to the second-order version EDM; however, both remain outperformed by the first-order version of EDM. Surprisingly, the first-order version of EDM is actually more efficient in terms of NFE than the second-order version for low NFE values, This baseline was not considered by Karras et al. (2022), so this is new observation; we confirmed this phenomenon using the official implementation of EDM on 10 (Krizhevsky, 2009). This behavior especially visible in our case as we test on a simpler dataset.
Therefore, additional experiments are necessary to form a conclusion on the relative efficiency of Discriminator Flows w.r.t. diffusion models. We stress that the current results are achieved with Discriminator Flows discretizing Equation (20) on a regular temporal grid, while EDM discretize time for both first- and second-order methods using eustom temporal grid that improves discretization performance. Finally, we believe that further tuning of Discriminator Flows could significantly improve its efficiency: while the final image appears quickly after a few steps as illustrated in Figure I, some imperceptible residual noise still needs to be eliminated in the remaining steps. This noise prevents steady convergence towards Pdatai removing it one of the main directions of improvement for Discriminator Flows.
C.6 Experiments on Gaussians
As toy example, we train all considered models (Discriminator Flows, GANs, EDM, and Score GANs) on synthetic samples generated from a two-dimensional mixture of Gaussian distributions. We provide visualization of the particle evolution w.r.t. training and inference time in our repository https://gii thub. Wom/White-Lan Linee Lank/ggmmm.. gpm, as well as in Figures 8 to 12. See the figure captions for more information.
We again observe that Discriminator Flows converge faster than EDM towards the data distribution.
D Experimental Details
We provide all details in this section that are necessary to reproduce our experiments. Our Python source code (tested on version 3.10.4), based PyTorch (Paszke al., 2019) (tested on version 1.13.1), is available as open source at https: //g///gtttt github..
gpm.
D.1 Datasets and Evaluation Metric
MNIST. MNIST is standard dataset introduced in LeCun et al. (1998), with clear license the best of our knowledge, composed of monochrome images of hand-writter digits. Each MNIST image single-channel, of size 28 28. We preprocess MNIST images by extending them to 32 x 32 frames (padding each image with black pixels). in order to better fit as imputs and outputs of standard convolutional networks. We linearly scale pixels values so that they lie in [-1, 1]. MNIST is comprised of training and testing dataset, but no validation set; we create one for each model training by randomly selecting 10% of the training images.
CelebA. CelebA (Liu et al., 2015) is a dataset composed of celebrity pictures. Its license permits use for non-commercial research purposes. Each CelebA image has three color channels, and is of size 178 x 218. We preprocess these images by center-cropping each to a square image and resizing 64 x 64 with Lanczos filter. We linearly scale pixels values that they lie in [-1, 1].
Gaussians. Our Gaussian dataset is composed of a mixture of two-dimensional Gaussian dis- tributions with standard deviation 2,2 with means evenly spaced over circle of radius 5. Training, validation, and testing datasets all consist of i.i.d. samples from this mixture.
FID. Throughout the paper we use the Fréchet Inception Distance (FID, Heusel et al., 2017) to measure the generative performance of the models we consider. our code we use the PyTorch implementation of TorchMetrics (Skafte Detlefsen et al., 2022).
22

---

D.2 Hyperparameters
We summarize the model hyperparameters used during training in Tables 4 to See our code for more information. We further discuss some aspects of our implementation choices in the remainder of this subsection.
Networks. Beyond multi-layer perceptrons (MLP) and EDM's U-Nets, we use three kinds of model architectures: DCGAN (Radford et al., 2016), ResNet (Kang al., 2022) based SAGAN with self-attention (Zhang et al., 2019), and FastGAN (Liu et al., 2021). We adapt these models in the following ways:
For MNIST images, changed the first (respectively, last) convolution of the DCGAN discrimi- nator (respectively, generator) to adapt for 32 x 32 inputs (respectively, outputs).
We adapted FastGAN to operate images of size 32 x 32 and 64 64 by removing layers with higher resolutions.
We added a bias to the first convolutional layers of discriminators, which did not have one.
We enhanced these models so that they can accept vectorial embedding of time t for Discrimina- tor flows, by modulating most of their convolution outputs (before the activation) channel-wise with an affine transformation. The affine transformation parameters are the outputs of MLP of depth 2, with SiLU activations (Hendrycks & Gimpel, 2016) and a hidden width that is twice as large as the input time embeddings.
Final generator activation. Since image pixel values lie in [-1, 1]. we add a final hyperbolic tangent activation to the output of the generators operating image data.
Score GAN noise distribution Po- Since during training Score GANs mimic the inference pro- cedure of a diffusion model, we choose to sample during Score GAN training time following the schedule chosen by EDM at inference time (Karras et al., 2022). In practice, we choose a minimal value min. a maximal value max, and an interpolation parameter To sample from Pr. we first sample an interpolation value (10 1J), and then compute as:
( Tmax + a (
Jhin Jfax ))
(28)
Note that here denotes a scalar hyperparameter following the EDM notation, and not the generated distribution in the main paper.
Usual generative modeling tricks. For simplicity in our proof-of-concept experiments, we avoided using standard performance improvement tricks such as exponential moving average and truncation tricks (Brock et al., 2019), EDM sampling tricks (Karras al., 2022), or spectral normalization (Miyato et al., 2018).
D.3 Compute
two Nvidia V100 GPUs with CUDA 11.8. When using two V100 Nvidia GPUs, the training time of both Discriminator Flow and Score GAN models at most one day for the largest dataset (CelebA).
For all experiments
As means to evaluate each model's efficiency, we provide the approximate amount of time we used to train the tested models over MNIST one NVIDIA V100 GPU:
for Discriminator Flow, 24 hours:
for Score GAN, 10 hours (excluding the pretrained diffusion model);
for EDM, 6 hours;
for GAN, 1 hour.
23

---

Table 4: Chosen hyperparameters for Discriminator Flows for each dataset. GP stands for Gradient Penalty (Gulrajani al., 2017). IPM stands for Integral Probability Metric (Müller, 1997). BN stands for Batch Normalization (loffe & Szegedy, 2015).
Hyperparameters
Gaussians
MNIST
CelebA
GAN loss
Non-saturating vanilla
IPM
IPM
a (Equation (15))
log(1 sigmoid)
id
id
b (Equation (15))
log sigmoid
id
id
C (Equation (15))
log sigmoid
id
id
R (Equation (15))
GP
GP
GP strength
0.04
0.05
GP center
0
(Equation (1))
N(O0, ID)
(Algorithm 3)
20
[2. 1]
2
N (Algorithm 3)
[64, 128]
25
Network architecture
MLP
DCGAN
ResNet
Width
512
64
64
Activation
Leaky ReLU, negative slope of 00.2
Depth
BN
No
Initialization Initialization gain
Normal 0.02
Orthogonal 1.41
Normal 0.02
Time embedding type
Fourier
Frequency scale
16
Time embedding size
128
Batch size
128
128
64 200
Number of optimization steps
4000
200 000
Optimizer
Adam (Kingma & Ba, 2015)
Learning rate
0.0002
(81. (22)
(0.5, 0.999)
Model selection
Best validation FID
Validation frequency
1000
Number of validation samples
6000
1000
E
Broader Impacts
Our work aims to better understand recent generative modeling methods. As such, from a practical point of view, our work shares much of the impact of other work in this domain. While the models we propose do not yet have the quality required for broad application, generative models have wide range of potential positive and negative impacts. Some of the positive impacts include enabling faster and more accurate natural language processing (Li et al., 2022), improving automated tasks like summarization (Liu et al., 2018), and automating certain aspects of content creation (Nichol et al., 2022). However, generative models are also susceptible to producing undesirable output, such as unethical text, adversarial attacks (Wang et al., 2021), and malicious manipulation of data. These models also have the potential to exacerbate issues such as bias and discrimination (Lucy & Bamman, 2021), enabling the creation of more effective fake text and videos, and expanding the scope and complexity of cyberattacks that can be launched by malicious actors (Seymour & Tully, 2018). For thorough discussion the potential dangers of deepfakes, see Fallis (2021).
systems
Discussions and debates around the responsible use of generative models and the potential broader impacts of deploying them more widely are still ongoing. We hope that our principled framework aimed at improving our understanding of generative models will contribute to these discussions and to better control of such models.
24

---

Table 5: Chosen hyperparameters for EDM for each dataset. Cf. Karras al. (2022) and code for more details.
Hyperparameters
Gaussians
MNIST
CelebA
min
0.002
max
40
Odata
0.5
Equation & Solver
Heun solver the deterministic ODE of Equation (8)
Number of solver steps
7
32
25
Network architecture Width
MLP 512
EDM
EDM
128
Number of residual blocks
1
2
Dropout
0.13
0.1
Depth
4
Activation
Leaky ReLU, slope of -02
Uniform Kaiming for convolutions, unit weight and zero bias for group normalization layers, otherwise PyTorch default
Initialization
Time embedding type Frequency scale Time embedding size
Fourier 16
Positional
Positional
128
256
256
Batch size Number of optimization steps
128
128
64
10 000
500 000
100 000
Optimizer
Adam (Kingma & Ba, 2015)
Learning rate
0.0002 (0.9, 0.999)
(81, 82)
Model selection
25

---

Table 6:
Chosen hyperparameters for GANs for each dataset. Hinge refers to Hinge GANs (Lim &
Ye, 2017).
Hyperparameters
Gaussians
MNIST
CelebA
GAN loss
Non-saturating vanilla
Hinge
Number of discriminator
steps per generator update
Latent space size
128 (0, Ia)
Ps (Definition 2)
Discriminator architecture
MLP
DCGAN
FastGAN
Width
512
64
32
Activation
Leaky ReLU, negative slope of -0.2
Depth
4
BN
No
Yes
Yes
Generator architecture
MLP
DCGAN
FastGAN
Width
512
64
32
Leaky ReLU,
Leaky ReLU,
Activation
ReLU
slope of -0.2
slope of -0.2
Depth
BN
No
Yes
Yes
Initialization Initialization gain
Normal 0.02
Batch size
128
128
64
Number optimization steps
1900
10.000
100
Optimizers
Adam (Kingma & Ba, 2015)
Learning rate
0.0002 (0.5, 0.999)
(31, 32)
Model selection
Best validation FID
Validation frequency
1000
Number of validation samples
6000
1280
26

---

Table
Chosen hyperparameters for Score GANs for each dataset.
Hyperparameters
Gaussians
MNIST
CelebA
min (Appendix D.2)
0.1
0.32
0.32
max (Appendix D.2)
10
40
40
(Appendix D.2)
3
K (Algorithm
10
Latent space size
128
P: (Definition 2)
(0, La)
Data score sii
EDM from Table 5
Gen. score si architecture
MLP
EDM
EDM
data Width
0.5
512
64
128
Number of residual blocks
2
Dropout
0.13
Depth
4
Leaky ReLU,
Activation
slope of -0.2
Initialization
EDM from Table 5
Generator architecture
MIP
DCGAN
FastGAN 32
Width
512
64
Activation
ReLU
Depth
4 No
BN
Yes Normal
Yes Orthogonal 1.41
Initialization
py Torch default
Initialization gain
0.02
Batch size Number of generator
128
256
32
3150
100
150 000
optimization steps Optimizers
Adam (Kingma Ba, 2015)
Score learning rate Generator learning rate (31,82)
0.0002
0.001
0.0004
0.0002
(0.9, 0.999)
Model selection
Best validation FID
Validation frequency
2500
Number of validation samples
6000
27

---

40-
10-
20-
-30
20
(a) Initialization (sampling from P0 m.
(b) 7
(c) T..
(d) f T.
23
t 47.
(l) T.
(h) T.
Figure 8: Sampling steps for EDM on a Gaussian mixture; 128 samples are shown for both the generated and the data distributions. The second- solver was used with 13 NFEs. Arrows show the gradients Vh,, associated with each generated sample, corresponding the direction provided by the score function, cf. Equation (8).

---

150
30
P
-3
(b) 11--4
37 10=110 (0)
L4 1-11111 (d)
(a) Initialization (sampling from Pu m)
29
80
(e) = 47.
(f) T.
gg) T
(h) T.
Sampling steps for Discriminator Flows on Gaussian mixture; cf. Figure 8. Our chosen discretization yields 14 NFEs. The colored background represents the pointwise generator loss function -hp- -co fpr (darker is lower, renormalized for every snapshot), from which the particle gradients, shown in the figures, are derived.
Figure 9:

---

15
0.
22
19.
-15
(a) Initialization (po- goo A
(b) (b) T.
(c) t- T.
(d) t= T.
30
Poo
(e) 47.
(l) f T.
gg t T.
(h) T.
Training snapshots of a GAN on Gaussian mixture; cf. Figure 8. Here time represents training time, and T is the end of training. The colored backgrounds represent the pointwise generator loss function -hp -co fp. (darker lower, renormalized for every snapshot), from which the particle gradients, shown in the figures, are derived and then fed to the generator following Equations (11) and (13).
Figure 10:

---

-2 -1
(a) Initialization (po = go. Ppa)
(b) t T.
(c) t= T.
(d) t T.
31
(e) t= 4T.
(l) f T.
T..
(h) t0 T.
Figure 11: Training snapshots of Score GAN on Gaussian mixture; cf. Figure 8. Here time represents training time, and the end of training.

---

(a) Initialization (po goo ).
(b) T.
(c) T.
32
(e) t 47.
(l) T.
ST.
(h) T.
Figure 12: Training snapshots of a Score GAN on a Gaussian mixture, identical to Figure 11, but with generated samples perturbed by Gaussian noise of standard deviation o 0.2. Arrows show the gradients Vh,, received by the generated particles this noise level. corresponding Equation (19), split into the data score (in blue) and minus the score of the generated distribution (in yellow). They are then fed to the generator following Equations (1 and (13).


Reviews:
[{"review_id": "BhvmiKHRad", "reviewer": "Reviewer_AQyb", "summary": "This work presents a unified generative particle model framework that explains generative adversarial networks (GANs) and score-based diffusion models.\nThis model is characterized by an evolution equation with a functional  ${h_{\\rho}}_t$ that captures the transitioning over the time index $t$. \nThis work explains that the generation/inference of score-based diffusion models and the training of generators for GANs follow this model.\nThese particle models allow the decoupling of gradient flow in the score-based models and the generator in the GANs.\nThis led to the two new models: \n(1). Scored-based GANs with generators that trained with score-based gradients instead of using adversarial learning. \n(2). GANs that train without a generator. Only the discriminator synthesizes samples. \n", "questions": "(1). The following phrase is not clear:  Line 51 on Page 2:  \"In prior work, ...\". It is not clear if this is referring to a specific work or prior works.\n \n(2). It is not obvious that the modeling of the generation/inference with score-based models is interchangeable with training generators. Please can you provide more information on this?\n\n(3). Please provide more details on the equivalence of the interacting particle model of GANs and the diffusion models as stated in Finding 6.\n\n(4). Is it possible to come up with a hybrid structure that combines the two main techniques under the unified PM model?  If yes, do you think this could improve the performance? ", "limitations": "The author(s) addressed the limitation of their work.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-organized.\nIt also cited relevant works in a way that makes it easy to understand the main point.\nThe work is also important because it offers a new perspective into the workings of two main classes of generative AI models.\nThis opens the door to understanding and ultimately making beneficial adjustments to these models. ", "weaknesses": "Some of the underlying assumptions and claims are not clear.\nThe particle model for score-based models represents the generation/inference of the models.\nThe interacting particle model for generator-based models is the training of the generator.\nIt is not clear if the functional $h_{\\rho}$ in both cases are interchangeable.\n\nAlso, it is not clear how you arrived at Finding 6. \nPlease provide more details on the equivalence of the interacting particle model of GANs and the diffusion models.\n \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address the weaknesses they raised by answering their questions below.\n\n## (1). Wording Clarification\n\n> (1). The following phrase is not clear: Line 51 on Page 2: \"In prior work, ...\". It is not clear if this is referring to a specific work or prior works.\n\nThis is referring to prior works, plural. We meant that the choice of a prior that is easy to sample from is a widely adopted choice in the literature.\n\n\n## (2). and (3). Equivalence Between Particle Models and GANs\n\n> (2). It is not obvious that the modeling of the generation/inference with score-based models is interchangeable with training generators. Please can you provide more information on this?\n> (3). Please provide more details on the equivalence of the interacting particle model of GANs and the diffusion models as stated in Finding 6.\n\nWe do not claim that particle model (PM) inference and generator training are equivalent or interchangeable. Instead, we show that interacting particle models (Int-PMs), and in particular GANs, ***generalize*** PM inference during generator training, as worded in Findings 4 and 6. Indeed, the particle evolution of an Int-PM (Eq. (13)) reduces to the particle evolution of a PM (Eq. (1)) with the same functional $h_{\\rho}$ when $k_{g_{\\theta_t}}(z, z') = \\delta_{z - z'}$ (lines 152 - 158), in which case $\\mathcal{A}_{\\theta_t}(z)$ disappears from Eq. (13) to obtain Eq. (1).\n\nThis indicates that a functional $h_{\\rho}$ used in an Int-PM may be used in a PM and vice versa. However, note that this does not guarantee successful performance, as the compatibility of the generator with $h_{\\rho}$ (through $\\mathcal{A}_{\\theta_t}(z)$) should also be taken into account.\n\nWe confirm the possibility of transferring $h_{\\rho}$ with the models we introduce: the PM Discriminator Flow is based on an Int-PM GAN functional, and the Int-PM Score GAN is based on an PM score functional.\n\nFurthermore, this is supported by Finding 6 where we notice that functionals $h_{\\rho}$ from typical PMs have been shown to be encompassed in the Int-PM GAN framework. Under some hypotheses on the discriminator, Yi et al. (2023) show that the $h_{\\rho}$ of $f$-divergence GANs is actually the functional of the forward KL divergence gradient flow (like in the NCSN diffusion model of Song et al. (2019)), and Franceschi et al. (2022) show that it is the one of the squared MMD gradient flow. Considering Finding 4, this therefore means that GANs generalize these PMs.\n\nNonetheless, we agree that the formulation of Finding 6 should be more specific. We will clarify our explanations in Section 3 and specify exactly which score models and Wasserstein gradient flows are generalized by GANs based on the previous paragraph.\n\n## (4). Hybrid Models\n\n> (4). Is it possible to come up with a hybrid structure that combines the two main techniques under the unified PM model? If yes, do you think this could improve the performance?\n\nThe introduced Score GAN and Discriminator Flow in Section 4 are examples of combinations of such techniques, as explained above: the PM Discriminator Flow is based on an Int-PM GAN functional, and the Int-PM Score GAN is based on an PM score functional. However, they are only proof-of-concept examples. As highlighted in the conclusion, we believe that finding a hybrid model that outperforms both diffusion and GANs is an interesting direction for future work regarding our framework."}, {"review_id": "CLLU33KPKh", "reviewer": "Reviewer_HYQ5", "summary": "This paper proposes a framework, i.e., a generalization of particle models, that unifies particle-based generative models like score-based diffusion and gradient flows with adversarial generative models like GANs, where generator training is framed as interacting Particle Models. Based on this perspective, the paper introduces two new hybrid models Score GANs (using score model to train generator) and Discriminator Flows (as prooof of concept. Experiments on MNIST and CelebA confirm the viability of the new framework.", "questions": "+ Why both $x \\sim p_{data}$ and $x = g_θ(z)$ exists in Algorithm 1?", "limitations": "Yes, the authors talked about the limitations of models proposed, and I believe these models are more of proof of concept. The authors also discussed social impact in Appendix.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The findings from mathematical derivation seem to be interesting.\n+ The writings are mostly clear and sound, well-supported by mathematical derivations and experiments.", "weaknesses": "+ The claim that \"Discriminator Flows have the expected advantage of converging faster to the target distribution as compared to the\nstate-of-the-art diffusion model EDM, and thus have higher efficiency during inference.\" may be overstated, as s Figure 5 in the Appendix demonstrates that EDM (Heun solver) exhibits comparable efficiency. Which solver is employed for EDM in Figure 1?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address the questions they raised below.\n\n> The claim that \"Discriminator Flows have the expected advantage of converging faster to the target distribution as compared to the state-of-the-art diffusion model EDM, and thus have higher efficiency during inference\" may be overstated, as Figure 5 in the Appendix demonstrates that EDM (Heun solver) exhibits comparable efficiency. Which solver is employed for EDM in Figure 1?\n\nIn Figure 1, we used the Heun solver for EDM. However, Figure 1 does not correspond to the setting of Figure 5. In Figure 5, we studied two different ways to reduce the number of NFE with Discriminator Flows: by discretizing the generating differential equation of Eq. (20) -- Figure 5(a) -- and by stopping the generation process early, like in Figure 1 -- Figure 5(b). We compared these two versions to the canonical way of reducing NFE in a score-based diffusion model, i.e. by discretizing the generating differential equation of Eq. (8) like Karras et al. (2022), either with Euler or Heun solver. Those are the two baselines identically reproduced in both Figures 5(a) and 5(b).\n\nWe included in the one-page PDF attached to our global response, in Figure 1(a), the merged plots of Figures 5(a) and 5(b). To compare directly performance vs NFE in the same setting as in Figure 1 in the paper, we evaluated EDM in early stopping similarly to Discriminator Flows in the one-page PDF attached to our global response, Figure 1(b). As expected, diffusion models need to completely denoise the image to achieve reasonable performance.\n\nNevertheless, after noticing an unrelated mistake in our experimental setting, we unfortunately have to retract the claim questioned by the reviewer. Please refer to the global response for more details.\n\n> Why both $x \\sim p_{\\mathrm{data}}$ and $x = g_{\\theta}(z)$ exists in Algorithm 1?\n\nWe apologize for the mistake. $x \\sim p_{\\mathrm{data}}$ should not appear in Algorithm 1. We will correct this in the updated version of the manuscript."}, {"review_id": "txi4r8ew6X", "reviewer": "Reviewer_kQU7", "summary": "The paper proposes a framework that unifies particle-based deep generative models, such as gradient flows and score-based diffusion models, and adversarial generative models, such as GANs, by framing generator training as a generalization of particle models. The authors suggest that a generator is an optional addition to any such generative model and show how it is possible to train a generator with score-based gradients which replace adversarial training and to train a GAN without a generator, using only the discriminator to synthesize samples. The paper presents two new models, a Score GAN and a Discriminator Flow, as proofs of concept to illustrate the viability of these ideas.", "questions": "- In line 29, what does it mean that the generator has the role of a smoothing operator?\n- What is the time cost of training Score GAN and Discriminator Flows? Do they have a greater training overhead than Score-based Model and GAN? ", "limitations": "No, the authors have not adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Overall, the paper is well-organized and well-written.\n- This paper investigates the two most prominent classes of generative models in the field, i.e., GANs and diffusion models, and provides a unified framework that explains their similarities. This work is valuable as it has the potential to inspire future research efforts to unify these two classes of generative models in addressing their respective problems.\n- This paper has a relatively strong theoretical foundation and meets the standards of the conference.", "weaknesses": "- Some mathematical symbols (e.g., $\\nabla_W$) require further explanation to enhance the readability of this paper for a wider range of readers. \n- The experiments in this paper are relatively weak, with a limited number of datasets and tests conducted. Furthermore, according to Table 3, the proposed methods (Discriminator Flow and Score GAN) fail to surpass baseline methods (EDM and GAN).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address the weaknesses and questions they raised below.\n\n## Wasserstein Notation\n\n> Some mathematical symbols (e.g., $\\nabla_W$) require further explanation to enhance the readability of this paper for a wider range of readers.\n\nThe notation $\\nabla_W$ is a standard notation for readers with expertise in Wasserstein gradient flows, but it may indeed be confusing for a wider range of readers. For a functional $\\mathcal{F}$ defined over probability measures, $\\nabla_W \\mathcal{F}(q)$ is the so-called Wasserstein gradient of the functional -- similar to the gradient of a functional defined over a Euclidean space. It is defined as $\\nabla_x \\frac{\\delta \\mathcal{F}}{\\delta q}$ (Eq. (3)), i.e. the Euclidean gradient of the functional’s first variation $\\frac{\\delta \\mathcal{F}}{\\delta q}$. We refer to Santambrogio (2017), cited in the paper, for more information.\n\n## Experimental Results\n\n> The experiments in this paper are relatively weak, with a limited number of datasets and tests conducted. Furthermore, according to Table 3, the proposed methods (Discriminator Flow and Score GAN) fail to surpass baseline methods (EDM and GAN).\n\nProviding state-of-the-art performance in our experimental results is beyond the scope of our paper. We include the experimental results in our paper primarily for proof-of-concept purposes, to validate that our proposed Discriminator Flow and Score GAN models are practically feasible. Please also refer to our global answer on this topic.\n\n## Generator as a Smoothing Operator\n\n> In line 29, what does it mean that the generator has the role of a smoothing operator?\n\nThis role of the generator is formally defined and explained in Definition 2 (lines 148-151). We note that compared to the particle model (PM) given in Definition 1 and Equation 1, the use of a generator results in applying a linear operator on top of the vector field $\\nabla h_{\\rho_t}$. It is this linear operator role that provides the intuition that the generator acts as a smoothing operator (analogous to a smoothing filter). Indeed, we notice in line 151 that this linear operator is a kernel integral operator, which by design smoothes the input signal $V$ with a kernel convolution in Eq. (14).\n\n## Training Costs\n\n> What is the time cost of training Score GAN and Discriminator Flows? Do they have a greater training overhead than Score-based Model and GAN?\n\nIndeed, the design of Score GAN and Discriminator Flow induces computational constraints that make each of their training iterations slower than those of baseline models.\n\nScore GAN requires pretraining a score network as specified in the paper, and its score-based update remains computationally more demanding than a discriminator-based update like in GANs (since the score function takes values in the data space, while the discriminator output is scalar).\n\nAs explained in lines 270 and 287, training Discriminator Flow requires sampling at every step from the generating ODE of Eq. (20). This makes its training iterations slower than both diffusion models (which do not require resampling through the ODE/SDE) and GANs (which have fast sampling). \n\nBesides the cost of individual training iterations, the total temporal cost of training also depends on the number of iterations which we specify in Appendix C.2. Overall, the approximate amount of time we used to train the tested models over MNIST on one NVIDIA V100 GPU are:\n- for Discriminator Flow, 24 hours;\n- for Score GAN, 10 hours (excluding the pretrained diffusion model);\n- for EDM, 6 hours;\n- for GAN, 1 hours.\n\nWe will highlight the elements of this discussion in the updated version of the manuscript.\n\n## Limitations and Societal Impact\n\n> No, the authors have not adequately addressed the limitations and potential negative societal impact of their work.\n\nWe would like to point out that **we did address potential negative impacts** of our work in the supplementary material, Appendix Section D.\n\nFurthermore, **we did address limitations** of our approach throughout the paper, including:\n- the generative performance of the introduced models, lines 279 to 282;\n- the training cost efficiency of the introduced models, lines 241, 270 and 287;\n- the omission of continuous training of the GAN discriminator, lines 169 and 342-343.\n\nNonetheless, we welcome specific suggestions from the reviewer to improve this part of the paper."}, {"review_id": "9wN9TrlTkT", "reviewer": "Reviewer_XiV8", "summary": "The authors interpret GANs through the lens of particle optimization and show the links to score-based diffusion models. Based on these theoretical results, they propose to either 1) do particle learning (without a generator) based on the flow of a  discriminator, or 2) learn a generator by minimizing the score of the generator from the real data score.", "questions": "line 21: 'challenge the conventional view that particle and adversarial generative models are opposed to each other': the Discriminator can be used to learn particles (learn x_fake directly); this is well known as far as I know.\n\n'Score GANs': I would encourage trying to find another name that is a bit more explicit about your method; it's a bit too generic, and other papers have used the same name.\n\nI mentioned some things like extended analyses in the 'Weakness' section.", "limitations": "The only limitation mentioned is that the generator-less method is slower. They don't mention many limitations. They say the results may be poor because they are too novel and have yet to be fine-tuned. I don't consider those enough.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "They show interesting links between score-matching and particle-based GANs. They test some interesting ideas. They try different solvers for the particle-based GANs.", "weaknesses": "Particle GANs are not very novel; people don't use them frequently because its hard to optimize in high-dimensional space; this is not really new, but as far as I know, its rarely done, so this could still be interesting if it was really investigated deeply which they do not do (e.g., they don't verify the effect of different initializations, trying different optimizers, etc.). I know many papers study post-training flow/Langevin methods (e.g., Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling), but this is slightly different from particle optimization. The score-based GANs might be novel, but I am not 100% certain.\n\nResults are much worse for the disc-flow method compared to regular GANs and Diffusion and also the best GAN-Hybrid methods (e.g., Tackling the Generative Learning Trilemma with Denoising Diffusion GANs). It is well known that optimizing directly in the particle-space is difficult in high dimensions (I sadly do not remember those prior works, but I remember reading this), so this is not surprising.\n\nResults are much worse for the ScoreGAN method compared to regular GANs and Diffusion.\n\nResults are very limited, with only MNIST and CelebA and 2 baseline methods. Maybe the proposed approaches are worse because the architectures are not that good, or maybe minimizing the score-function loss is not enough; more in-depth analyses using different optimizers, initializations for the particles, and architectures on slightly bigger datasets would add a lot. \n\nIn the related work, I would consider adding papers that combine GANs with diffusion or score-based models. (These are some I could come up with, but a few more exist)\nDiffusion non-score-based:\n- Diffusion-GAN: Training GANs with Diffusion (2022)\n- Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (2022)\nScore-based:\n- Adversarial score matching and improved sampling for image generation (2020)\n\nOverall: Weak results, limited novelty, limited experimentation to try to make these approach perform better than their current performance (worse than every baseline methods).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback. We address the weaknesses they raised below.\n\n## Novelty\n\n> Particle GANs are not very novel; [...] this is not really new, but as far as I know, its rarely done\n\n> line 21: 'challenge the conventional view that particle and adversarial generative models are opposed to each other': the Discriminator can be used to learn particles (learn x_fake directly); this is well known as far as I know.\n\nOur understanding is that the reviewer questions the novelty of Section 3.2 and of the introduced Discriminator Flow model. However, they did not provide any reference to support their assertion, so we have no choice but to respectfully maintain our claims. We welcome any suggestion to improve our positioning w.r.t. related work that we may have missed.\n\nMoreover, we would like to point out that we did appropriately cite and build upon prior works studying the particle model aspect of GANs, albeit not as general as our approach: e.g., Franceschi et al. (2022) on line 162, Yi et al. (2023) on line 183, as well as Chu et al. (2020) and Durr et al. (2022) on line 196.\n\nIn particular regarding the novelty of Discriminator Flows, the closest model we are aware of is the one of Heng et al. (2023). We notice in lines 305-306 that their model is a particular case of Discriminator Flows, although the original article did not make the link with discriminators and GANs.\n\n## Experiments and Performance\n\n> Results are much worse for the disc-flow [and the ScoreGAN] method compared to regular GANs and Diffusion and also the best GAN-Hybrid methods [...].\n\n> Results are very limited, with only MNIST and CelebA and 2 baseline methods. [...] more in-depth analyses using different optimizers, initializations for the particles, and architectures on slightly bigger datasets would add a lot.\n\nWe understand that better performance and more thorough experiments could only improve the paper and support our framework. However, the reviewer's focus on state-of-the-art experiments differs from the paper's scope which is mostly theoretical. In our support, we recall the following statement from Reviewer iGU3:\n> **Reviewer iGU3:** *As acknowledged by the authors as well, the models are not really contenders for the state of the art at the moment. It’s possible (but by no means clear) that they could be significantly improved by a deep dive into engineering the design choices, but at the moment they mostly serve as a reasonable basic validation of the theory.*\n\n## Optimization in the Particle Space\n\n> people don't use [particle GANs] frequently because its hard to optimize in high-dimensional space\n\n> It is well known that optimizing directly in the particle-space is difficult in high dimensions (I sadly do not remember those prior works, but I remember reading this)\n\nWe respectfully disagree with the reviewer. We know that gradient descent in particle space is not inherently flawed: after all, diffusion models are also particle models and they are state-of-the-art in image generation (Dhariwal & Nichol, 2021). Furthermore, Heng et al. (2023) (with their model which we show is a special case of Discriminator Flows) and Fan et al. (2022) also present particle models with competitive performance.\n\nIt is indeed known that *some* particle models suffer from the the curse of dimensionality, but not all. How the curse of dimensionality affects gradient descent of a functional in the particle space is closely related to how the curse of dimensionality affects the functional itself. Indeed, if the sample complexity of the functional scales badly in the dimension, it is expected that the resulting gradient flow also inherits this curse. A prototypical example of this is given by the 1-Wasserstein distance (Dudley, 1969), which the reviewer may have thought about. However, in our case, the functional is (implicity) implemented by a neural network, known to be particularly effective in high dimensions. This explains why particle models such as ours can be effective even in high dimensions.\n\nR. M. Dudley. The speed of mean Glivenko-Cantelli convergence. Ann. Math. Statist. 1969.\n\n## Related Work\n\nThe reviewer suggests that we discuss the following papers:\n1. Wang et al. Diffusion-GAN: Training GANs with diffusion. arXiv, 2022.\n2. Xiao et al. Tackling the generative learning trilemma with denoising diffusion GANs. ICLR 2022.\n3. Jolicoeur-Martineau et al. Adversarial score matching and improved sampling for image generation. ICLR 2021.\n\nWe thank the reviewer for the suggestion. We would like to point out that **we already cite 1. in Section 4.4, line 299** as strongly linked to Score GAN. **We also cite 2. in our introduction**. We will further discuss 2. and 3. in the next version of the manuscript.\n\nMore specifically, 2. and 3. are not directly related to our unified view of score-based diffusion and GANs as particle models. 2. trains several GANs to successively denoise an image, similarly to diffusion. 3. augments the denoising objective of score-based diffusion models with an adversarial objective to improve the denoising image quality. None of these references leverage the link we make between GANs and particle models.\n\n## Limitations\n\n> The only limitation mentioned is that the generator-less method is slower. They don't mention many limitations. They say the results may be poor because they are too novel and have yet to be fine-tuned. I don't consider those enough.\n\nAs most of the reviewers noticed (AQyb, HYQ5, iGU3 and 6P1y), we addressed the limitations of our approach in the paper, including but not limited to the ones the reviewer stated in this quote. We would welcome any suggestion of additional limitations to discuss from the reviewer.\n\nBeyond this, we would also like to point out that we addressed potential negative impacts of our work in the supplementary material, Appendix Section D."}, {"review_id": "8kBotGpsTE", "reviewer": "Reviewer_iGU3", "summary": "The paper points out commonalities between diffusion and GAN models by examining the time evolution of the sample particles. For diffusion, this evolution happens at inference time by following the theory-given flow that transports the initial noise distribution to match the data distribution (in practice this flow is memorized by a neural network in the training stage). For GANs, it happens during training: at any given training step, the current generator induces positions for the particles, and they “slide down” the discriminator surface (which likewise jointly evolves to respond to the new positions). The paper formulates these modalities into a common framework of (interacting) particle models.\n\nBased on this insight, the paper presents two practical algorithms, showing that either the discriminator or the generator can be removed and replaced with other diffusion-inspired mechanisms, while maintaining some GAN-like properties. The Score-GAN aims to follow the dynamics of a particle model in its training iterations by replacing the discriminator with an on-the-fly learned score that is combined with a pre-trained score of the noised data (as such, the model is not really a fully independent generative method). The Discriminator Flow drops the generator, and trains a time-varying discriminator, such that sliding initially white noise particles along its gradient brings them closer to the data.", "questions": "How do latent interpolations (in z-space) with the proposed algorithms behave? Do you find a similar smooth morphing behavior as with standard GANs?\n\nAlgorithm 1 is, ultimately, a somewhat loose implementation of the theory: the estimated score is not trained to completion, expectation is evaluated stochastically, etc. I assume you also use something like Adam for training the generator with the update on Line 5 (?) Do you have any insight on whether the theory can accommodate some of these practicalities, or does it come down to intuition and empirics in the end? Are things like mode collapse a consequence of inexact adherence to theory?\n\nAre these typos or such?\n  - Eq. 14, should it be z’ in a couple of places instead of z?\n  - Algorithm 1, you have x ~ p_data but then “overwrite” it with x = g(z), in two places – if I understood correctly, you shouldn’t be sampling from data distribution at all here?\n  - Algorithm 1, line 5, again confusing wrt. line 4: x or x^sigma isn’t even used, alternatively it seems like you should instead have a noise addition on line 5 (like in appendix)\n  - Algorithm 2 and Eq. 20, are there some gradient symbols missing around f? Isn’t it the gradient of c(f()) that you follow?", "limitations": "The paper acknowledges reasonably well various shortcomings of the methods, and the fact that the results are far from state of the art. There is a discussion of the broader impacts in the appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I am not extensively familiar with the literature in this line of work, so I don’t have strong opinions about the magnitude of the novel insights in this paper. Personally I found it quite interesting and thought provoking, and as far as I can tell, theoretically sound.\n\nThe authors have put a good amount of effort into validating the theory and ensuing algorithms in practice. While the results are not high-quality by modern standards, the two models are somewhat intriguing, and could inspire further research. The combination of scores and reintroduced freedom for the network to decide on the paths, shape of the latent-to-image mapping, etc., might open up some interesting avenues, such as the speculated distillation use case.\n\nThe paper is polished and pleasant enough to read. However, there might be some miscellaneous errors in how the formulas are written (see Questions below), so going through to double check the rest of them once more could be helpful.", "weaknesses": "As acknowledged by the authors as well, the models are not really contenders for the state of the art at the moment. It’s possible (but by no means clear) that they could be significantly improved by a deep dive into engineering the design choices, but at the moment they mostly serve as a reasonable basic validation of the theory.\n\nThe Score-GAN approach seems to be limited in that it requires a pre-trained score function for the data. This seems to rule out the most common usage scenario of straight-out generation. Of course, one could argue that training that model is the first stage of the method, but at that point one could use it in standard diffusion. The actual algorithm seems to then essentially be distillation into a single-step method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback.\n\nWe concur with the reviewer regarding the weaknesses raised, which are acknowledged in the paper. Notably, we mention that Score GAN can serve as a distillation method in the conclusion. We address the reviewer's questions below.\n\n## Latent Interpolations\n\nWe provide examples of interpolations on MNIST for all four tested models in the one-page PDF attached to our global response, Figure 2. We tested four interpolation methods on Gaussian priors considered by Leśniak et al. (2019): linear, spherical, Cauchy-linear, and spherical Cauchy-linear. We reached the same conclusion for each of these methods and thus show in the additional document the result of the most visually appealing one, spherical Cauchy-linear.\n\nWe notice that the generator-based models, Score GAN and GAN, show smoother transitions between generated images than the particle models EDM and Discriminator Flow, for which abrupt changes of digit identity and shape can be seen between consecutive interpolation steps. This confirms that interacting particle models (generator-based) can perform feature learning via their smaller latent space, allowing for smoother generation, while particle models operating in the data space are less prone to such phenomenon. In this regard, Score GAN and Discriminator Flow are no different than their parent method in the same model category.\n\nWe will add this discussion in the appendix of the next version of our paper.\n\nLeśniak et al. Distribution-Interpolation Trade off in Generative Models. ICLR 2019.\n\n## Theory vs Practice in Algorithm 1\n\nWe address the points raised by the reviewer below. As a preamble, we would like to notice that these remarks also hold for GANs, regarding the alternating updates of the discriminator and the generator, the use of Adam, and the stochasticity of expectation estimation via mini-batch training. This is then no suprise that Score GANs and GANs share properties like mode collapse, as pointed out in lines 285-286.\n\n### Estimation of the Score\n\n> the estimated score is not trained to completion\n\nWe would like to highlight that this is only true at the very beginning of training of Score GANs. Indeed, the score function is continuously optimized as its weights are carried on after each generator update, like a discriminator in GANs.\n\nTo confirm this, we added an analysis of the training loss of the score function in the one-page PDF attached to our global response, Figure 3. In this experiment, we perform $K = 100$ discriminator steps in-between generator updates. At the beginning of training, the score function needs a few updates to converge. However, after a low number of generator updates, it is already close to the optimum before its first update.\n\n### On Adam\n\n>  I assume you also use something like Adam for training the generator with the update on Line 5 (?) Do you have any insight on whether the theory can accommodate some of these practicalities, or does it come down to intuition and empirics in the end?\n\nWe thank the reviewer for the interesting question. In fact, we do use Adam in practice. Theoretically, it's entirely possible to formulate continuous-time equations for Adam. By fixing the values of $(\\beta_1, \\beta_2)$ and allowing the time step to approach zero, we can recover the continuous version of SignSGD. A relevant example of SignSGD's study within a non-convex context was presented by Bernstein et al. (2018). Furthermore, exploring the scenario where non-interaction is present (where $\\mathcal{A}_{\\theta_t}$ disappears from Eq. (13) and described in lines 156-158) reveals a particle gradient flow with a renormalized gradient. This yields intriguing connections with continuous acceleration, as demonstrated by Wibisono & Wilsan (2015). We consider this avenue to be a promising direction for future investigation.\n\nA. Wibisono & A. C. Wilson. On accelerated methods in optimization. arXiv, 2015.\\\nJ. Bernstein et al. signSGD: Compressed optimisation for non-convex problems. ICML 2018.\n\n### Other Factors and Mode Collapse\n\n> Do you have any insight on whether the theory can accommodate some of these practicalities, or does it come down to intuition and empirics in the end? Are things like mode collapse a consequence of inexact adherence to theory?\n\nWhile various factors, such as the inherent stochasticity of non-full-batch training, could contribute to mode collapse, we suppose that the primary cause lies in the departure from strict theoretical adherence. This departure occurs due to discretization errors introduced during the translation of the gradient flow into a discrete setting. This discrepancy between theory and practice could also be an interesting area for further exploration.\n\n\n## Typos\n\nWe would like to thank the reviewer for highlighting these typos and apologize for the mistakes.\n- The first part of Eq. (14) should indeed read as: $[\\mathcal{A}\\_{\\theta\\_t}(z)] (V) \\triangleq \\mathbb{E}\\_{z' \\sim p\\_z} \\left[k\\_{g\\_{\\theta\\_t}}(z, z') V(g\\_{\\theta\\_t}(z'))\\right]$.\n- All mentions of $x \\sim p_{\\mathrm{data}}$ in Algorithm 1 should be deleted; we sample from the generated distribution only.\n- For the two last typos, the reviewer is correct and they were already corrected in the PDF file of the supplementary material (where the main paper is reproduced)."}, {"review_id": "jayXlfzIQk", "reviewer": "Reviewer_6P1y", "summary": "In this paper, the authors consider the task of generative modeling as one of particle flow, and provide a unifying theory for score- and diffusion based generator-free models, and generator-based GANs. They propose the particle-flow models that encompassed diffusion and score-based methods such as NCSN or EDMs and subsequently like the approach to incorporate GAN losses to result in flows based on the GAN discriminators. The authors subsequently propose two novel training algorithms. One based on training GAN generators with the score of the data, and its own push-forward distribution score, and another, based on discriminator-based flow. While the performance is not state-of-the-art, given the current interest in diffusion based models, a unifying theory that brings to light connections between GANs and flow based models is certainly interesting. I am convinced that these results could be used to leverage insights learned in the past decade of GAN training to potentially improve the performance of score-based models. ", "questions": "Beyond the concerns remained in terms of the ODE formulation, I have a few additional concerns regarding the experimental formulation: \n\n- I find it a bit hard to understand the overall picture in the training algorithms. I think the authors could add a bit more clarity on how both the score network for the generator in Score GANs, or the discriminator in discriminator flows are trained/used. First, in score GANs, if I understand correctly,  the score of the generator would be required at each training step $t$, to approximate the score of a sampling iteration index $t$, correct? The generator, as the authors also remark, is essentially distilling the entire sampling process of a score-based model, to a single generator sampling step. If so, it is not clear how Score GANs are trained. From Algorithm 1, I understand that $s_{\\phi}^{\\rho}$ is trained for K steps on a DMS loss, when the iterations on $t$ are replaced with noise levels $\\sigma$, starting with $x=g_{\\theta}(z)$. Is this $g_{\\theta_t}(z)$? Does this mean that there are K steps of update per generator update. How large would K need to be to estimate the generator score accurately? Have the authors considered ablation experiments on this, at least say, on Gaussian data? I suspect the need for larger K as the data complexity increases.  Minor comment: Maybe mention what $p_{\\sigma}$ is, in the main document as well (a one line summary of Lines L714-L720 might suffice). \n\n - I have similar concerns with the discriminator training.The presented explanation is not too clear. If my understand is correct, the explanations provided in the appendix indicate that the discriminator, provided with batches of data, considers random $t$-step Langevin sampling, and is trained such that the discriminator mimics this sampling behavior using a chosen loss? What step 6 in Algorithm 3 actually simplifies to, is unclear. It might help to add some intuition on what this loss amounts to minimizing, in the GAN context, (assuming the WGAN-GP loss, as the authors do, as per Table 4). Again, would this be an alternating scheme? Since $f_{\\phi}$ is part of the sampler in lines 3-5, Algo. 3, would we need to reevaluate these samples after ever M steps of update on the discriminator? Is there an interplay between N and the number we would need for M? The authors explore this to a small degree in B.3, but I feel that the ablation experiments on these parameters could be a lot more exhaustive, or at least discussed in more detail.\n\n- Did the authors also explore various embedding schemes for the time index in discriminator flows, and how different would the proposed discriminator be, from the formulation of Denoising Diffusion GANs (Xiao et al., ICLR 2022)? While the embedding is described in L707-711, it is not clear what Table 4 implies, regarding \"Fourier\" time embedding, or how frequency scale and embedding size matter. Lastly, while the time-efficiency analysis in B.3 does answer a few concerns I had on this regard, I’m not entirely convinced as to what the discriminator is acutely achieving, in comparison to a 2nd order Heun solver. I get that the performance of the discriminator flow is one par with second order methods, but can we say more about this? Would it be feasible to explore training the discriminator with samples generated via second order discretization?\n\n - __Minor__: (Figure 5, caption) … neural of function evaluations … — I’m not familiar with the original definition of the measure, but NFE should either be neural function evaluations or number of function evaluations. Equation (14),  on the right of the first equation, I do not see an occurrence of $z^{\\prime}$. Should the summation be over $z^{\\prime}$?\n\n\nOverall, while I find that experimentation is lacking in some regards, such as ablation experiments, given that the paper is more theoretically aligned, it might at least help to provide insights or intuitive comments for variables/hyper-parameters where experiments are not included. I also think the authors would benefit from making the algorithms a bit clearer to understand, in terms of how the models are trained. \n\n[1] Tackling the Generative Learning Trilemma with Denoising Diffusion GANs, Xiao et al., ICLR 2022", "limitations": "The paper is clear about its limitations on being an analytical exploration of GANs and diffusion-based approaches, and not an experiments one. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": " - The proposed formulation is well presented and easy to understand. The overall concept on unifying score-based and GAN generator based modeling is clearly presented. \n - The contributions of the paper are generally made clear, presented as definitions and/or finding.\n - The limitations of the proposed method in achieving state-of-the-art performance are addresses head-on, which, in my opinion, is better, as it manages expectations on what the paper hopes to achieve — a clearer focus on theory, and not on showing SOTA. ", "weaknesses": "- The training algorithm for Score GANs and discriminator flow could be better clarified. While the supplemental sections do address some of the questions I has, the algorithms / experimentation sections leave a good amount to be desired when it comes to understanding how training progresses (please see the __Questions__ section below for detailed concerns on experiments). \n - While most formulations are covered, I feel that a few additional comments could be made in the main text on how sampling/gridding schemes (of the SDE/PDE) serves to affect this formulation. For example, the authors could talk about the possible links between adjusted/unadjusted Langevin dynamics, and the NCSN/EDM formulations of PMs. does having $\\sigma^{\\prime}(t)\\sigma(t)$ affect the discretization in any way. Across the framework, the authors stop at formulating the particle flow PDE, but maybe including comments on how discretizing this equation affects the understand could help. Could we drawn some connections with generator training (which, in some sense, is a “way to solve” to PM PDE) and how one discretized these SDE in diffusion/score-based models. Is this direction something the authors have considered? \n \n - The transition from equation (7) to (8) is treated a bit too casually, in my opinion. While (7) is an SDE, Karras et al., present a discussion on how the discretization could be split into a Probability flow ODE (PF-ODE), and a stochastic component. The formulation in this paper entirely ignores the stochastic component $\\mathrm{d}W_t$, and places itself within the PF-ODE framework. Do the authors feel that this limit the scope of the proposed framework? For example, could they comment on what the implications of having a generator-based framework which solves a variant of (7), including  an exploratory noise term? Is it implicitly present in GAN training? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their detailed feedback. Due to rebuttal size constraints, we answer the reviewer's main questions below. We will follow-up with the rest of the response during the discussion period.\n\n## Training of Discriminator Flows\n\nDiscriminator Flows do not attempt at mimicking Langevin sampling behavior. The principle of Discriminator Flows is to **make particles follow the same functional gradient $\\nabla h_{\\rho}$ as particles in GANs**, their corresponding interacting particle model, without the generator smoothing of Eq. (13). This corresponds to the gradient of the generator's loss that depends on the discriminator (cf. Eq. (20) which relates to Eq. (16) / Finding 5 in GANs). \n\nThe most direct way to achieve this involves successively learning a discriminator per discretization step of Eq. (20), as follows.\n- Sample a batch of $(x_0^b)_b$ i.i.d. from the prior $\\pi$.\n- For $i = 0$ to $N - 1$:\n  - $t = \\frac{i}{N}$.\n  - Learn a new discriminator $f_{t}$ between the current generated distribution  $(x\\_t^b)\\_b$ and $p\\_{\\mathrm{data}}$ using a standard discriminator loss from Eq. (15) like WGAN-GP, i.e. like step 6 of Algorithm 3.\n  - Create the new batch of samples from the updated distribution following Eq. (20): $x_{t + \\frac{1}{N}}^b = x_t^b - \\frac{\\eta}{N} \\nabla_{x_t^b}(c \\circ f_t)(x_t^b)$.\n\nHowever, this is impractical as the large number of independent networks to learn and the successive learning procedure make learning prohibitively slow.\n\nIn our proposed version, we aim at simultaneously learning all discriminators to alleviate this issue. To this end, instead of learning multiple independent discriminators to cover all sampling times, we learn in Algorithm 3 a single time-parameterized network $f_{\\phi}(\\cdot, t)$ on all time steps at once. This may be seen as an alternating scheme, because updating of the discriminator naturally changes all intermediate generated distributions. As specified in step 2 of Algorithm 2, steps 3-4 of Algorithm 3 and lines 263-264, we do so after each discriminator update, i.e. $M = 1$.\n\nOverall, Discriminator Flows do not mimic a known sampling behavior a priori. The path that particles take depends on the chosen functions $a$, $b$, and $c$ parameterizing the losses of Eq. (15) and (16). Determining this path for general GAN losses such as WGAN-GP is an open problem. Nonetheless, it can be described under simplifying hypotheses. If $a$, $b$, and $c$ are chosen to implement an $f$-divergence GAN loss, then Discriminator Flows will implement a forward KL divergence gradient flow (Yi et al., 2023); if they correspond to an IPM GAN loss, then Discriminator Flows will implement a squared MMD gradient flow (Franceschi et al., 2022).\n\nThis makes Discriminator Flows fundamentally different from Denoising Diffusion GANs (Xiao et al., 2022). On the one hand, Denoising Diffusion GANs train time-parameterized GANs to follow the inverse diffusion path. Hence, their discriminators are trained to discriminate between the intermediate generated distribution and the *noisy* data distribution. On the other hand, in Discriminator Flows, discriminators are trained to discriminate between the intermediate generated distribution and the *true*, non-noisy data distribution, thereby learning the path towards $p_{\\mathrm{data}}$.\n\n## Integration of the Stochastic Component in our Framework\n\nWe thank the reviewer for this question, which allows us to develop this aspect of our framework. As the reviewer pointed out, for diffusion models, we focused on the probability flow of Eq. (9) -- which is the true equivalent of Eq. (7), instead of Eq. (8) as stated in most works on diffusion.\n\nWe put aside the probabilistic formulation of Eq. (7) for purposes of clarity of exposition. Fortunately, it is possible to generalize our framework by taking into account the stochastic component of Eq. (7), as explained in the following. We will add this discussion is the appendix of the updated manuscript.\n\nGenerally, the following equation shares the same probability path as both Eq. (7) and (9), for any $\\alpha \\in [0, 1]$:\n$$ \\mathrm{d}x\\_t = 2 \\sigma'(t) \\sigma(t) \\nabla \\log \\left[p\\_{\\mathrm{data}} \\star k\\_{\\mathrm{RBF}}^{\\sigma(t)}\\right] (x\\_t) \\mathrm{d}t - \\alpha \\sigma'(t) \\sigma(t) \\nabla \\log \\rho\\_t(x\\_t) \\mathrm{d}t + \\sqrt{2 \\alpha \\sigma'(t) \\sigma(t)} \\mathrm{d}W\\_t. $$\nThis corresponds to an interpolation between Eq. (7) and (9) that trades Brownian noise with its deterministic equivalent.\n\nThis stochastic component can then be integrated in the formulation of interacting particle models, via Eq. (13). The latter equation takes the directions followed by the particles in the particle model, and transforms them via the operator $\\mathcal{A}\\_{\\theta\\_t}(z)$. Since this operator is linear, it is possible to integrate a stochastic component in the equation (Klebaner, 2012), allowing us to take into account stochastic particle models:\n$$ \\mathrm{d}g\\_{\\theta\\_t}(z) = [\\mathcal{A}\\_{\\theta\\_t}(z)]\\left(\\nabla h\\_{\\rho\\_t} \\mathrm{d}t + \\gamma(t) \\mathrm{d}W\\_t\\right), $$\nwhere $\\gamma(t)$ is a scalar function of time.\n\nThis makes it possible to integrate the stochastic component of diffusion models in Score GANs by interpolating between Gaussian noise and the score of the generated distribution in step 5 of Algorithm 1, similarly to the previous equation using $\\alpha$. Nonetheless, this comes with no guarantee on the experimental performance, as we found in preliminary experiments that adding such a stochastic component is often detrimental to the resulting FID. Indeed, to succeed, the chosen gradient vector field to follow with the generator must be compatible with the generator architecture (i.e., compatible with the generator preconditioning $\\mathcal{A}\\_{\\theta\\_t}(z)$), which may not be the case with white noise of high variance.\n\nF. C. Klebaner. Introduction to stochastic calculus with applications. Imperial College Press. 2012."}]

Author rebuttal:
## Response to Reviews

We would like to thank the reviewers for their insightful, constructive, and timely reviews. We are glad that most reviewers appreciated the paper and the perspective we propose, generally agreeing on its relevance and potential for the community.

Below, we respond to each reviewer individually. The readers will find attached to this global response a one-page PDF of figures illustrating some of our discussions, including interpolations in the latent space and an analysis of the training dynamics of Score GANs.

A recurring concern raised by the reviewers deals with the experimental results. We understand that better performance and more thorough experiments could improve the paper and provide further support to our framework. Nonetheless, we would like to remind the reviewers that the proposed models and experiments serve as a "reasonable basic validation of the theory" (Reviewer iGU3), given the more theoretical focus of the paper, as also noted by Reviewers HYQ5 and 6P1y.

Some reviewers spotted typos throughout the paper. We would like to point out that we had already corrected the most important ones in the appendix provided in the supplementary material. We refer the reviewers to this document for a more polished version of the paper.

We look forward to further discussion with the reviewers during the discussion period.

## Correction: Appendix B.3

During our work on this rebuttal, we noticed a mistake in the experiments in Appendix B.3, which changes the conclusion that Discriminator Flows are more efficient in terms of NFE than the diffusion model EDM with a first-order solver. Thus, we report this correction to the reviewers.

In the original submission, the results of the "EDM (Euler solver)" baseline in Figure 5 were computed for the stochastic version of EDM instead of the deterministic one like in the rest of the paper. The corrected figure with the deterministic baseline can be found in the attached one-page PDF, Figure 1(a).

Surprisingly, the first-order version of EDM is actually more efficient in terms of NFE than the second-order version for low NFE values. This baseline was not considered in the EDM paper (Karras et al., 2022), so this is a new observation. We confirmed this phenomenon using the official implementation of EDM on CIFAR10.

Unfortunately, this invalidates our initial claim that Discriminator Flows are more efficient that the first-order EDM with the chosen discretizations. We still expect that, with proper tuning, Discriminator Flows can be more efficient than diffusion models, based on Figure 1 of our paper -- which remains valid. The final image appears quickly after a few steps, but some residual noise still needs to be eliminated in the remaining steps. 

Nevertheless, this result was secondary and all other results of the paper remain valid, as this mistake only impacted this experiment. We apologize for the confusion and thank the reviewers in advance when taking these new elements into account.

Meta-review: 2x A, 2x WA, and 2x BA. This paper proposes to unify GANs and score-based diffusion models through particle optimization. The reviewers agree on accepting the paper due to its (1) clear presentation, (2) important topic, and (3) sound theoretical derivation. The rebuttal has addressed their concerns.
Decision: Accept (poster)