SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Prior works showed strong links between such flows and GAN optimization, which help us to see that GANs are Int-PMs. In the following, we will see that Stein Gradient flows serve as an example of a functional generator-based counterpart of a generator-less PM.
We begin by generalizing the reasonings of Chu et al. (2020), Durr et al. (2022) and Franceschi et al. (2022), which initially applied only to the case of GANs. We assume that for Int-PMs in Equations (13) and (14), the generator's NTK is constant throughout training, i.e.  $k_{g\theta_t} = k_g$ , like for many networks with infinite width (Jacot et al., 2018; Liu et al., 2020). Then, when  $\nabla h_{\rho} = -\nabla_{W}\mathcal{F}(\rho_{t})$ , e.g., for gradient flows as in Finding 2 or for GANs following Finding 5, we obtain an equation similar to Equation (17):
$$
\frac {\mathrm {d} g _ {\theta_ {t}} (z)}{\mathrm {d} t} = - \eta \mathbb {E} _ {z ^ {\prime} \sim p _ {z}} \left[ k _ {g} \left(z, z ^ {\prime}\right) \nabla_ {W} \mathcal {F} \left(\rho_ {t}\right) \left(g _ {\theta_ {t}} \left(z ^ {\prime}\right)\right) \right]. \tag {18}
$$
This is a special case of Equation (17) with an invertible generator (Chu et al., 2020). However, in the general case, the kernel  $k_g\colon \mathbb{R}^d\times \mathbb{R}^d\to \mathbb{R}^{D\times D}$  acts on the latent space, which makes Equation (18) define a generalized latent-driven Stein flow that was uncovered by Franceschi et al. (2022).
Finding 7. Stein gradient flows are Int-PMs with an invertible generator in the NTK regime and  $h_{\rho_t} = -\frac{\partial\mathcal{F}(\rho_t)}{\partial\rho_t} + \mathrm{cst}$ . They can be generalized to non-invertible generators with Equation (18).
4 Decoupling the Generator and the Flow in Particle Models
We saw in the last section that Int-PMs, such as GANs, generalize PMs, such as diffusion and Wasserstein gradient flows, by applying their generative equations to generator training. This implies that many generative models can be defined by their PM flow, which allows an optional generator to be trained. From Section 3.3, this is the case for gradient flows, which can either define a PM or an Int-PM with the same  $\nabla h$  as a Wasserstein gradient. As per Finding 6, this also holds for GANs, which share a common particle movement with gradient flows and diffusion models.
Consequently, our framework suggests that a functional  $h_{\rho}$  used in an Int-PM may equivalently be used in a PM, and vice versa. This leads us to formulate the following claim.
Claim 1. A generator can be trained using the gradient flow of a score-based diffusion model instead of adversarial training, and it is possible to remove the generator in a GAN by synthesizing samples with the discriminator only.
We confirm this hypothesis in this section by introducing corresponding new hybrid models, which we call respectively Score GANs and Discriminator Flows (see Table 1), and by empirically demonstrating their viability. Note that we introduce these models as proofs of concepts of the applications of our framework. Since they challenge many assumptions and standard practices of generative modeling, they do not benefit from the same wealth of accumulated knowledge that classic models have access to, and thus are harder to tune than standard models.
Figure 1: From left to right, generation process on MNIST of EDM and Discriminator Flow for every 8 evaluations of  $\nabla h_{\rho_t}$ . The last row shows the first 7 steps of Discriminator Flow.
Figure 2: Uncurated samples of studied models on CelebA and MNIST.
Algorithm 1: Training iteration of Score GANs; all operations can be performed in parallel for batching. See Appendix B for details on the practical implementation of lines 3 and 5.
Input: Noise distribution  $p_{\sigma}$ , number of intermediate score training steps  $K$ , learning rates  $\lambda, \eta \in \mathbb{R}_{+}$ , previous generator  $g_{\theta} \colon \mathbb{R}^{D} \to \mathbb{R}$ , previous  $\rho$  score model  $s_{\phi}^{\rho} \colon \mathbb{R}^{D} \times \mathbb{R} \to \mathbb{R}^{D}$ , pretrained  $p_{\mathrm{data}}$  score model  $s_{\psi}^{p_{\mathrm{data}}} \colon \mathbb{R}^{D} \times \mathbb{R} \to \mathbb{R}^{D}$ .
Output: Updated generator  $g_{\theta}$  and  $\rho$  score model  $s_{\phi}^{\rho}$ .  
1 for  $k = 1$  to  $K$  do // Updates of  $s^{\rho}$  by denoising score matching  
2  $\begin{array}{r}z\sim p_z,x = g_\theta (z),\sigma \sim p_\sigma ,x^\sigma \sim \mathcal{N}\big(x,\sigma^2 I_D\big);\\ \phi \leftarrow \phi -\lambda \nabla_\phi \Big\| s_\phi^\rho (x^\sigma ,\sigma) + \frac{x^\sigma - x}{\sigma^2}\Big\| _2^2 \end{array}$   
3 // Score matching with generator, Equations (11) and (19)  
4  $z\sim p_z,\sigma \sim p_\sigma ,\varepsilon \sim \mathcal{N}(0,I_D)$   
5  $\theta \gets \theta +\eta \cdot \nabla_{\theta}g_{\theta}(z)^{\top}\Big(s_{\psi}^{p_{\mathrm{data}}}\big(g_{\theta}(z) + \sigma \varepsilon ,\sigma \big) - s_{\phi}^{\rho}\big(g_{\theta}(z) + \sigma \varepsilon ,\sigma \big)\Big)$
Experimental setting. We conduct experiments on the unconditional generation task for two standard datasets composed of images: MNIST (LeCun et al., 1998) and  $64 \times 64$  CelebA (Liu et al., 2015). We consider two reference baselines, EDM (the score-based diffusion model of Karras et al. (2022)) and GANs, and use the Fréchet Inception Distance (FID, Heusel et al., 2017) to test generative performance
Table 3: Test FID of studied models.
Dataset	PMs (no generator)	Int-PMs (generator)
EDM	Discr. Flow	GAN	Score GAN
MNIST	3	4	3	15
CelebA	10	41	19	35
in Table 3. Training details are given in Appendix D; our open-source code is available at https: //github.com/White-Link/gpm. We refer to Appendix C and the code for more experimental results and samples for each baseline.
4.1 Training Generators with Score-Based Diffusion: Score GANs
We propose training a generator with the score-based diffusion flow of NCSN, Equation (9), left. This involves applying Equation (13) with  $h_{\rho_t} = \log \left[p_{\mathrm{data}} \star k_{\mathrm{RBF}}^\sigma \right] - \log \rho_t$ , where  $t$  represents the training time of the generator. To do so, we directly use the generator weight update formula of Equation (11), as this avoids the problem of estimating  $h_{\rho_t}$  and only requires its gradient  $\nabla h_{\rho_t} = \nabla \log \left[p_{\mathrm{data}} \star k_{\mathrm{RBF}}^\sigma \right] - \nabla \log \rho_t$ . Composed of the scores of, respectively, the noised data distribution and the generated distribution,  $\nabla h_{\rho_t}$  can be efficiently estimated via score matching techniques.
In practice, we use a score network  $s_{\psi}^{p_{\mathrm{data}}}$  pretrained with the latest denoising score matching techniques (Karras et al., 2022) to estimate the static term  $\nabla \log \left[p_{\mathrm{data}} \star k_{\mathrm{RBF}}^{\sigma}\right]$ . Moreover, as  $\nabla \log \rho_t$  is dynamic and needs to be continuously estimated, we leverage GAN discriminator practices and train a network  $s_{\phi}^{\rho}$  by alternating with generator updates to estimate this score.
However, our proposed solution remains impractical for two reasons. First, since the dynamics would match  $p_{\mathrm{data}} \star k_{\mathrm{RBF}}^{\sigma}$  and  $\rho_t$ , we would need to schedule  $\sigma s$  during training, similar to what Song & Ermon (2019) do during inference. Second, while  $\nabla \log \rho_t$  can be estimated using sliced score
Algorithm 2: Training iteration of Discr. Flows. Cf. batching and discretization in Appendix B.
Input: Initial distribution  $\pi = \rho_0$  gradient strength  $\eta \in \mathbb{R}_+$  , learning rate  $\lambda \in \mathbb{R}_{+}$  , previous discriminator  $f_{\phi}\colon \mathbb{R}^{D}\times \mathbb{R}\to \mathbb{R}$    
Output: Updated discriminator  $f_{\phi}$  .   
1  $x\sim p_{\mathrm{data}},x_0\sim \pi ,t\sim \mathcal{U}(0,1);$  // Initialization, random sampling time   
2  $x_{t}\gets x_{0} - \eta \int_{0}^{t}\nabla_{x_{s}}\Big[(c\circ f_{\phi})(x_{s},s)\Big]\mathrm{d}s;\quad /$  Partial generation, Equation (20)   
3  $\phi \leftarrow \phi +\lambda \nabla_{\phi}\bigg\{\mathcal{L}_{\mathrm{d}}\big(f_{\phi}(\cdot ,t);\delta_{x_t},\delta_x\big)\bigg\} ;\quad /$  Train  $f_{\phi}$  at time t, cf. Equation (15)
matching (Song et al., 2020), this approach is less performant than denoising score matching and leads to estimation issues when  $\rho_{t}$  lies on a manifold (Song & Ermon, 2019), as in our case with a pushforward generator. Both of these problems can be solved by instead matching  $p_{\mathrm{data}} \star k_{\mathrm{RBF}}^{\sigma}$  and  $\rho_{t} \star k_{\mathrm{RBF}}^{\sigma}$  for a range of  $\sigma \sim p_{\sigma}$ , using Equations (12) and (13):
$$
h _ {\rho_ {t}} = \log \left[ p _ {\text {d a t a}} \star k _ {\mathrm {R B F}} ^ {\sigma} \right] - \log \left[ \rho_ {t} \star k _ {\mathrm {R B F}} ^ {\sigma} \right]. \tag {19}
$$
This approach allows us to leverage denoising score matching to train  $s_{\phi}^{\rho}$ , and to avoid scheduling  $\sigma s$  by instead sampling them during training and noisig the generated distribution with the chosen noise levels. Overall, we obtain the algorithm for Score GANs described in Algorithm 1.
Finding 8. Score GANs are Int-PMs with  $h_{\rho} = \mathbb{E}_{\sigma \sim p_{\sigma}}\left[\log \left[\frac{p_{\mathrm{data}}\star k_{\mathrm{RBF}}^{\sigma}}{\rho_t\star k_{\mathrm{RBF}}^{\sigma}}\right]\right]$  and  $T = +\infty$ .
Like in GANs, the generated score network  $s_{\phi}^{\rho}$  is trained for a small number of steps  $K$  in between two generator updates. Accordingly, as is the case for a discriminator,  $K$  is an important parameter for Score GANs. We study its impact of the method's performance in Appendix C.4, where we ensure that a small  $K$  suffices to accurately estimate the score of the generated distribution.
4.2 Removing the Generator from GANs: Discriminator Flows
Based on Findings 4 and 5, we see that removing the generator from GANs to make them PMs, as in Definition 1, simply requires that we define  $h_{\rho_t} = -c \circ f_{\rho_t}$ , where  $f_{\rho_t}$  is the discriminator between  $\rho_t$  and  $p_{\mathrm{data}}$  at sampling time  $t$  in Equation (1):
$$
\mathrm {d} x _ {t} = - \nabla \left(c \circ f _ {\rho_ {t}}\right) \left(x _ {t}\right) \mathrm {d} t. \tag {20}
$$
This makes Equation (20) the equivalent of GAN training, but as a PM without a generator. In other words, Discriminator Flows make individual particles follow the gradient of the generator loss of Equation (16), defined through the discriminator, without the generator smoothing of Equation (13).
Finding 9. Discriminator Flows are PMs with  $h_{\rho} = -c \circ f_{\rho_t}$  and  $T = +\infty$ .
Such a model would a priori require us to successively train a neural discriminator  $f_{\phi_t} \equiv f_{\rho_t}$  per time step  $t$ . This results, however, in a prohibitively slow and heavy training procedure. As a more scalable alternative, we train a single time-dependent neural discriminator  $f_{\phi} \colon \mathbb{R}^{D} \times \mathbb{R} \to \mathbb{R}$ , that takes as input both a sample  $x_{t} \sim \rho_{t}$  and its corresponding time  $t \in \mathbb{R}$ , on all time steps at once. Each  $x_{t} \sim \rho_{t}$  must then be computed both in training and inference from  $x_0 \sim \rho_0 = \pi$  using Equation (20). This results in the training procedure of Algorithm 2 for Discriminator Flows. For practical convenience, we restrict, without loss of generality,  $t \in [0,1]$ .
Compared to diffusion models, for which the score can be freely estimated at each  $t$  because  $\rho_{t}$  is assumed to equal  $p_{\mathrm{data}}\star k_{\mathrm{RBF}}^{\sigma (t)}$ , it is more challenging to elucidate the particle movement of Equation (20). Prior studies on GAN optimization help answer this problem under simplifying assumptions. Indeed, if  $a,b,$  and  $c$  (cf. Section 3.2) are chosen to implement an  $f$ -divergence GAN loss, then Discriminator Flows will implement a forward KL divergence gradient flow (Yi et al., 2023) If they instead correspond to an IPM GAN loss, then Discriminator Flows will implement a squared MMD gradient flow (Franceschi et al., 2022).
In the general case, the generation process of Discriminator Flows is not known in advance. In fact, we must simulate the entire process during training, making each training iteration slower. However,
our approach has the advantage of generalizing the diffusion process and allowing the discriminator to learn another path from the initial distribution  $\rho_0 = \pi$  to  $p_{\mathrm{data}}$ . We believe that, when properly tuned, Discriminator Flows could provide faster sampling times than diffusion models because they can learn shorter paths towards  $p_{\mathrm{data}}$ .
We see uncurated samples from our proposed Score GAN and Discriminator Flow models in Figure 2, as well as the baseline EDM and GAN models. In Table 3 we report the FID scores for all models. We observe that both of our introduced models produce reasonable results, which experimentally confirms our initial claim. Nonetheless, these models exhibit worse (higher) FID scores than our baselines, although Discriminator Flows provide good performance on the MNIST dataset. We attribute these performance results to the fact that our proposed models are novel and do not benefit from the accumulated knowledge regarding training best practices that the standard models possess.
Interestingly, the proposed models exhibit typical properties of generator-free and generator-based models. Since Score GANs use a generator, as in GANs, only one function evaluation is required to draw a sample, making it orders of magnitude faster to sample from than EDM. Like GANs, Score GANs may produce mode collapse and benefit from smooth latent space interpolations, as shown in Appendices C.2 and C.3. Of course, Discriminator Flows are slower at both training and inference time than such generator-based models, but this comes with the additional flexibility of operating directly in the data space (Voleti et al., 2022; Couairon et al., 2023). As a consequence, like diffusion models, the latent space interpolation (i.e., interpolating the initial noise  $x_0 \sim \pi$  in Equation (1)) capabilities of Discriminator Flows remain below those of generator-based models - cf. Appendix C.3.
However, as shown in Figure 1 and in the numerical results of Appendix C.5, Discriminator Flows have the expected advantage of converging to  $p_{\mathrm{data}}$  faster than the state-of-the-art diffusion model EDM. In particular, we notice that images generated by Discriminator Flows are well-formed early in the generation process, hinting at potential temporal cost reduction by stopping the process early. Unfortunately, this has not yet yielded a better time efficiency than EDM; we nevertheless believe it can be achieved with additional architectural and model tuning of Discriminator Flows. We further discuss the time efficiency of our introduced methods in Appendices A.4, C.5 and D.3.
4.4 Relationship with Prior Work
Score GANs. Conceptually, Score GANs implement for each  $\sigma$  a forward KL gradient flow, similarly to Yi et al. (2023), who theoretically proved that some GAN models approximate such a flow without noising, under optimality assumptions on the discriminator. However, Score GANs differ from traditional GANs as they do not involve a discriminator, but rather split the flow  $\nabla h_{\rho_t}$  into two parts: one part that can be estimated before generator training as it depends only on the data distribution, and another part that must be continuously estimated during training as it depends on the generated distribution. While  $h_{\rho_t}$  could be estimated by adding noise to the inputs of a discriminator (Wang et al., 2022), Score GANs instead only need to estimate the score of the generated distribution, which is no longer adversarial.
Discriminator Flows. As a general concept, Discriminator Flows provide an encompassing framework that helps to reveal the connections of various approaches to GAN training.
Recently, Heng et al. (2023) introduced deep generative Wasserstein gradient flows (DGGF), a method that relies on  $f$ -divergence gradient flows approximated by estimating the ratio  $\rho_t / p_{\mathrm{data}}$  with a neural network. Examined under our framework, DGGF's training objectives correspond to that of a discriminator in  $f$ -divergence GANs, allowing us to frame DGGF as a special case of Discriminator Flows. Nonetheless, we stress that Discriminator Flows have a larger scope than DGGF since we can handle all types of GAN objectives; all our experiments were performed with WGAN objectives. Moreover, unlike DGGF, which in practice removes the time dependency in its estimation without a theoretical justification, our method does handle time as input to the discriminator.
Discriminator Flows also relate to, and generalize, methods that finetune GAN outputs with gradient flows (Tanaka, 2019; Che et al., 2020; Ansari et al., 2021). The latter use discriminator gradients to approximate such flows, making them naturally expressible as Discriminator Flows in our framework.
In practice, they only apply their sampling procedures in the latent space of the generator, as applying them in pixel space leads to artifacts. We resolve this issue with a principled training procedure for the discriminator conditioned on sampling time.
We note that previous works, such as Xiao et al. (2022) and Jolicoeur-Martineau et al. (2021), also combine score-based models and GANs. Xiao et al. (2022) train several GANs to successively denoise an image, mimicking a reverse diffusion process. Jolicoeur-Martineau et al. (2021) augment the denoising objective of score-based diffusion models with an adversarial objective to improve the denoising image quality. However, while these works do draw links between both models, they do not aim to unify score-based models and GANs.
Finally, we provide intuition on the sampling efficiency of Discriminator Flows observed in Figure 1. We observe that diffusion models smooth the data distribution with a Gaussian kernel by Equation (5), while discriminators were shown to smooth the data distribution with their NTK (Franceschi et al., 2022). This observation brings diffusion models and GANs closer to each other, while explaining the fast convergence speed of Discriminator Flows thanks to the properties of NTKs for generative modeling. Indeed, Franceschi et al. (2022) presented empirical evidence that using NTKs of standard discriminators as kernels in squared MMD gradient flows (which some GAN models implement, cf. Finding 6), instead of Gaussian kernels, accelerates the convergence of Equation (1) towards  $p_{\mathrm{data}}$  by several orders of magnitude. More generally, we hypothesize that it is the natural effectiveness of neural networks in high dimensions that endows Discriminator Flows with a faster convergence speed than diffusion models, for which the particles' paths are chosen explicitly.
In this paper we have unified score-based diffusion models, GANs, and gradient flows under a single framework based on particle models which can be complemented with a generator. Since this framework unifies models that have been customarily opposed in the literature, this work paves the way for new perspectives in generative modeling. As an example of potential applications, we have shown that our framework naturally leads to two novel generative models: a generator that follows score-based gradients, and a generator-free GAN that uses a discriminator-guided generation process.
Of course, generator-less and generator-based models each retain their unique attributes. On the one hand, generator training provides a simple and efficient sampling procedure and endows the generative model with a low-dimensional structured latent space, at the cost of potential instability and mode collapse. On the other hand, generator-less models, despite their slow sampling, may be easier to train, since the generator component has been removed from their flow, and are more flexible as they rely on a continuous-time process directly defined in the data space. We believe that our framework, by revealing the close relationship between these models, can help them to improve upon one another, or can even help create other new hybrid models.
Beyond potential applications, our study could be enhanced and expanded in many ways for future work. On the theoretical side, we would like to tackle the challenging task (Hsieh et al., 2021) of taking into account the fact that the discriminator in GANs is actually continuously trained with the generator. It would also be interesting to generalize our framework to second-order and stochastic particle movement in generator-based models, and to study the impact of discretization on the studied differential equations, as hinted in Appendices A.1 to A.3. On the practical side, while we have proposed new models that function reasonably well, we would be interested in refining them further for state-of-the-art generative performance. Furthermore, Score GANs could serve as a distillation method for score-based diffusion models (Salimans & Ho, 2022), while Discriminator Flows could outperform diffusion models for generation efficiency.
We would like to thank Lorenzo Croissant and Ugo Tanielian for helpful discussions and comments on this paper, as well as Edouard Delasalles for inspiring the architecture of our code.
This work was granted access to the HPC/AI resources of IDRIS under the allocation 2023-AD011013503R1 made by GENCI (Grand Equipement National de Calcul Intensif). Emmanuel de Bezenac is financially supported by the ETH Foundations of Data Science.
In this section we develop a number of elements of our framework and models to provide more context and perspective on our contributions.
A.1 Extension to Stochastic Particle Models
We focus in the main paper on deterministic particle movement. However, some PMs like diffusion models also have equivalent stochastic particle movements as described in Equations (4) and (7). We provide some evidence in this section that stochastic particle movement can also be integrated to Int-PMs, thereby strengthening the generality of our approach.
Generally, using the Fokker-Planck equation, the following equation shares the same probability path as both Equations (7) and (9), for any  $\alpha \in [0,1]$ :
$$
\mathrm {d} x _ {t} = 2 \sigma^ {\prime} (t) \sigma (t) \nabla \log \left[ p _ {\text {d a t a}} \star k _ {\mathrm {R B F}} ^ {\sigma (t)} \right] \left(x _ {t}\right) \mathrm {d} t - \alpha \sigma^ {\prime} (t) \sigma (t) \nabla \log \rho_ {t} \left(x _ {t}\right) \mathrm {d} t + \sqrt {2 \alpha \sigma^ {\prime} (t) \sigma (t)} \mathrm {d} W _ {t}. \tag {21}
$$
This corresponds to an interpolation between Equations (7) and (9) that trades Brownian noise with its deterministic equivalent.
This stochastic component can then be integrated into the formulation of interacting particle models, via Equation (13). This latter equation takes the directions followed by the particles in the particle model, and transforms them via the operator  $\mathcal{A}_{\theta_t}(z)$ . Since this operator is linear, it is possible to integrate a stochastic component into the equation (Klebaner, 2012), allowing us to take into account stochastic particle models:
$$
\mathrm {d} g _ {\theta_ {t}} (z) = \left[ \mathcal {A} _ {\theta_ {t}} (z) \right] \left(\nabla h _ {\rho_ {t}} \mathrm {d} t + \gamma (t) \mathrm {d} W _ {t}\right), \tag {22}
$$
where  $\gamma (t)$  is a scalar function of time.