SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jean-Yves Franceschi
Criteo AI Lab, Paris, France
jycja.franceschi@criteo.com
Mike Gartrell
Criteo AI Lab, Paris, France  
mike.gartrell@acm.org
Ludovic Dos Santos*
l.dossantos@criteo.com
Thibaut Issenhuth*
LIGM, Ecole des Ponts, Univ Gustave Eiffel,
CNRS, Marne-la-Vallee, France
t.issenhuth@criteo.com
Emmanuel de Bézenac*
Seminar for Applied Mathematics,
D-MATH, ETH Zürich, Rämistrasse 101,
Zürich-8092, Switzerland
emmanuel.debezenac@sam.math.ethz.ch
Mickaël Chen*
Valeo.ai, Paris, France
mickael.chen@valeo.com
Alain Rakotomamonjy*
a.rakotomamonjy@criteo.com
Particle-based deep generative models, such as gradient flows and score-based diffusion models, have recently gained traction thanks to their striking performance. Their principle of displacing particle distributions using differential equations is conventionally seen as opposed to the previously widespread generative adversarial networks (GANs), which involve training a pushforward generator network. In this paper we challenge this interpretation, and propose a novel framework that unifies particle and adversarial generative models by framing generator training as a generalization of particle models. This suggests that a generator is an optional addition to any such generative model. Consequently, integrating a generator into a score-based diffusion model and training a GAN without a generator naturally emerge from our framework. We empirically test the viability of these original models as proofs of concepts of potential applications of our framework.
Score-based diffusion models (Song et al., 2021) have recently received significant attention within the machine learning community, due to their striking performance on generative tasks (Rombach et al., 2022; Ho et al., 2022). Similarly to gradient flows, these models involve systems of particles, where the displacement of the particle distribution is described by a differential equation parameterized by a gradient vector field. Such particle-based deep generative models are typically seen as opposed to generative adversarial networks (GANs, Goodfellow et al., 2014), as the latter involves adversarial training of a generator network (Dhariwal & Nichol, 2021; Song, 2021; Xiao et al., 2022).
In this paper, we challenge the conventional view that particle and adversarial generative models are opposed to each other. We make the following contributions.
A unified framework. We present a novel framework that unifies both classes of models, showing that they are based on similar particle evolution equations. Particle models follow a gradient vector
*Authors listed in a randomly chosen order.
Table 1: Taxonomy of particle models, including our proposed hybrid models: Score GANs and Discriminator Flows.
Model	Generator	Flow type ∇hρt
Wasserstein gradient flows	X	Wasserstein gradient -∇WF(ρt)
Stein gradient flows	✓
Score-based diffusion models	X	αt∇ log[pdata*krBF] - βt∇ log ρt
Score GANs	✓
Discriminator Flows	X	-∇(c○fρt)
GANs	✓	where fρt is a discriminator between ρt and pdata
field during inference; in a similar fashion, the generator's outputs can be seen as following the same gradient field during training, up to a specific smoothing due to the generator. Building upon prior scattered literature, we propose a new framework that encompasses a variety of methods, which are listed in Table 1 together with their respective flow type.
Decoupling generators and flows. By uncovering the role of the generator as a smoothing operator on vector fields, we suggest that the existence of a generator and the flow that particles follow can be decoupled. We deduce that it is possible to train a generator with score-based gradients which replace adversarial training (which we call a Score GAN); and that a GAN can be trained without a generator, using only the discriminator to synthesize samples (which we call a Discriminator Flow); cf. Table 1. We introduce these new models as proofs of concept, which we empirically assess to support the validity of our framework and illustrate the new perspectives it opens up in this active field of research.
Throughout the paper, we call out some specifics of the contributions of our framework as Definitions and Findings (italicized in the main text).
Outline. We begin with a discussion of particle models without generators in Section 2, covering Wasserstein gradient flows and score-based diffusion models. In Section 3 we discuss how generator training can be framed as a particle model with a generator, including GANs and Stein gradient flows. Section 4 then highlights how our framework allows us to decouple the generator and flow components of particle models, leading to the aforementioned new models, Score GANs and Discriminator Flows. Finally, we discuss the implications of our findings and conclude the paper in Section 5.
Notations. We consider the evolution of generated distributions  $\rho_{t}$  over  $\mathbb{R}^D$  w.r.t. time  $t\in \mathbb{R}_+$ , where  $t$  is the inference time for particle models without generators, or the training time for generator training, respectively. This evolution until some finite or infinite end time  $T\in \mathbb{R}_{+}\cup \{+\infty \}$  then yields a final generated distribution  $p_{\mathrm{g}} = \rho_T$ , which ideally approaches the data distribution  $p_{\mathrm{data}}$ .
2 Particle Models without Generators: Non-interacting Particles
In this section we formally introduce the notion of generative particle models that do not use a generator, and present two standard instances: Wasserstein gradient flows and score-based diffusion models. They involve the manipulation of particles  $x_{t} \sim \rho_{t}$  following a differential equation parameterized by some vector field. We characterize these models by noticing that their particles actually optimize, independently from one another, a loss that depends on the current particle distribution.
Definition 1 (Particle Models, PMs). PMs model particles  $x_{t} \sim \rho_{t}$  starting from a prior  $\rho_0 = \pi$ :
$$
x _ {0} \sim \pi = \rho_ {0}, \quad \mathrm {d} x _ {t} = \nabla h _ {\rho_ {t}} \left(x _ {t}\right) \mathrm {d} t, \tag {1}
$$
where  $h_{\rho_t} \colon \mathbb{R}^D \to \mathbb{R}$  is a functional that depends on the current particle distribution  $\rho_t$ . Time  $t$  corresponds to generation/inference time from  $\rho_0$  to the final distribution  $p_g = \rho_T$ .
Finding 1. In PMs, the evolution of Equation (1) makes each generated particle  $x_{t}$  individually follow a gradient ascent path on the objective  $h_{\rho_t}(x_t)$ .
In prior works, the prior  $\pi$  is conventionally chosen to be easy to sample from, such as a Gaussian.  $h_{\rho_t}$  is usually defined in a theoretical manner so that the flow of Equation (1) conveys good convergence
properties for  $\rho_{t}$  towards  $p_{\mathrm{data}}$  when  $t\to T$ . Because analytically computing  $h_{\rho_t}$  is often intractable, it is empirically estimated and replaced by a neural network in practice.
2.1 Wasserstein Gradient Flows
A Wasserstein gradient flow is a generalization of gradient descent on a functional in the space of probability measures. More formally, it is an absolute continuous curve of probability distributions in a Wasserstein metric space  $\mathcal{P}_2$  over  $\mathbb{R}^D$  that satisfies a continuity equation (Santambrogio, 2017), and equivalently an evolution of particles  $x_{t}\sim \rho_{t}$  under mild hypotheses (Jordan et al., 1998):
$$
\partial_ {t} \rho_ {t} - \nabla \cdot \left(\rho_ {t} \nabla_ {W} \mathcal {F} (\rho_ {t})\right) = 0, \quad \mathrm {d} x _ {t} = - \nabla_ {W} \mathcal {F} (\rho_ {t}) (x _ {t}) \mathrm {d} t, \tag {2}
$$
where  $\mathcal{F} \colon \mathcal{P}_2 \to \mathbb{R}$  is the functional to minimize in the Wasserstein space. This definition involves the Wasserstein gradient of the functional  $\mathcal{F}(\rho_t)$  - similar to the gradient of a functional defined over a Euclidean space - which for some functionals can be obtained in closed form by computing the first variation of the functional  $\mathcal{F}$  (Santambrogio, 2017, Section 4.3):
$$
\nabla_ {W} \mathcal {F} \left(\rho_ {t}\right) = \nabla \frac {\partial \mathcal {F} \left(\rho_ {t}\right)}{\partial \rho_ {t}}: \mathbb {R} ^ {D} \rightarrow \mathbb {R} ^ {D}. \tag {3}
$$
Finding 2. Wasserstein gradient flows are PMs with  $h_{\rho_t} = -\frac{\partial \mathcal{F}(\rho_t)}{\partial \rho_t} + \mathrm{cst}$  and  $T = +\infty$ .
The partial differential equation governing the particle evolution, as well as its convergence properties toward  $p_{\mathrm{data}}$ , strongly depends on the functional  $\mathcal{F}$ . We detail the standard examples of the forward Kullback-Leibler (KL) divergence (Kullback & Leibler, 1951),  $f$ -divergences (Renyi, 1961), the squared Maximum Mean Discrepancy (MMD, Gretton et al., 2012; Arbel et al., 2019), and entropy regularization in Table 2. They can be additively combined for a variety of objectives  $\mathcal{F}$ . More
Table 2: Gradient flows for standard objectives  $\mathcal{F}$  .
Objective F(ρ)	hρ
Forward KL	Eρ log ρ/Pdata	- log ρ/Pdata
f-divergence	EPdata f(ρ/Pdata)	-f'(ρ/Pdata)
Squared MMD w.r.t. kernel k	EBx, x'~ρy, y'~Pdata[k(x, x') + k(y, y') - 2k(x, y)]	EBy~Pdata[k(y,·)] - E×~ρ[k(x,·)]
Entropy	Eρ log ρ	- log ρ
examples exist in the literature (Liutkus et al., 2019; Mroueh et al., 2019; Glaser et al., 2021).
Several methods have been explored in the literature to solve Equation (2) in practice, either using input-convex neural networks (Mokrov et al., 2021; Alvarez-Melis et al., 2022) to discretize the continuous flow, or parameterizing  $\nabla h_{\rho}$  by a neural network (Gao et al., 2019; Fan et al., 2022; Heng et al., 2023). In all cases these methods fit within the class of PMs as framed in Definition 1.
2.2 Score-Based Diffusion Models
Early score-based models (Noise Conditional Score Networks [NCSN], Song & Ermon, 2019) rely on Langevin dynamics as described in the following stochastic differential equation, converging towards  $p_{\mathrm{data}}$  when  $t \to \infty$ :
$$
\mathrm {d} x _ {t} = \nabla \log p _ {\text {d a t a}} (x _ {t}) \mathrm {d} t + \sqrt {2} \mathrm {d} W _ {t}. \tag {4}
$$
Several methods that use neural networks to estimate the score function of the data distribution  $\nabla \log p_{\mathrm{data}}$  (Hyvärinen, 2005), coupled with the use of Langevin dynamics, can work in practice even for high-dimensional distributions. Nonetheless, because of ill-definition and estimation issues of the score for discrete data on manifolds, a Gaussian perturbation of the data distribution is introduced to stabilize the dynamics. Thus,  $p_{\mathrm{data}}$  in Equation (4) is replaced by the distribution  $p_{\mathrm{data}}^{\sigma}$  of  $x + \sigma \varepsilon$ , where  $x \sim p_{\mathrm{data}}$  and  $\varepsilon \sim \mathcal{N}(0, I_D)$ . Denoting  $\star$  as the convolution of a probability distribution  $p$  by a kernel  $k: \mathbb{R}^{D} \times \mathbb{R}^{D} \to \mathbb{R}$ , we notice that  $p_{\mathrm{data}}^{\sigma}$  is actually the convolution of  $p_{\mathrm{data}}$  by a Gaussian kernel:
$$
p _ {\mathrm {d a t a}} ^ {\sigma} = p _ {\mathrm {d a t a}} \star k _ {\mathrm {R B F}} ^ {\sigma}, \quad p \star k \triangleq \int_ {x} k (x, \cdot) \mathrm {d} p (x), \quad k _ {\mathrm {R B F}} ^ {\sigma} (x, y) \triangleq \frac {1}{\sigma \sqrt {2 \pi}} \mathrm {e} ^ {- \frac {\| x - y \| _ {2} ^ {2}}{2 \sigma^ {2}}}. \tag {5}
$$
NCSN then follows this equation, estimating the score with denoising score matching (Vincent, 2011) and repeating the process for a decreasing sequence of  $\sigma s$ :
$$
\mathrm {d} x _ {t} = \nabla \log \left[ p _ {\text {d a t a}} \star k _ {\mathrm {R B F}} ^ {\sigma} \right] \left(x _ {t}\right) \mathrm {d} t + \sqrt {2} \mathrm {d} W _ {t}. \tag {6}
$$
Building on Song et al. (2021), newer score models (Karras et al., 2022) make  $\sigma$  a continuous function of time  $\sigma(t)$ , decreasing towards 0 in finite time to improve convergence. Many of these approaches share the following generation equation, corresponding to the reverse of a noising process for  $p_{\mathrm{data}}$  (Elucidating the Design Space of Diffusion-Based Generative Models [EDM], Karras et al., 2022):
$$
\mathrm {d} x _ {t} = 2 \sigma^ {\prime} (t) \sigma (t) \nabla \log \left[ p _ {\text {d a t a}} \star k _ {\mathrm {R B F}} ^ {\sigma (t)} \right] \left(x _ {t}\right) \mathrm {d} t + \sqrt {2 \sigma^ {\prime} (t) \sigma (t)} \mathrm {d} W _ {t}, \tag {7}
$$
where  $\sigma'(t)$  is the derivative of  $\sigma(t)$ . These approaches admit an equivalent deterministic flow yielding the same probability path  $\rho_t$ :
$$
\mathrm {d} x _ {t} = \sigma^ {\prime} (t) \sigma (t) \nabla \log \left[ p _ {\text {d a t a}} \star k _ {\mathrm {R B F}} ^ {\sigma (t)} \right] \left(x _ {t}\right) \mathrm {d} t. \tag {8}
$$
However, we notice that this only holds under the implicit assumption that Equation (7) perfectly reverses the initial noising process, i.e.,  $\rho_{t} = p_{\mathrm{data}}\star k_{\mathrm{RBF}}^{\sigma (t)}$ . This assumption can be broken in practice when the score is not well estimated or for coarse time discretization. In the general case for both NCSN and EDM, by using the Fokker-Planck equation (Jordan et al., 1998), which allows us to substitute the stochastic component  $\mathrm{d}W_{t}$  by the negative of the score of the generated distribution, we obtain, respectively, the following equivalent exact probability flows:
$$
\frac {\mathrm {d} x _ {t}}{\mathrm {d} t} = \nabla \log \left[ \frac {p _ {\mathrm {d a t a}} \star k _ {\mathrm {R B F}} ^ {\sigma}}{\rho_ {t}} \right] (x _ {t}), \quad \frac {\mathrm {d} x _ {t}}{\mathrm {d} t} = \sigma^ {\prime} (t) \sigma (t) \nabla \log \left[ \frac {1}{\rho_ {t}} \left(p _ {\mathrm {d a t a}} \star k _ {\mathrm {R B F}} ^ {\sigma (t)}\right) ^ {2} \right] (x _ {t}). \tag {9}
$$
Finding 3. Score-based diffusion models are PMs:
- NCSN (Song & Ermon, 2019) with  $h_{\rho_t} = \log \left[p_{\mathrm{data}} \star k_{\mathrm{RBF}}^\sigma \right] - \log \rho_t$  and  $T = +\infty$ ;
- EDM (Karras et al., 2022) with  $h_{\rho_t} = \sigma'(t)\sigma(t)\left(2\log\left[p_{\mathrm{data}} \star k_{\mathrm{RBF}}^{\sigma(t)}\right] - \log\rho_t\right)$  and  $T < +\infty$ .
NCSN actually implements through Langevin dynamics a forward KL gradient flow, which is common knowledge in the related literature (Jordan et al., 1998; Yi et al., 2023).
3 Particle Models with Generators: Training of Interacting Particles
In the previous section we presented a framework for particle models (PMs) that, in the absence of a generator, individually manipulate particles in the data space, and optimize a distribution-dependent objective  $h_{\rho_t}$  via a differential equation. In this section we frame generator training as a generalization of PMs involving direct interaction between particles. We show, supported by prior literature, that our framework applies to the case of GANs and Stein gradient flows.
3.1 Generator Training as a Modified Particle Model
We begin with the training of a neural generator  $g_{\theta} \colon \mathbb{R}^d \to \mathbb{R}^D$  parameterized by  $\theta$ . Associated with a prior distribution on its latent space  $p_z$ ,  $g_{\theta}$  produces a generated distribution  $p_{\theta}$  as the pushforward of  $p_z$  through  $g_{\theta}$ :  $p_{\theta} = g_{\theta} \sharp p_z$ , by which we seek to imitate  $p_{\mathrm{data}}$ . Unlike PMs which progressively construct synthesized samples directly in the data space  $\mathbb{R}^D$ , generators enable models like GANs to generate samples starting from a different latent space. When  $d < D$ , this latent space allows the resulting distribution to be naturally embedded into a lower-dimensional manifold, thereby integrating the manifold hypothesis (Bengio et al., 2013). The parameters  $\theta$  evolve during training, making the generated distribution move accordingly:  $\rho_t = p_{\theta_t}$ .
We characterized in Section 2, Finding 1, PMs as models that make free generated particles optimize an objective  $h_{\rho} \colon \mathbb{R}^{D} \to \mathbb{R}$  that conveys desirable convergence properties. We leverage this observation to show that generator training can be framed as a PM as well. We see that generators involve generated particles  $x_{t} \sim \rho_{t}$  as generator outputs  $x_{t} = g_{\theta_{t}}(z)$ , with  $z \sim p_{z}$ , which move during training. We proceed by making the generator optimize the same objective  $h_{\rho_t}$  as in PMs, that is, the generator parameters are trained to minimize at each optimization step:
$$
\mathcal {L} _ {\text {g e n}} (\theta) = - \mathbb {E} _ {z \sim p _ {z}} \left[ h _ {\rho_ {t}} (g _ {\theta} (z)) \right]. \tag {10}
$$
In the optimization of Equation (10), we intentionally ignore the dependency of  $\rho$  on  $\theta$ , i.e. in practice  $\rho = \mathrm{StopGradient}(g_{\theta}\sharp p_z)$ . This allows us to mimic PMs where generated particles  $x_{t}$ s optimize the objective  $h_{\rho_t}$ , without taking into account that  $\rho_{t}$  is actually a mixture of all the  $x_{t}$ s.
By optimizing  $\theta_{t}$  via gradient descent for the loss of Equation (10) with learning rate  $\eta$ , idealized in the continuous training time setting, we obtain using the chain rule:
$$
\frac {\mathrm {d} \theta_ {t}}{\mathrm {d} t} = \eta \nabla_ {\theta_ {t}} \mathbb {E} _ {z \sim p _ {z}} \left[ h _ {\rho_ {t}} \left(g _ {\theta_ {t}} (z)\right) \right] = \eta \mathbb {E} _ {z \sim p _ {z}} \left[ \nabla_ {\theta_ {t}} g _ {\theta_ {t}} (z) \nabla h _ {\rho_ {t}} \left(g _ {\theta_ {t}} (z)\right) \right]. \tag {11}
$$
As a consequence, using the chain rule again, each generated particle  $x_{t} = g_{\theta_{t}}(z) \sim \rho_{t}$  evolves as:
$$
\frac {\mathrm {d} g _ {\theta_ {t}} (z)}{\mathrm {d} t} = \nabla_ {\theta_ {t}} g _ {\theta_ {t}} (z) ^ {\top} \frac {\mathrm {d} \theta_ {t}}{\mathrm {d} t} = \eta \mathbb {E} _ {z ^ {\prime} \sim p _ {z}} \left[ k _ {g _ {\theta_ {t}}} \left(z, z ^ {\prime}\right) \nabla h _ {\rho_ {t}} \left(g _ {\theta_ {t}} \left(z ^ {\prime}\right)\right) \right], \tag {12}
$$
where  $k_{g_{\theta}}:z,z^{\prime}\mapsto \nabla_{\theta_{t}}g_{\theta_{t}}(z)^{\top}\nabla_{\theta_{t}}g_{\theta_{t}}(z^{\prime})$  is the matrix Neural Tangent Kernel (NTK, Jacot et al., 2018) of the generator. Equation (12) describes the dynamics of the generated particles as a modified version of Equation (1) for PMs. We formalize this as follows.
Definition 2 (Interacting Particle Models, Int-PMs). Int-PMs model particles resulting from the pushforward  $g_{\theta_t} \sharp p_z$  of a generator  $g_{\theta_t}$  applied to a prior  $p_z$ , with the following training dynamics:
$$
\mathrm {d} g _ {\theta_ {t}} (z) = \eta \left[ \mathcal {A} _ {\theta_ {t}} (z) \right] \left(\nabla h _ {\rho_ {t}}\right) \mathrm {d} t, \tag {13}
$$
where  $h_{\rho_t} \colon \mathbb{R}^D \to \mathbb{R}$  is a functional that depends on the current distribution  $\rho_t$ , time  $t$  is training time, and  $\mathcal{A}_{\theta_t}(z)$  is a linear operator operating on vector fields (Sriperumbudur et al., 2010), defined as:
$$
\left[ \mathcal {A} _ {\theta_ {t}} (z) \right] (V) \triangleq \mathbb {E} _ {z ^ {\prime} \sim p _ {z}} \left[ k _ {g _ {\theta_ {t}}} \left(z, z ^ {\prime}\right) V \left(g _ {\theta_ {t}} \left(z ^ {\prime}\right)\right) \right], \quad k _ {g _ {\theta_ {t}}} \left(z, z ^ {\prime}\right) \triangleq \nabla_ {\theta_ {t}} g _ {\theta_ {t}} (z) ^ {\top} \nabla_ {\theta_ {t}} g _ {\theta_ {t}} \left(z ^ {\prime}\right). \tag {14}
$$
Similarly to PMs, the vector field  $\nabla h_{\rho_t}$  in Int-PMs indicates which direction each generated particle will follow to get closer to the data distribution. However,  $\mathcal{A}_{\theta_t}$  smooths this gradient field using the generator's NTK, and generated particles thus interact with each other. Indeed, moving one particle makes its neighbors move accordingly because of their underlying parameterization by the generator. Notably, Int-PMs generalize PMs: in the degenerate case where  $k(z,z^{\prime}) = \delta_{z - z^{\prime}}I_{D}$ , with  $\delta$  the Dirac delta function centered on 0, i.e., when particles can move freely with a sufficiently powerful generator, the effect of parameterization disappears with  $\left[\mathcal{A}_{\theta_t}(z)\right](V) = V\big(g_\theta (z)\big)$ , and therefore Equation (13) reduces to Equation (1).
Finding 4. Int-PMs generalize PMs. Each Int-PM is therefore defined by two components: the objective function  $h_{\rho_t}$  and the choice of generator architecture  $g_{\theta}$ .
We will show in the remainder of this section that Int-PMs encompass both GANs and Stein gradient flows, borrowing Franceschi et al. (2022)'s results which the previous reasoning generalizes.
3.2 GANs as Interacting Particle Models
In GANs, each generator  $g_{\theta}$  is accompanied by a discriminator  $f_{\rho}$  that depends on the generated distribution.  $f_{\rho}$  is optimized as a neural network via gradient ascent (GA) to maximize an objective of the following form:
$$
f _ {\rho} = \mathrm {G A} _ {f} \left\{\mathcal {L} _ {\mathrm {d}} (f; \rho , p _ {\text {d a t a}}) \triangleq \mathbb {E} _ {\rho} [ a \circ f ] - \mathbb {E} _ {p _ {\text {d a t a}}} [ b \circ f ] + \mathcal {R} (f; \rho , p _ {\text {d a t a}}) \right\}, \tag {15}
$$
for some functions  $a, b \colon \mathbb{R} \to \mathbb{R}$  (e.g. for the WGAN of Arjovsky et al. (2017),  $a = b = \mathrm{id}$ ) and regularization  $\mathcal{R}$  (e.g., the gradient penalty of Gulrajani et al. (2017)). In this work, we remain oblivious to how the discriminator is trained in practice as a single network alongside the generator. Nonetheless, we note that this GA is usually stopped early and not run until convergence, because the discriminator is trained only for a few steps between generator updates.
This discriminator is then used to train the generator, as usually framed in a min-max optimization setting. However, several works (Metz et al., 2017; Franceschi et al., 2022; Yi et al., 2023) showed that generator optimization deviates from min-max optimization, because alternating updates between the generator and the discriminator make the generator minimize a loss function of the form (Franceschi et al., 2022):
$$
\mathcal {L} _ {\mathrm {G A N}} \left(g _ {\theta}\right) = \mathbb {E} _ {z \sim p _ {z}} \left[ \left(c \circ f _ {\rho}\right) \left(g _ {\theta} (z)\right) \right], \tag {16}
$$
for some  $c\colon \mathbb{R}\to \mathbb{R}$  (e.g., for WGAN,  $c = \mathrm{id}$ ). Using Definition 2, we deduce the following.
Finding 5. GANs are Int-PMs with  $h_{\rho} = -c \circ f_{\rho}$ , where  $f_{\rho}$  is the current discriminator.
Under some assumptions on the outcome of discriminator training in Equation (15), the resulting  $\nabla h_{\rho}$  for GANs has been proven to implement a Wasserstein gradient  $-\nabla_W\mathcal{F}(\rho)$ . Two notable examples are (see also Section 2.1):  $f$ -divergence GANs (Nowozin et al., 2016), which are linked to the forward KL divergence gradient flow (Yi et al., 2023) and therefore to diffusion models; and Integral Probability Metrics (IPM) GANs, which are linked to the squared MMD gradient flow w.r.t. the NTK of the discriminator (Franceschi et al., 2022). However, these links have been made under strong simplifying assumptions, and the GAN formulation as an Int-PM in this paper is far more general.
Finding 6.  $f$ -divergence GANs as Int-PMs generalize the forward KL gradient flow and Langevin diffusion models, and IPM GANs generalize MMD gradient flows.
3.3 Stein Gradient Flows as Int-PMs
Int-PMs as framed in Definition 2 are similar to Stein gradient flows (Liu & Wang, 2016; Liu, 2017; Duncan et al., 2023). The latter are a generalization of Wasserstein gradient flows in another geometry shaped by a matrix kernel  $k \colon \mathbb{R}^D \times \mathbb{R}^D \to \mathbb{R}^{D \times D}$  defined in the data space:
$$
\mathrm {d} x _ {t} = - \mathbb {E} _ {x _ {t} ^ {\prime} \sim \rho_ {t}} \left[ k \left(x _ {t}, x _ {t} ^ {\prime}\right) \nabla_ {W} \mathcal {F} \left(\rho_ {t}\right) \left(x _ {t} ^ {\prime}\right) \right] \mathrm {d} t. \tag {17}
$$