SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This makes it possible to integrate the stochastic component of diffusion models into Score GANs by interpolating between Gaussian noise and the score of the generated distribution in step 5 of Algorithm 1, similarly to the previous equation using  $\alpha$ . Nonetheless, this comes with no guarantee on experimental performance, as we found in preliminary experiments that adding such a stochastic component is often detrimental to the resulting FID. Indeed, to succeed, the chosen gradient vector field to follow with the generator must be compatible with the generator architecture (i.e., compatible with the generator preconditioning  $\mathcal{A}_{\theta_t}(z)$ ), which may not be the case with white noise of high variance.
A.2 Discretizing Continuous-Time Equations
Studying how discretizing the considered continuous-time phenomena could affect our formulations is an interesting perspective for future work. We initiate a discussion on this topic below.
Choosing the best discretization method for diffusion models is challenging (Karras et al., 2022); since this depends on the chosen  $\sigma(t)$ , its efficiency is assessed w.r.t. the number of score queries instead of the number of discretization steps like in numerical methods, and the final purpose of discretization (generating realistic data) differs from its initial purpose (approximating a solution to a differential equation). Therefore, standard approaches like EDM rely on empirical discretization grids and custom solvers, tailored to the generation task. Our framework, by identifying the true probability flow in Equation (9), may help diffusion models cope with discretization errors through the score of the generated distribution.
The previous discussion, however, only holds for score-based diffusion models, for which the probability path is known in advance. This is not possible for other particle models, and studying the convergence properties of their discretizations, like Arbel et al. (2019) do for MMD, is non-trivial.
Adding a generator is an alternative way to solve the underlying particle model differential equation. By generalizing the parallel between Wasserstein and Stein gradient flows in Section 3.3, generators can be seen in our framework as a preconditioning over the particle model differential equation via the linear operator  $\mathcal{A}_{\theta_t}(z)$  in Equation (13). A well-chosen architecture, adapted to the particle flow  $\nabla h$ , may speed up and simplify the dynamics towards the data distribution.
A.3 Second-Order Methods and Adam
The framework we introduce relies on first-order solvers for PMs and first-order optimization for Int-PMs. Yet, in practice, we use Adam (Kingma & Ba, 2015), a second-order momentum-based optimizer, to train generators - cf. Appendix D.2. Theoretically, it is entirely possible to formulate continuous-time equations for Adam, paving the way for a generalization of our results to a second-order setting.
By fixing the values of  $(\beta_{1},\beta_{2})$  and allowing the time step to approach zero, we can recover the continuous version of SignSGD. A relevant example of SignSGD's study within a non-convex context was presented by Bernstein et al. (2018). Furthermore, exploring the scenario where non-interaction is present (when  $\mathcal{A}_{\theta_t}$  disappears from Equation (13) as discussed in Section 3.1) reveals a particle gradient flow with a renormalized gradient. This yields intriguing connections with continuous acceleration, as demonstrated by Wibisono & Wilson (2015). We consider this avenue to be a promising direction for future investigation.
A.4 Time Efficiency of Score GAN and Discriminator Flow
The design of Score GAN and Discriminator Flow induces computational constraints that make each of their training iterations slower than those of baseline models. Score GAN requires pretraining a score network as specified in the paper, and its score-based update remains computationally more demanding than a discriminator-based update like in GANs – since the score function takes values in the data space, while the discriminator output is scalar. Training Discriminator Flow requires sampling at every step from the generating differential equation of Equation (20). This makes its training iterations slower than both diffusion models (which do not require resampling through the differential equation) and GANs (which have fast sampling).
Besides the cost of individual training iterations, the total temporal cost of training also depends on the number of iterations, which we specify in Appendix D.3.
B Algorithmic Details
We detail in this section some aspects of the Score GAN and Discriminator Flow algorithms that were described at a high level in Section 4.
B.1 Score GANs
Algorithm 1 involves two major steps: line 3 performs denoising score matching for the generated distribution, and line 5 updates the generator parameters using the resulting gradient flow in Equation (19). We describe some implementation tricks for these steps in the following subsections.
B.1.1 Denoising Score Matching
Denoising score matching was described in line 3 of Algorithm 1, as originally used by Song & Ermon (2019). Following best practices later introduced by Karras et al. (2022), instead of directly training a network  $s_{\phi}^{\rho}$  to estimate the score, we instantiate and train a denoising network  $d_{\phi}^{\rho}$  using the following update:
$$
\phi \leftarrow \phi - \lambda \nabla_ {\phi} \left\| d _ {\phi} ^ {\rho} \left(x ^ {\sigma}, \sigma\right) - x \right\| _ {2} ^ {2}, \tag {23}
$$
where  $d_{\phi}^{\rho}$  is implemented as a U-Net (Ronneberger et al., 2015), with additional input-output skip connections for better preconditioning (Karras et al., 2022). We then use  $d_{\phi}^{\rho}$  to compute the estimated score:
$$
s _ {\phi} ^ {\rho} (x, \sigma) = \frac {d _ {\phi} ^ {\rho} (x , \sigma) - x}{\sigma^ {2}}. \tag {24}
$$
We use the same tricks to pretrain the score model of the data distribution,  $s_{\psi}^{p_{\mathrm{data}}}$ .
B.1.2 Generator Training
Line 5 of Algorithm 1 indicates how to update the generator parameters, following Equation (11). In order to facilitate its implementation in deep learning frameworks, we instead use the equivalent
Algorithm 3: Training iteration for Discriminator Flow (detailed).
Input: Batch size  $B\in \mathbb{N}^*$  , number of steps  $N\in \mathbb{N}^*$  , initial distribution  $\pi = \rho_0$  gradient strength  $\eta \in \mathbb{R}_{+}$  , previous discriminator  $f_{\phi}\colon \mathbb{R}^{D}\times \mathbb{R}\to \mathbb{R}$    
Output: Updated discriminator  $f_{\phi}$    
1 for  $b = 1$  to  $B$  do // In parallel   
2  $\begin{array}{rlr}{x^b\sim p_{\mathrm{data}},x_0^b\sim \pi ;} & {} & {\mathrm{//~Initialization}} \end{array}$    
// Partial generation   
3 for  $i = 0$  to  $N - 1$  do // Solve Equation (20)   
4  $\left| \begin{array}{l}x_{i + 1 / N}^b\gets x_{i / N}^b -\frac{\eta}{N}\nabla_{x_{i / N}^b}\Big[(c\circ f_\phi)(x_{i / N}^b,\frac{i}{N})\Big];\\ i_b\sim \mathcal{U}([0,N - 1])\end{array} \right.$  // Select random step   
// Train discriminator  $f_{\phi}$  at the chosen random steps, cf. Equation (15)   
6  $\phi \gets \mathrm{GA}_{\phi}\left\{\mathcal{L}_{\mathrm{d}}\Bigg(f_{\phi}\Big(\cdot ,\frac{i_b}{N}\Big);\mathcal{U}\Bigg(\Big\{x_{i_b / N}^b\Big\}_{b\in [1,B]}\Bigg),\mathcal{U}\Big(\{x^b\}_{b\in [1,B]}\Big)\Bigg)\right\}$
weight update:
$$
\left. \right. \theta \leftarrow \theta + \eta \nabla_ {\theta} \left[ g _ {\theta} (z) ^ {\top} \operatorname {S t o p G r a d i e n t} \left(s _ {\psi} ^ {p _ {\text {d a t a}}} \left(g _ {\theta} (z) + \sigma \varepsilon , \sigma\right) - s _ {\phi} ^ {\rho} \left(g _ {\theta} (z) + \sigma \varepsilon , \sigma\right)\right)\right], \tag {25}
$$
that is, we optimize the loss  $-g_{\theta}(z)^{\top}\mathrm{StopGradient}\Big(s_{\psi}^{p_{\mathrm{data}}}\big(g_{\theta}(z) + \sigma \varepsilon ,\sigma \big) - s_{\phi}^{\rho}\big(g_{\theta}(z) + \sigma \varepsilon ,\sigma \big)\Big)$ . We also adapt the weight update as follows. While keeping the same  $h_\rho$  as in Equation (19), we multiply it in the weight update by  $\sigma$  (which amounts to changing the noise level sampling distribution  $p_{\sigma}$  accordingly). When implemented with a denoiser network as described in Appendix B.1.1, the score estimation  $s_{\phi}^{\rho}(x,\sigma)$  can be numerically unstable for small  $\sigma$ s when using Equation (24). This explains why Karras et al. (2022) consistently multiply scores by  $\sigma$ , which is a choice that we follow for Score GANs. Furthermore, we can generalize Score GANs to other functionals  $h_\rho$  encompassing the EDM formulation of Equation (9), right:
$$
h _ {\rho_ {t}} = \mu_ {p _ {\mathrm {d a t a}}} \log \left[ p _ {\mathrm {d a t a}} \star k _ {\mathrm {R B F}} ^ {\sigma} \right] - \mu_ {\rho} \log \left[ \rho_ {t} \star k _ {\mathrm {R B F}} ^ {\sigma} \right], \tag {26}
$$
where  $\mu_{p_{\mathrm{data}}}$  and  $\mu_{\rho}$  are constant hyperparameters (e.g., for EDM,  $\mu_{p_{\mathrm{data}}} = 2$  and  $\mu_{\rho} = 1$ ).
Overall, in practice we implement the generator weight update of line 5 from Algorithm 1 as:
$$
\theta \leftarrow \theta + \eta \nabla_ {\theta} \left[ \sigma \cdot g _ {\theta} (z) ^ {\top} \operatorname {S t o p G r a d i e n t} \left(\mu_ {p _ {\text {d a t a}}} s _ {\psi} ^ {p _ {\text {d a t a}}} \left(g _ {\theta} (z) + \sigma \varepsilon , \sigma\right) - \mu_ {\rho} s _ {\phi} ^ {\rho} \left(g _ {\theta} (z) + \sigma \varepsilon , \sigma\right)\right) \right]. \tag {27}
$$
B.2 Discriminator Flows
In Algorithm 3 we provide a detailed implementation of the high-level Algorithm 2 for Discriminator Flows, including how batching and differential equation discretization are handled. For the latter, we use the Euler method with a uniform temporal grid, with  $N$  steps, in [0, 1]. While higher-order solvers and more optimal temporal grid choice could have been used, as in EDM (Karras et al., 2022), we avoid using any refinement of time discretization for Discriminator Flows for the sake of simplicity. For batch-parallel execution on GPUs, we solve the differential equation of Equation (20) over  $[0, 1]$  for all batch samples, and then select a random time for each sample to compute the discriminator loss and update its parameters.
In this section we present additional experiments that will help the reader develop better intuition on the behavior of both Score GANs and Discriminator Flows in practice.
Figure 3: Uncurated samples from studied models trained on CelebA and MNIST.
Figure 4: Uncurated samples of a Score GAN variant, which shows strong mode collapse on MNIST. Note that the Score GAN parameters were intentionally chosen to obtain this behavior.
C.1 Additional Samples
Figure 3 shows additional samples for all four tested models. Furthermore, for better visualization purposes, our public repository https://github.com/White-Link/gpm includes animated images illustrating the generation process for EDM and Discriminator Flows, as illustrated in Figure 1.
C.2 Mode Collapse on Score GANs
A widely known issue for GANs, identified soon after their introduction, is known as mode collapse (Goodfellow, 2016), which occurs when the generator only covers a fraction of the generated distribution. Interestingly, we observe the same phenomenon in Score GANs. We illustrate this with an extreme example in Figure 4, where we intentionally change parameters in our original model in order to induce the mode collapse issue. We induce mode collapse by choosing  $\mu_{p_{\mathrm{data}}} = 2$  and  $\mu_{\rho} = 1$  in Equation (27) (i.e., EDM parameters instead of NCSN parameters in the original model).
Since mode collapse is absent from the generator-less particle models that we tested (the score models on which Score GANs are based, and also Discriminator Flows), this observation suggests that mode collapse is primarily caused by the generator. This is in line with previous theoretical and empirical findings identifying the generator as the cause of mode collapse (Tanielian et al., 2020; Durr et al., 2022).
C.3 Latent Interpolations
We provide examples of interpolations on MNIST for all four tested models in Figure 5. We tested four interpolation methods on Gaussian priors considered by Lesniak et al. (2018): linear, spherical, Cauchy-linear, and spherical Cauchy-linear. We reached the same conclusion for each of these methods and thus only show the result of the most visually appealing one, spherical Cauchy-linear.
We notice that the generator-based models, Score GANs and GANs, show smoother transitions between generated images than the particle models EDM and Discriminator Flows, for which abrupt changes of digit identity and shape can be seen between consecutive interpolation steps. This confirms that interacting particle models (generator-based) can perform feature learning via their smaller latent space, allowing for smoother generation, while particle models operating in the data space are less prone to such phenomenon. In this regard, Score GANs and Discriminator Flows are no different than their parent method in the same model category.
C.4 Alternating Updates of Generator and Score in Score GANs
There are  $K$  steps of score updates per generator update in Score GANs, similarly to discriminators in GANs. Accordingly,  $K$  is an important parameter in Score GANs. First of all, like in GANs, the tuning of  $K$  heavily depends on the ratio  $r = \frac{\lambda}{\eta}$  between the learning rates of the score network and
(a) EDM.
(b) Discriminator Flow.
(c) Score GAN.
(d) GAN.
Figure 5: Interpolations (row-wise) in the latent space of the studied models (uncurated samples) at regular intervals.
(a) At the  $50^{\mathrm{th}}$  generator training iteration.
Figure 6: Evolution on MNIST of the training loss in-between two generator updates of the score of the generated distribution  $s_{\phi}^{\rho}$ , for  $K = 100$  and equal score/generator learning rates  $\lambda = \eta$ .
(b) At the  $3200^{\mathrm{th}}$  generator training iteration.
the generator (Jelassi et al., 2022) - cf. Algorithm 1. A higher ratio may allow us to decrease the necessary number of steps  $K$ .
In our experiments on image data, we use  $r \geq 2$  (Table 7). However, to gain more intuition empirically, we performed a set of experiments on MNIST with  $\lambda$  such that  $r = 1$ , by making the number of steps  $K$  vary from 1 to 10. We obtained similar results across this range of values for  $K$ , close to the values reported in Table 3. This indicates that even low values of  $K$  can provide a sufficient approximation of the score of the generated distribution.
To observe this qualitatively, we plot in Figure 6 the evolution of the score training loss in between generator updates for  $K = 100$ . At the beginning of training,  $s_{\phi}^{\rho}$  needs around 20 updates to converge. Yet, after a small number of generator updates, it is already close to the optimum before its first update.
This confirms that the continuous update of the score of the generated distribution  $s_{\phi}^{\rho}$ , like a discriminator, coupled with a learning rate ratio  $r > 1$ , makes a small number of score updates  $K$  between generator updates sufficient for adequate performance.
C.5 Time Efficiency of Discriminator Flows
The qualitative experimental results in Section 4 suggest that Discriminator Flows, by learning a path towards the data distribution with a discriminator, may converge faster towards  $p_{\mathrm{data}}$  than diffusion
(a) Stopping the generative process of Equation (20) at an earlier time  $T'$  than in training  $(0 < T' < T = 1)$ .
(b) Time efficiency comparison between alternative samplers of Discriminator Flows and EDM.
Figure 7: FID performance versus NFE (neural of function evaluations, i.e., the number of times  $\nabla h_{\rho}$  is queried to produce a single image) for EDM and Discriminator Flow on MNIST. We test different NFE modulation methods for the Discriminator Flow, in order to evaluate the time efficiency of each method. We consider two standard differential equation solvers for EDM: Euler and Heun.
models, for which the path towards the data distribution is determined by the diffusion equation, which must be followed until the final time  $T$ .
We quantitatively confirm this observation in Figure 7(a), which displays the performance of EDM and Discriminator Flow on MNIST for intermediate generated distributions  $\rho_{t}$ ,  $t < T$ , i.e., early stopping of the generation process. As expected given that intermediate generations from diffusion models are noisy, Discriminator Flows converge faster towards the data distribution. This raises the question of whether this faster convergence can yield a better efficiency for Discriminator Flows than for diffusion models. We investigate this claim in the following.
Time efficiency in diffusion models is usually measured, as done by Karras et al. (2022), in terms of generative performance versus number of neural function evaluations (NFEs): the most costly operation is the repeated evaluation of the score network throughout the diffusion process. We then study the efficiency of EDM and Discriminator Flow using this metric.
Since Discriminator Flows are based on a discretized differential equation, we can measure the time efficiency of this model by decreasing the number of neural function evaluations (NFEs) in two ways: by using a larger time discretization of Equation (20) or by an early stop of the generative process at time  $T' < T$  as already described above. For the first alternative, we train the evaluated model with 128 generative steps and  $\eta = 1$ ; for the second one, we train the evaluated model with 64 generative steps and  $\eta = 2$ , i.e., with half as many generative steps as the first one but with a doubled velocity, c.f. Algorithm 3. We compare both alternatives against two standard inference methods for EDM
based on the (first-order) Euler solver (Kloeden & Platen, 1992), and the (second-order) Heun/EDM sampler (Karras et al., 2022). We show the results obtained for MNIST in Figure 7(b).
Overall, the Discriminator Flow model is comparable to the second-order version EDM; however, both remain outperformed by the first-order version of EDM. Surprisingly, the first-order version of EDM is actually more efficient in terms of NFE than the second-order version for low NFE values. This baseline was not considered by Karras et al. (2022), so this is a new observation; we confirmed this phenomenon using the official implementation of EDM on CIFAR10 (Krizhevsky, 2009). This behavior is especially visible in our case as we test it on a simpler dataset.
Therefore, additional experiments are necessary to form a conclusion on the relative efficiency of Discriminator Flows w.r.t. diffusion models. We stress that the current results are achieved with Discriminator Flows discretizing Equation (20) on a regular temporal grid, while EDM discretize time for both first- and second-order methods using a custom temporal grid that improves discretization performance. Finally, we believe that further tuning of Discriminator Flows could significantly improve its efficiency: while the final image appears quickly after a few steps as illustrated in Figure 1, some imperceptible residual noise still needs to be eliminated in the remaining steps. This noise prevents steady convergence towards  $p_{\mathrm{data}}$ ; removing it one of the main directions of improvement for Discriminator Flows.
C.6 Experiments on Gaussians
As a toy example, we train all considered models (Discriminator Flows, GANs, EDM, and Score GANs) on synthetic samples generated from a two-dimensional mixture of Gaussian distributions. We provide a visualization of the particle evolution w.r.t. training and inference time in our repository https://github.com/White-Link/gpm, as well as in Figures 8 to 12. See the figure captions for more information.
We again observe that Discriminator Flows converge faster than EDM towards the data distribution.
We provide all details in this section that are necessary to reproduce our experiments. Our Python source code (tested on version 3.10.4), based on PyTorch (Paszke et al., 2019) (tested on version 1.13.1), is available as open source at https://github.com/White-Link/gpm.
D.1 Datasets and Evaluation Metric
MNIST. MNIST is a standard dataset introduced in LeCun et al. (1998), with no clear license to the best of our knowledge, composed of monochrome images of hand-written digits. Each MNIST image is single-channel, of size  $28 \times 28$ . We preprocess MNIST images by extending them to  $32 \times 32$  frames (padding each image with black pixels), in order to better fit as inputs and outputs of standard convolutional networks. We linearly scale pixels values so that they lie in  $[-1,1]$ . MNIST is comprised of a training and testing dataset, but no validation set; we create one for each model training by randomly selecting  $10\%$  of the training images.
CelebA. CelebA (Liu et al., 2015) is a dataset composed of celebrity pictures. Its license permits use for non-commercial research purposes. Each CelebA image has three color channels, and is of size  $178 \times 218$ . We preprocess these images by center-cropping each to a square image and resizing to  $64 \times 64$  with a Lanczos filter. We linearly scale pixels values so that they lie in  $[-1, 1]$ .
Gaussians. Our Gaussian dataset is composed of a mixture of 5 two-dimensional Gaussian distributions with standard deviation  $\frac{1}{2}$ , with means evenly spaced over a circle of radius 5. Training, validation, and testing datasets all consist of i.i.d. samples from this mixture.
FID. Throughout the paper we use the Fréchet Inception Distance (FID, Heusel et al., 2017) to measure the generative performance of the models we consider. In our code we use the PyTorch implementation of TorchMetrics (Skafte Detlefsen et al., 2022).
We summarize the model hyperparameters used during training in Tables 4 to 7. See our code for more information. We further discuss some aspects of our implementation choices in the remainder of this subsection.
Networks. Beyond multi-layer perceptrons (MLP) and EDM's U-Nets, we use three kinds of model architectures: DCGAN (Radford et al., 2016), a ResNet (Kang et al., 2022) based on SAGAN with self-attention (Zhang et al., 2019), and FastGAN (Liu et al., 2021). We adapt these models in the following ways:
- For MNIST images, we changed the first (respectively, last) convolution of the DCGAN discriminator (respectively, generator) to adapt it for  $32 \times 32$  inputs (respectively, outputs).
- We adapted FastGAN to operate on images of size  $32 \times 32$  and  $64 \times 64$  by removing layers with higher resolutions.
- We added a bias to the first convolutional layers of discriminators, which did not have one.
- We enhanced these models so that they can accept a vectorial embedding of time  $t$  for Discriminator flows, by modulating most of their convolution outputs (before the activation) channel-wise with an affine transformation. The affine transformation parameters are the outputs of a MLP of depth 2, with SiLU activations (Hendrycks & Gimpel, 2016) and a hidden width that is twice as large as the input time embeddings.
Final generator activation. Since image pixel values lie in  $[-1, 1]$ , we add a final hyperbolic tangent activation to the output of the generators operating on image data.
Score GAN noise distribution  $p_{\sigma}$ . Since during training Score GANs mimic the inference procedure of a diffusion model, we choose to sample  $\sigma$  during Score GAN training time following the schedule chosen by EDM at inference time (Karras et al., 2022). In practice, we choose a minimal value  $\sigma_{\mathrm{min}}$ , a maximal value  $\sigma_{\mathrm{max}}$ , and an interpolation parameter  $\rho$ . To sample from  $p_{\sigma}$ , we first sample an interpolation value  $\alpha \sim \mathcal{U}\big([0,1]\big)$ , and then compute  $\sigma$  as:
$$
\sigma = \left(\sigma_ {\max } ^ {\frac {1}{\rho}} + \alpha \left(\sigma_ {\min } ^ {\frac {1}{\rho}} - \sigma_ {\max } ^ {\frac {1}{\rho}}\right)\right) ^ {\rho}. \tag {28}
$$