Generative Sliced MMD Flows with Riesz Kernels
Authors: Johannes Hertrich
Christian Wald
Fabian Altekrüger
Paul Hagemann
Venue: ICLR 2024 poster (2024)
Keywords: sliced maximum mean discrepancy
energy distance
gradient flows
Riesz kernels
generative modelling
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Maximum mean discrepancy (MMD) flows suffer from high computational costs in large scale computations.
In this paper, we show that MMD flows with Riesz kernels $K(x,y) = - \|x-y\|^r$, $r \in (0,2)$
have exceptional properties which allow their efficient computation.
We prove that the MMD of Riesz kernels, which is also known as energy distance, coincides with the MMD of their sliced version.
As a consequence, the computation of gradients of MMDs can be performed in the one-dimensional setting.
Here, for $r=1$, a simple sorting algorithm can be applied to reduce the complexity
from $O(MN+N^2)$ to $O((M+N)\log(M+N))$ for two  measures with $M$ and $N$ support points.
As another interesting follow-up result, the MMD of compactly supported measures
can be estimated from above and below by the Wasserstein-1 distance.
For the implementations we approximate the gradient of the sliced MMD by using only a finite number $P$ of slices. 
We show that the resulting error has complexity \smash{$O(\sqrt{d/P})$}, where $d$ is the data dimension. 
These results enable us to train generative models by approximating MMD gradient flows by neural networks even
for image applications. We demonstrate the efficiency of our model by image generation on MNIST, FashionMNIST and CIFAR10.

Published as a conference paper at ICLR 2024
GENERATIVE SLICED MMD FLOWS WITH RIESZ KERNELS
Johannes Hertrich', Christian Wald Fabian Altekrüger", Paul Hagemann* 1 University College London, Technische Universität Berlin, Humboldt- Universitt zu Berlin Correspondence to: j w.eherichhuull.. ac..llacc... uk
ABSTRACT
Maximum mean discrepancy (MMD) flows suffer from high computational costs in large scale computations. In this paper, we show that MMD flows with Riesz kernels K (zz y) -II yl E (0, 2) have exceptional properties which allow their efficient computation. We prove that the MMD of Riesz kernels, which is also known energy distance, coincides with the MMD of their sliced version. As a consequence, the computation of gradients of MMDs can be performed in the oie-dimensional setting. Here, for 1, simple sorting algorithm can be applied to reduce the complexity from ( N + 2 to O (M + N) log( M + N)) for two measures with M and N support points. As another interesting follow-up result, the MMD of compactly supported measures can be estimated from above and below by the Wasserstein-I distance. For the implementations we approximate the gradient of the sliced MMD by using only a finite number P of slices. We show that the resulting error has complexity O P(Vd/P), where d is the data dimension. These results enable us to train generative models by approximating MMD gradient flows by neural networks even for image applications. We demonstrate the efficiency of our model by image generation on MNIST, FashionMNIST and CIFARIO.
INTRODUCTION
With the rise of generative models, the field of gradient flows in measure spaces received increasing attention. Based on classical Markov chain Monte Carlo methods, |Weling & Tch (2011) proposed to apply the Langevin dynamics for inferring samples from a known probability density function. This corresponds to simulating Wasserstein gradient flow with respect to the Kullback-Leibler
divergence, see Jordan et al. (1998). Closely related this approach are current state-of-the-an
image generation methods like score-based models (Song & Ermon 2019: 2020) or diffusion models 202T), which significantly outperform classical generative models
(Ho et al. 2020 Song et al.
2014) or VAEs (Kingma & Welling) 2014). A general aim of such algorithms (Arbel et al. 2021: Ho et al. 2020: Wu et al. 2020) is to establish a path between input and target distribution, where 'unseen" data points established via the randomness of the input distribution. Several combinations of such Langevin-type Markov chain Monte Carlo methods with other generative models were proposed in (Ben-Hamu al. 2022 Hagemann et al.. [023 Wu et al.)
like GANs (Goodfellow et al.
2020). Gradient flows on measure spaces with respect to other metrics are considered in (di Langosco
al. 2022 Dong et al., 2023 Grathwohl et al. 2020
Liu 2017
Liu & Wang 2016) under the name
Stein variational gradient descent.
For approximating gradient flows with respect to other functionals than the KL divergence, the authors of (Altekrüger et al.) (2023] Ansari et al.) 2021; Alvarez-Melhi et al.) 2022 Bunne et al.
2022. Fan et al.) 2022; [ao et al., 2019; [Gariia Trillos al., [023] (Hnn et al. 2023 Mokrov et al. 2021; Peyré 2015) proposed the use of suitable forward and backward discretizations. To reduce the computational effort of evaluating distance measures on high-dimensional probability distributions, the sliced Wasserstein metric was introduced in (Rabin et al.) 2012). The main idea of the sliced Wasserstein distance is to compare oie-dimensional projections of the corresponding probability distributions instead of the distributions themselves. This approach be generalized to more general probability metrics (Kolouri et al. 2022) and was applied in the context of Wasserstein
gradient flows in (Bonet al. 2022b Liutkus et al. 2019)

---

Published as a conference paper at ICLR 2024
10 Puutic PP550003)
Dne P-N 9-223
107
THE JIND -3300
1300
Reuove Sme
226
1003
150C
3330
10
Projections
Dimerraon km
Figure 1: Left: Comparison of run time for 1000 gradient evaluations of naive MMD and sliced MMD with different number of projections P in the case d 100. Middle and right: Relative error of the gradients of sliced MMD and MMD with respect to the number P of projections and the dimension d. The results show the relative error behaves asymptotically as O (/)d/P) as shown in Theorem
For many generative gradient-flow methods it is required that the considered functional can be evaluated based samples. For divergence-based functionals like the Kullback-Leible or the Jensen- -hanennn- divergence, a variational formulation leading to a GAN-like evaluation procedure is provided in (Fan et al. 2022). In contrast, the authors of (Altekrüger al., 2023; Arbel al. 2019 Glaser et al. (222)) use functionals based on the maximum mean discrepancy (MMD) which can be directly evaluated based on empirical measures. For positive definite kernels, it can be shown under some additional assumptions that MMD defines a metric on the space of probability distributions, see e.g., (Gretton et al. 2012; Sriperumbudur et al. 2011 2010). If the considered kernel is smooth, then AArbel et al. (2019) proved that Wasserstein gradient flows can be fully described by particles. Even though this is no longer true for non-smooth kernels (Hertrich et al. 2023b), Altekrüger et al. (2023) pointed out that particle flows are Wasserstein gradient flows at least with respect to restricted functional. particular, we can expect that particle flows provide an accurate approximation of Wasserstein gradient flows as long the number of particles is large enough.
Contributions. The computational complexity of MMD between two empirical measures with N and M support points depends quadratically on N and M, which makes large scale computations impossible. In this paper, we focus on the MMD with Riesz kernels
K k(,,v) = - || y/l",
(0,
also known as energy distance (Sejdinovic et al.) 2013 Székely 2002 Székely & Rizzo 2013). We show that Riesz kernels have the outstanding property that their MMD coincides with the sliced MMD of univariate Riesz kernels. this property that enables us to reduce the computation of (gradients of) MMD to the one- mne-iiensioonal setting. In the case of r 1, we propose simple and computationally very efficient sorting algorithm for computing the gradient of the one-dimensional MMD with complexity O(MM + log + N)). Considering that our numerical examples use between 10.000 and 50.000 particles, this leads to an incredible speed-up for gradient computations of (sliced) MMD as illustrated in the left plot of Figure [ Our approach opens the door to applications in image processing, where we have usually to cope with high dimensional data.
In practice, sliced probability metrics are evaluated by replacing the expectation over all projections by the empirical expectation resulting in a finite sum. In the case of sliced MMD with Riesz kernels and
r= 1, we prove that the error induced by this approximation behaves asymptotically as ((d/)) where d is the data dimension and P the number of projections, see the middle plot in Figure I for an illustration. The square root scaling of the error in the dimension d ensures that an accurate computation of the sliced MMD with Riesz kernels is possible even in very high dimensions as
demonstrated in the right plot in Figure] I Taking into account the number of projections, the overall complexity of the computation of the derivatives of MMD is (M + N) log M + N)).
We apply the cheap evaluation of MMD gradients to compute MMD particle flows starting with samples from an initial probability measure (Ao to samples from a predefined target distribution v. which is given by samples. Finally, we derive generative model by training sequence (4:) , of neural networks, where each . approximates a certain number of steps of the particle flow. This allows us to train network iteratively. In particular, during the training and evaluation procedure, we always consider only one of the networks D at the same time. This allows an efficient training with relatively low resources even though all networks 4 together have a large number of parameters.
2

---

Published as a conference paper at ICLR 2024
We demonstrate the efficiency of our generative sliced MMD flows for image generation on MNIST, FashionMNIST and CIFARIO.
Related Work. Gradient flows with respect to MMD functionals are considered in (Altekrüger et al., 2023 Arbel et al. 2019; Hertrich et al. 2023b Kolouri et al. 2022). However, due to the quadratic complexity of the computation of the derivative of MMD functionals in the number of support points of the involved measures, these papers have a rather theoretical scope and applications are limited to measures supported on few hundred points. In order reduce the dimension of the problem, Kolouri et al. (0222) consider a sliced version of MMD. This is motivated by the success of sliced Wasserstein distances (Rabin et al., 2012), which were used for deriving gradient flows in (Bonet itt al. 20226; Liutkus et al., 2019: N guyen et al., 2023; 2021). In particular, Kolouri et al. (2022) observe that the sliced MMD is again MMD functional with a different kernel. We use this result in Section 2 Vayer & Gribonval (223) bound Wasserstein distances and MMD against each other. However, they use strong assumptions on the kernel, which are not fulfilled for the negative distance kernel. In very low dimensions, fast evaluations of MMD and their gradients were proposed in (Gräl ct al. 2012 Teuber et al.] (01T) based on fast Fourier summation using the non-equispaced fast Fourier transforms (NFFT), (Plonka al., 2018 Sec. 7) and references therein. Unfortunately, since the complexity of the NFFT depends exponentially on the data-dimension these approaches limited to applications in dimension four smaller. In one-dimensional setting, the energy distance is related to the Cramer distance, see (Székely) (002). In the context of reinforcement learning, Lhéritier Bondoux (202I) developed fast evaluation algorithms for the latter based on the calculation of cumulative distribution functions.
Finally, the authors of (Binkowski al. 2018 Dziugaite al., 2015 Li al. 2017 2015) apply
MMD for generative modelling by constructing so-called MMD-GANS. However, this is conception- ally very different approach since in MMD-GAN the discriminator in the classical GAN framework (Goodfellow et al.] (014) is replaced by MMD distance with variable kernel. Also relevant is the direction of Sobolev-GANs (Mroueh et al. 2018) in which the discriminator is optimized in a Sobolev space, which is related to the RKHS of the Riesz kernel. Similar to GAN ideas this results in a max-min problem which is solved in an alternating fashion and is not related to gradient flows.
Outline of the Paper. In Section 2 we prove that the sliced MMD with the one-dimensional Riesz kernel coincides with MMD of the scaled d-dimensional kernel. This can be used to establish an interesting lower bound on the MMD by the Wasserstein-1 1 distance. Then, in Section 3 we propose a sorting algorithm for computing the derivative of the sliced MMD in an efficient way. We apply the fast evaluation of MMD gradients to simulate MMD flows and to derive a generative model in Section 4 Section 5 shows numerical experiments on image generation. Finally, conlusions are drawn in Section 6 The appendices contain the proofs and supplementary material.
2 SLICED MMD FOR RIESZ. KERNELS
Let P(RR)) denote the set of probability measures on Rdd and (RR) its subset of measures with finite p-th moment, i.e., fe [|=]()))(())) < co. Here || denotes the Fuclidean norm on IRd, For a symmetric, positive definite kernel K: Rd x Rdd + R, the maximum mean discrepancy (MMD) Dx: P(R)) x P(Rd)) + IR is the square root of Di (u, v) = EK (u (u-v), v), where & is the interaction energy of signed measures on Rd defined by
Ex = JR JRd dn(())n()))
Due to its favorable properties, see Appendix EE] we are interested in Riesz kemnlss
y) - vll", E (0, 2).
These kernels are only conditionally positive definite, but can be extended to positive definite kernels by K (xxz) = K (x, (1,
K (xx,00 - (0. v). see also Remark 13 Then it holds for μ4, 2/ E P, (Rdd)
that D& v)) v) = DR R (1, v), see (Nuumayer & Steidl 2021 Lemma 3.3 ii)). Moreover, for Riesz
kernels, Dx is metric P, (Rd). which is also known as so-called energy distance (Sejdinovic
jet al. 2013) Szzkell & Rizzo 2013). Note that we exclude the case r = 2, since k is no longer a
metric in this case.
3

---

Published as a conference paper at ICLR 2024
However, computing MMDs high dimensional spaces is computationally costly. Therefore, the sliced MMD SD (P((I) x (2((R - IR. was considered in the literature, see e.g., Kolouri et al. (2022).
For a symmetric ID kernel
- R it given by
(u, Eg Eg- [D pe
with the push-forward measure Pe = μ P5- of the projection Pę(x) = (E, x) and the uniform distribution Uga-1 on the sphere S 1 By interchanging the integrals from the expectation and the definition of MMD, Kolouri et al. (202)) observed that the sliced MMD is equal to the MMD with an associate kernel K: R R² + R. More precisely, it holds
SDi( = (p, with ((z, y) = Bg- [k())()), Pe(v))
By the following theorem, this relation becomes more simple when dealing with Riesz kernels, since in this case the associate kernel is a Riesz kernel as well.
Theorem (Sliced Riesz Kernels are Riesz Kernels). Let k(x,, y) = x yl", r E (0,2). Then, for μ. V E (,R()) holds (D, v)) = Di (H, v) with the associated scaled Riesz kernel
K(x, v) = -c x0 yll",
Cd,r =
r
The proof given Appendix A The constant Cd. depends asymptotically with Odd'/²) on the dimension. In particular, it should be "harder" to estimate the MMD its gradients in higher dimensions via slicing. We will discuss this issue more formally later Remark For just write ca = Cd,1 and can consider measures in Pi (R.). Interestingly, based on Theorem can establish relation between the MMD and the Wasserstein-1 distance on Pi (IR defined by
we
/ || yll dz(,,)); *Ell(u,))
Wi v)
min
where I[(,, v) denotes the set measures in PIRR x Rd)) with marginals and v. This also shows
that Conjecture in (Modeste & Dombry 2023) can only hold for non-compactly supported measures.
The proof is given in Appendix B
Theorem 2 (Relation between D and Wi for Distance Kernels). Let K(x, y) == =- yll- Then, it holds for p., E Pi (R4) that 2DR (1, v) < Wi(,,,) If p. and are additionally supported on the ball (0), then there exists a constant Ca > 0 such that Wi (14, v) S C.R Dx v) गोr.
The fact that the sample complexities of MMD and Wasserstein-1 are Onn Gretton et al. 2012 Chapter 4. and (nn11/d) (Peyré Cuturi 2020 Chapter 8.4.1) suggests, that the exponent of D in Theorem 2 cannot improved over I/d.
GRADIENTS OF SLICED MMD
Next, consider the functional F,: P2(RR) + R given by
Fr(p) = (x()) + (M,) = Di const,,
(2)
where (H) is the so-called potential energy
(K,( == f
[ K(x, v) du(y) du(x)
nd
acting as an attraction term between the masses of μ and v, while the interaction energy & is a repulsion term enforcing a proper spread of p.. For the rest of the paper, we always consider the negative distance kernel K(x, y) := -Iix - y/I, which the Riesz kernel [ with r = 1. Then, we obtain directly from the metric property of MMD that the minimizer of the non-convex functional F, given by v. We are interested in computing gradient flows of F, towards this minimizer. However, the computation of gradients in measure spaces for discrepancy functionals with non-smooth kemelss is highly non conttiivial and computationally costly, see e.g., (Altekrüger et al., [2033 CCrrillo et al. 2020
Hertrich et al. 2023b).
4

---

Published as a conference paper at ICLR 2024
As remedy, we focus on a discrete form of the d-dimensional MMD. More precisely, we assume that N i and are empirical measures, i.e., they are of the form μ N Σ ii= 0g. and मे Σj= 1 o2z for some Mj E Rd. Let æ == (x1;-- .NN and y == (V1,...,,)) ".,....;MM) yM). Then the functional Fu reduces to the function Fa Fa(- INd + given by
N
Fa(xlly)
| -e-xxll
2 -ylll
2N
MN
M
D&
8
8ws
consty
M
In order to evaluate the gradient of Fa with respect to the support points æ, we use Theorem Ito rewrite Fa(
Fa(øly) uug [Fi ((), (S. ( ui),... (E, uM))].
Then, the gradient of Fa with respect to x, is given by
V xq (xlv)) = caBe- [0, (()(( (1),,,)))., (6, (s) vn), -.., (5. yM))s,
(4)
where 0 F1 denotes the derivative of F1 with respect to the i-th component of the input. Consequently, suffices to compute gradients of Fi order to evaluate the gradient of Fa-
A Sorting Algorithm for the 1D-Case. Next, we derive a sorting algorithm to compute the gradient of Fi efficiently. In particular, the proposed algorithm has complexity O((M + N) log(M + N))
even though the definition of F1 in (3) involves N + MM summands.
To this end, we split the functional Fj into interaction and potential energy, i.e., (a F((x) (ally) with
N
N
M c - My
F(x)
x rzl. V(xly) = NM
(5)
2N
j=
;=1
Then, we can compute the derivatives of and V by the following theorem which proof is given in Appendix C
Theorem 3 (Derivatives of Interaction and Potential Energy). Let Let ...., EN E R. be pairwise disjoint and yı...., UM E IR such that + yi for all I,..,NN and 1, M. Then, E and V are differentiable with
N N+1 20 20)
V Va 2 #Li E {1, M} yj < xx) M
Va, F(ar)
N2
MN
where a: {l,..., N) + (1, N} is the permutation with (1) <.. < (o((N)
Since V is convex, we can show with the same proof that
2 2 (1,.., M} M, < xx <x,- - M
E Oz, V(xly)
MN
where 0g is the subdifferential ov V with respect to x, whenever I; = J/ for some i.j. By Theorem 3 obtain that V (x]l) = E(x) + (V(w)) can be computed by Algorithm I and Algorithm with complexity O(N log( and O((M + N) log(M + N)), respectively. The complexity is dominated by the sorting procedure. Both algorithms can be implemented in vectorized form for computational efficiency. Note that by Lemma 9 from the appendix, the discrepancy with Riesz kernel and measures. Since the cdf of an one-dimensional empirical measure can be computed via sorting, we also obtain an O(NN + +M) log(M + N)) algorithm for computing the one-dimensional MMD itself and not only for its derivative.
can be represented by the cumulative distribution functions (cdfs) of the involved
Stochastic Approximation of Sliced MMD Gradients for 1. To evaluate the gradient of Fa efficiently, we use a stochastic gradient estimator. For æ1,..., n V/,, !/,..., ..., VM E Rdd, we define for P E N the stochastic gradient estimator of 4 as the random variable
N
V p Fa = (V Fa(a))))))))))
(6)
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 Derivative of the interaction energy E from (5).
Input: Æi, ....... IN R with 2; + zj for i # j.
Algorithm:
Compute OI,...ON
(21;
Compute gi 20,"" I-N
Output: (01,..., 9N) = VE(
Algorithm 2 Derivative of the potential energy V from (5)
Input: JI.... IN IR, y1...., VM with x; + U/- Algorithm:
Compute oI....: ON+ = argsort(z1.... .... NN y2...., yM
Initialize in =.... WM+N = 0.
f -,,,., do
Set ho(N+)) = 1. end for Set cumsum for i = 1, N do
Set gi MN
end for
Output: (g1. IN) =
yass)
where
P.a. Fa(xly) = Cd P
O, D. xI),... zi.. (Sp: 2N) vi),... (Epx Mar) )
for independent random variables Ep Uga-1. We obtain by that Fa is unbiased, ie.. (x)y) FFa(xly). Moreover, the following theorem shows that the error of V FF converges to zero for a growing number of projections. The proof uses classical concentration incqualities and follows directly from Corollary 12 in Appendix
Theorem 4 (Error Bound for Stochastic MMD Gradients). Let T1, -i,.... JN U1 UM E IRd. Then, it holds
V VFa(=))))1 VVFdeelll E o (
To verify this convergence rate numerically, we draw 1000 samples æI...., Ny from a Gaussian mixture model with two components and M 1000 samples y1, yas from Gaussian mixture model with ten components. The means are chosen randomly following a uniform distribution in [-1,1] and the standard deviation is set 0.01. Then, we compute numerically the expected relative approximation етror between FF and F. for different choices of P and The results are illustrated in the middle and in the right plot of Figure We observe that this numerical evaluation
underlines the convergence rate of O (v+ In particular, the error scales with O ((/d/ /d/P). which makes the approach applicable for high-dimensional problems.
Remark 5 (Computational Complexity of Gradient Evaluations). The appearance of the v in the error bound due to the scale factor ca between the MMD and the sliced MMD, which can be seen in the proof of l'Thorrem 4 In particular, we require O(d) projections in order to approximate Fa(x y) by PPFa(x ry) up to fixed expected error of e. Together with the computational complexity of (dP((N + M) log(N + M)).or Fala,, y), obtain overall complexity of O&l (N + M) log(N + M)) in order to approximate Fa(x yy) up to an expected error of e. On the other hand, the naive computation (gradients of) (a(xly) has complexity of Odd(N2 + MN)).

---

Published as a conference paper at ICLR 2024
Consequently, improve the quadratic complexity in the number of samples O log(N)). Here, we pay the price of quadratic instead of linear complexity in the dimension.
4 GENERATIVE MMD FLOWS
In this section, we use MMD flows with the negative distance kernel for generative modelling. Throughout this section, assume that we are given independent samples yi..... VM E R from target measure v (2((R) and define the empirical version of by PM 11 Sy.
4.1 MMD PARTICLE FLOWS
In order to derive a generative model approximating v, we simulate a gradient flow of the functional F. from 2). Unfortunately, the computation gradient flows in measure spaces for F, is highly non-trivial and computationally costly, see (Altekrüger et al. 2023 Hertrich et al. 2023b). Therefore, we consider the (rescaled) gradient flow with respect to the functional Fa instead. More precisely, we simulate for Fd from (3), the (Euclidean) differential equation
i = = FF
x(0) = (I)
(0) are drawn independently from some measure Ho P2())). our where the initial points x; numerical experiments, we set μi to the uniform distribution on [0. 1jd Then, for any solution azt) = (I)(t), ... (t)) of (7). is proven in (Altekrüger et al. 2023 Proposition 14) that the curve YM.N (0, oc) P2 (R defined by (M, -E=1 Sz,(1) is Wasserstein gradient flow with respect to the function
FUSS =f on, for some x, # xj Rd
F: P2(TT)) + TR (00), μM
+00, otherwise.
Hence, expect for M,N DC, that the curve "M,N approximates the Wasserstein gradient flow with respect to Fu.. Consequently, we can derive generative model by simulating the gradient flow 77. To this end, use the explicit Euler scheme
(k+1) (k)) TNVFF(x y).
(8)
where (k) (i) (k)
IN and 7 > is some step size. Here, the gradient on the right-hand side can be evaluated very efficiently by the stochastic gradient estimator from (6).
(K)
Momentum MMD Flows. To reduce the required number of steps in (8). introduce momentum
parameter. More precisely, for some given momentum parameter m E [0, 1) we consider the momentum MMD flow defined by the following iteration
(k+1) VFa( (y) + m (k)
(9)
x (++1) = x N (*+1))
where 7 >> 0 is some step size, x; (0) are independent samples from a initial measure μi and v Note that the MMD flow ( is special case of the momentum MMD flow with 0.
0.
In Figure 2 we illustrate the momentum MMD flow (9) and MMD flow (8) without momentum from uniform distribution [0, 1] to MNIST (LeCun al. 1998) and CIFARIO Krizzevsky (2009). The momentum is set to y = 0.9 for MNIST and to w
0.6 for CIFARIO. We observe that the momentum MMD flow (9) converges indeed faster than the MMD flow without momentum.
4.2 GENERATIVE MMD FLOWS
The (momentum) MMD flows from ( and 9 transform samples from the initial distribution μo into samples from the target distribution v. Therefore, we propose to train a generative model which approximates these schemes. The main idea is to approximate the Wasserstein gradient flow
7: [0, oo) + P2 (2 with respect to F, from 2 starting at some latent distribution Ho = 7(0). Then, iteratively train neural networks .,,... 2r such that 7(4) (1-7(111) for some I, with
7

---

Published as a conference paper at ICLR 2024
7
U
/
G
G
7
4
4
4
4
Z
8
8
K
5
(
6
5
5
6
6
6
9
Figure 2: Samples and their trajectories from MNIST (left) and CIFARIO (right) in the MMD flow with momentum [9] top) and without momentum (8) bottom) starting in the uniform distribution on [0. 1j after 2 steps with k E {o...., 16) (for MNIST) and k E We observe that the momentum MMD flow ( converges faster than the MMD flow (8) without momentum.
(for CIFAR 10).
0 0= 40 <1 < < lr. Then, for Lr large enough, holds v ((x.) (Ø,, ....o 11) (0)
with 7(0) 140 Such methods learning iteratively an "interpolation path" are exploited several times in literature, e.g., Arbel et al. (202T); T): Fan et al. (2022); ; Ho et al. (2020). To implement this
numerically, we train each network 2 such that it approximates Ti number of steps from (8) (9)
The training procedure of our generative MMD flow is summarized in Algorithm 3 in Appendix F
Once the networks 11 ..... & are trained, target distribution
infer new sample x from our (approximated)
follows. We draw sample (0)) from Moo compute x1() (1(-1) (x((11)
for / = 1, ...,. L and set = ((4), In particular, this allows us to simulate paths of the discrepancy flow we have not trained on.
Remark (Iterative Training and Sampling). Since the networks are not trained in an end-to-end fashion but separately, their GPU memory load is relatively low despite high number of trainable parameters of the full model (or) This enables training of our model on an 8 GB GPU. Moreover, the training can easily be continued by adding additional networks D., = 1..,...., Lt to already trained generative MMD flow (Dt) which makes applications more flexible.
NUMERICAL EXAMPLES
In this section, we apply generative MMD flows for image generation on MNIST, FashionMNIST (Xiao et al. 2017), CIFARIO and CelebA (Liu et al. 2015). The images from MNIST and Fashion- MNIST are 28 x 28 gray-value images, while CIFARIO consists of 32 x 32 RGB images resulting in the dimensions d 784 and d 3072, respectively. For CelebA, we centercrop the images to 140 X 140 and then bicubicely resize them to 64 x 64. We run all experiments either on a single To
NVIDIA GeForce RTX 3060 RTX 4090 GPU with 12GB or 24GB nemory,respectiine evaluate our results, we use the Fréchet inception distance (FID) (Heusel et al. 2017 between
We use the implementation from http://wwwrrritoooo//t./oooootte
httpaa/....oonneeeeee/etteteeeee fid
2
2
2
3
8
1
0
1
8
2
2
0
9
2
6
~
3
3
0
O
9
4
4
2
0
7
7
1
9
9
5
9
O
2
5
6
9
7
0
9
4
3
4
3
7
9
3
/
9
9
O
9
0
5
3
S
3
5 #
5 6
@
R
3
O
4
O
Figure 3: Generated samples of our generative MMD Flow.
8

---

Published as a conference paper at ICLR 2024
Table 1: FID scores for different datasets and various methods.
Method
MNIST
FashionMNIST
CIFARIO
CelebA
Auto-enccdder Intsed
CWAF [Knop et al, 2020
23.6
50.0
120.0
49.7
SWF+ Autoencoder RV
2022b
17.8
90.9
2-stage VAE Dai & Wipr 2019
12.6
29.3
72.9
44.4
GLF (Xiao
8.2
21.3
88.3
53.2
Adversarial
WGAN (Aruvskyy et 2017 el
41.3
MRD
[iikkowaki di 201N
4.2
48.1
29.2
Score-Inised
NCSN
Song & Ermon 2019
25,3
Fiow Buxed
SWF
Liuckus
2019
225.1
207.6
91.2
SIG
Lia Sclaak 4041
4.5
13.7
66.5
37.3
W-SWF
59.7
38.3
Generative Sliced MMD Flow (ours)
3.1 +: 0.06
11.31 0.07
54.8 1: 0.26
32.14 0.17
10K generated samples and the test dataset. Here, smaller FID value indicates a higher similarity between generated and test samples.
We choose the networks (4y f to be UNets (Ronneberger et al. 2015), where we use the implemen- tation from (Huang al. 2021 1) based on (Ho al. 2020). Then, run the generative MMD flow for L = 55 (MNIST), L- 67 (FashionMNIST), L S6 (CIFARIO) and L = 71 (CelebA) networks.
The exact setup is described in Appendix H We compare the resulting FIDs with other gradient- flow-based models and various further methods in T
We computed the standard deviations by independently sampling ten times from one training run and computing the corresponding FID. We observe that we achieve excellent performance on MNIST and FashionMNIST well as very good results on CIFARIO and CelebA. Generated samples are given in Figure 3 and more samples given in Appendix The L2-nearest neighbors of the generated samples MNIST are also illustrated in Figure in Appendix
CONCLUSIONS
Discussion. We introduced an algorithm to compute (gradients) the MMD with Riesz kernel efficiently via slicing and sorting reducing the dependence of the computational complexity on the number of particles from O M + N²) to O((N + .) log(N + M)). For the implementations, approximated the gradient of sliced MMD by finite number of slices and proved that the corresponding approximation error depends by a square root on the dimension. We applied our algorithm for computing MMD flows and approximated them by neural networks. Here, a sequential learning approach ensures computational efficiency. We included numerical examples for image generation on MNIST, FashionMNIST and CIFARIO.
Limitations. One of the disadvantages of interacting particle methods is that batching is not easily possible: The particle flow for one of training points does not give a helpful approximation for another set of training points. This is due to the interaction energy and a general problem of particle flows. Furthermore, taking the projections involves multiplication of every data point with "full" projection and therefore scales with the dimension d. Taking "local" projections like in (Du et al.
2023
Nguyen& & Ho. 2022) can much more efficient.
Outlook. Our paper is the first work which utilizes sliced MMD flows for generative modelling. Consequently the approach can be extended in several directions. Other kernels are considered in the context of slicing in the follow-up paper (Hertrich] 2024). From a theoretical viewpoint, the derivative formulas from Theorem can be extended to the non-discrete case by the use of quantile functions, (Bonaschi al., 2015) 1 Hertrich et al.) 2023a) for some first approaches into this direction. Towards applications, we could extend the framework to posterior sampling in Bayesian inverse problems. In this context, the fast computation of MMD gradients can be also of interest for applications which are not based on gradient flows, see e.g., Ardizzone et al. (2019). Finally, the
consideration of sliced probability metrics is closely related to the Radon transform and is therefore
of interest also for non-Euclidean domains like the sphere, see e.g., (Bonet et al. 2022a Quellmalz
al. 2023).
vvaluss taken from Bonet al. 2022h

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
Many thanks to J. Chemseddine for providing parts of the proof of Theorem 2, and to R. Beinert and G. Steidl for fruitful discussions, We thank Mitchell Krock for finding typo. J.H. acknowledges funding by the German Research Foundation (DFG) within the project STE 571/16-1 and by the EPSRC programme grant "The Mathematics of Deep Learning" with reference EP/V026259/1, C.W. by the DFG within the SFB "Tomography Across the Scales" (STE 571/19-1, project number: 4953653111 1), F.A. by the DFG under Germany's Excellence Strategy - The Berlin Mathematics Research Center MATH+ (project AA 5-6), and P.H. from the DFG within the project SPP 2298 "Theoretical Foundations of Deep Learning".
REFERENCES
Fabian Altckrüger, Johannes Hertrich, and Gabricle Steidl. Neural Wasserstein gradient flows for maximum mean discrepancies with Riesz kernels. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 664-690. PMLR, 2023.
David Alvarez-Melis, Yair Schiff, and Youssef Mrouch. Optimizing functionals on the space of probabilities with input convex neural networks. Transactions on Machine Learning Research, 2022.
Abdul Fatir Ansari, Ming Liang Ang. and Harold Soh. Refining deep generative models via discrimi- nator gradient flow. In International Conference on Learning Representations, 2021.
Michael Arbel, Anna Korba, Adil Salim, and Arthur Gretton. Maximum mean discrepancy gradient flow. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d'Alché-Buu Buc, E. Fox, and R. Gameet (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Michael Arbel, Alex Matthews, and Arnaud Doucet. Annealed flow transport Monte Carlo. In International Conference on Machine Learning. PMLR, 2021.
Lynton Ardizzone, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. In International Conference on Learning Representations, 2019.
Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, 2017.
Kendall Atkinson and Weimin Han. Spherical harmonics and approximations on the unit sphere: an introduction, volume 2044. Springer Science & Business Media, 2012.
Heli Ben-Hamul Samuel Cohen, Joey Bose, Brandon Amos, Maximillian Nickel, Aditya Grover, Ricky T. Q. Chen, and Yaron Lipman. Matching normalizing flows and probability paths on manifolds. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1749-1763. PMLR, 2022.
Mikotaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.
Giovanni Bonaschi, José A Carrillo, Marco Di Francesco, and Mark A Peletier. Equivalence of gradient flows and entropy solutions for singular nonlocal interaction equations in ID. ESAIM: Control, Optimisation and Calculus of Variations, 21(2):414 14 441, 2015.
Clément Bonet, Paul Berg, Nicolas Courty, François Septier, Lucas Drumetz, and Minh-Tan Pham. Spherical sliced Wasserstsssrsttein arXiv preprint arXiv:2206.088 08788, 2022a.
Clément Bonet, Nicolas Courty, François Septier, and Lucas Drumetz. Efficient gradient flows in Hiced-Wassersse space. Transactions on Machine Learning Research, 2022b.
Nicolas Bonnotte. Unidimensional and evolution methods for optimal transportation. PhD Thesis, Univ. Paris-Sud, 2013.
10

---

Published as a conference paper at ICLR 2024
Charlotte Bunne, Laetitia Papaxamthos, Andreas Krause, and Marco Cuturi. Proximal optimal transport modeling of population dynamics. In Gustau Camps- Valls, Francisco J. R. Ruiz, and Isabel Valera (eds.). Proceedings of The 25th International Conference Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 6511-6528, PMLR, 2022.
José Antonio Carrillo, Marco Di Francesco, Antonio Esposito, Simone Fagioli, and Markus Schmidtchen. Measure solutions to a system of continuity equations driven by newtonian nonlocal interactions. Discrete and Continuous Dynamical Systems, 40(2): 1 191-1231, 2020.
Djalil Chafai, Adrien Hardy, and Mylène Maïda. Concentration for Coulomb gases and Coulomb transport incqualities. Journal of Functional Analysis, 275(6):1447-1483, 2018
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Conference on Learning Representations, 2019.
Biwei Dai and Uros Seljak. Sliced iterative normalizing flows. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2352-2364. PMLR, 2021.
Lauro Langosco di Langosco. Vincent Fortuin, and Heiko Strathmann. Neural variational gradient descent. In Fourth Symposium on Advances in Approximate Bayesian Inference, 2022.
Hanze Dong, Xi Wang, LIN Yong, and Tong Zhang. Particle-based variational inference with preconditioned functional gradient flow. In The Eleventh International Conference on Learning Representations, 2023.
Chao Du, Tianbo Li, Tianyu Pang, Shuicheng Yan, and Min Lin. Nonparametric generative modeling with conditional sliced-We Wasserstein flows. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 3565-8584 PMLR, 2023.
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural net- works via maximum mean discrepancy optimization. In Proceedings of the Thirty-Firs. Conference on Uncertainty in Artificial Intelligence, pp. 258-267, 2015.
Jiaojiao Fan, Qinsheng Zhang, Amirhossein Taghvaei, and Yongxin Chen. Variational Wasserstein gradient flow. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 6185-6215. PMLR, 2022.
Yuan Gao, Yuling Jiao, Yang Wang. Yao Wang, Can Yang, and Shunkang Zhang. Deep generative learning via variational gradient flow. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2093-210000 PMLR, 2019.
Nicolas Garcia Trillos, Bamdad Hosseini, and Daniel Sanz-Alonso. From optimization sampling through gradient flows. arXiv preprint arXiv:2302. / 1449, 2023.
Pierre Glaser, Michael Arbel, and Arthur Gretton. KALE Flow: A relaxed KL gradient flow for probabilities with disjoint support. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. Curran Associates, Inc., 2021.
8018-8031,
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Manuel Gräf, Daniel Potts, and Gabriele Steidl. Quadrature errors, discrepancies, and their relations to halftoning on the torus and the sphere. SIAM Journal on Scientific Computing. 34(5):A2760- A2791, 2012.
11

---

Published as a conference paper at ICLR 2024
Will Grathwohl, Kuan-Chiet Wang, Jörn- Jacobsen, David Kristjanson Duvenaud, and Richard S. Zemel. Learning the stein discrepancy for training and evaluating energy-based models without sampling. In International Conference Machine Learning, 2020.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kemnel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Paul Lyonel Hagemann, Johannes Hertrich, and Gabriele Steidl. Generalized normalizing flows via Markov chains. Cambridge University Press, 2023.
Alvin Heng, Abdul Fatir Ansari, and Harold Soh. Deep generative Wasserstein gradient flows. Preprint OpenReview: https://peeeerieessttttooodddddd
2023.
Johannes Hertrich. Fast kemnel summation in high dimensions via slicing and Fourier transforms. arXiv preprint arXiv:2441.00055 2024.
Johannes Hertrich, Robert Beinert, Manuel Gräf, and Gabriele Steidl. Wasserstein gradient flows of the discrepancy with distance kernel on the line. In Scale Space and Variational Methods in Computer Vision, pp. 431-4 Springer, 2023a.
Johannes Hertrich, Manuel Gräf, Robert Beinert, and Gabriele Steidl. Wasserstein steepest descent flows of discrepancies with Riesz kernels. arXiv preprint arXiv:2211.01 2023b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter, GANs trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates, Inc., 2020.
Chin-Wei Huang, Jae Hyun Lim, and Aaron Courville. A variational perspective on diffusion-based generative models and score matching. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the Fokker-Plann Planck equation. SIAM Journal on Mathematical Analysis, 29(1):1-17, 1998.
David Kershaw. Some extensions of W. Gautschi's inequalities for the Gamma function. Mathematics of Computation, 11(164):607-61,, 1983.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference Learning Representations, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Szymon Knop, Przemyslaw Spurek, Jacek Tabor, Igor Podolak, Marcin Mazur, and Stanistaw Jastrzębski. Cramer-wold auto-encoder. Journal of Machine Learning Research, 21(164) 2020.
Jonas Moritz Kohler and Aurelien Lucchi. Sub-sampled cubic regularization for non-convex opti- mization. In International Conference on Machine Learning, pp. 1895-1904. PMLR, 2017.
Soheil Kolouri, Kimia Nadjahi, Shahin Shahrampour, and Umut Şimşekli. Generalized sliced probability metries. IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4513-4517, 2022.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University Toronto, ON, Canada, 2009.
Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(11):2278-2324 1998.
12

---

Published as a conference paper at ICLR 2024
Alix Lhéritier and Nicolas Bondoux. A Cramer distance perspective quantile regression based distributional reinforcement learning. arXiv preprint arXiv:21 10.00535, 2021.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang. and Barnabas Poczos. MMD GAN: Towards deeper understanding of moment matching network. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International conference on machine learning, pp. 1718-1727. PMLR, 2015.
Qiang Liu. Stein variational gradient descent as gradient flow. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose Bayesian inference algorithm. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of the ICCV '15, pp. 3730-3738, 2015.
Antoine Liutkus, Umut Simsekli, Szymon Majewski, Alain Durmus, and Fabian-Robers Stöter. Sliced-Wasserste Wasserstein flows: Nonparametric generative modeling via optimal transport and diffusions. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.
4104-4113, PMLR, 2019.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs created equal? A large-scale study. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Thibault Modeste and Clément Dombry. Characterization of translation invariant MMD on Rd and connections with Wasserstein distances. HAL Preprint, hal-03855093, 2023.
Petr Mokrov, Alexander Korotin, Lingxiao Li, Aude Genevay, Justin M Solomon, and Evgeny Burnaev. Large-scale Wasserstein gradient flows. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 15243-15256, 2021.
Youssef Mroueh, Chun-Liang Li, Tom Sercu, Anant Raj. and Yu Cheng. Sobolev GAN. In International Conference Learning Representations, 2018.
Sebastian Neumayer and Gabriele Steidl. From optimal transport discrepancy. Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging: Mathematical Imaging and Vision, pp. 1-36, 2021.
Khai Nguyen and Nhat Ho. Revisiting sliced Wasserstein on images: From vectorization to convolu- tion. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Khai Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Distributional sliced- Wasserstein and applications to generative modeling. In International Conference Learning Representations, 2021.
Khai Nguyen, Tongzheng Ren, Huy Nguyen, Litu Rout, Tan Minh Nguyen, and Nhat Ho. Hierarchical sliced Wasserstein distance. In The Eleventh International Conference on Learning Representations, 2023.
Erich Novak and Henryk Wozniakowski. Tractability Multivariate Problems. Volume II, volume 12 of EMS Tracts in Mathematics. EMS Publishing House, Zürich, 2010.
Gabriel Peyré. Entropic approximation of Wasserstein gradient flows. SIAM Journal on Imaging Sciences, (44)2333-2351) 2015.
13

---

Published as a conference paper at ICLR 2024
Gabriel Peyré and Marco Cuturi. Computational optimal transport. arXiv preprint arXiv:1803.0056; 2020.
Gerlind Plonka, Daniel Potts, Gabriele Steidl, and Manfred Tasche. Numerical Fourier Analysis. Springer, 2018.
Michael Quellmalz, Robert Beinert, and Gabricle Steidl. Sliced optimal transport on the sphere. arXiv preprint arXiv:2304.09092 2023.
Julien Rabin, Gabriel Peyré. Julie Delon, and Mare Bernot. Wasserstein barycenter and its application to texture mixing. In Scale Space and Variational Methods in Computer Vision, pp. 435 446.
Springer, 2012.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention, pp-
234-241 234-241.65 Springer, 2015.
Dino Sejdinovic, Bharath Sriperumbudur, Arthur Gretton, and Kenji Fukumizu. Equivalence of distance- based and RRKS-bbase statisties in hypothesis testing. The Annals of Statistics, 41(5): 2263 2291, 2013.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12438-12448. Curran Associates, Inc., 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Bharath K Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, and Gert RG Lanckriet. Hilbert space embeddings and metries on probability measures. The Journal of Machine Learning Research, 11:1517-1561, 2010.
Bharath K Sriperumbudur, Kenji Fukumizu, and Gert RG Lanckriet. Universality, characteristic kernels and RKHS embedding of measures. Journal of Machine Learning Research, 12(7), 2011.
Ingo Steinwart and Andreas Christmann. Support vector machines. Springer Science & Business Media, 2008.
Gabor Székely. E-statistics: The energy of statistical samples. Techical Report, Bowling Green University, 2002.
Gábor J. Székely and Maria L. Rizzo. Energy statistics: A class of statistics based distances. Journal of Statistical Planning and Inference, 14999):124911272, 2013.
Tanja Teuber, Gabriele Steidl, Pascal Gwosdek, Christian Schmaltz, and Joachim Weickert. Dithering by differences of convex functions. SIAM Journal Imaging Sciences, 4(1):79-108, 2011.
Titouan Vayer and Rémi Gribonval. Controlling wasserstein distances by kernel norms with ap- plication to compressive statistical learning. Journal of Machine Learning Research, 24:149-1, 2023.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning, pp. 681-688, 2011.
Hao Wu, Jonas Köhler, and Frank Noe. Stochastic normalizing flows. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balean, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5933-5944. Curran Associates, Inc., 2020.
14

---

Published as a conference paper at ICLR 2024
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNISTA novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
Zhisheng Xiao, Qing Yan, and Yali Amit. Generative latent flow. arXiv preprint 1rX.v:9905.10485, 2019.
15

---

Published as a conference paper at ICLR 2024
A PROOF OF THEOREM
Let Uga-- be the uniform distribution on gd-1 and let k(r, y) = -|2 y!" for I, E R # and 2). Moreover, denote by (1, 0) gd-1 the first unit vector. Then, we have for r, E I= that
K K(x,y) =
JS-- gg 1(5,a)) (5. (5) -|| yll"
(
E I y
)
cldga (5)
J3N-1
|| - ull
= --11> - wll" J ((E,e) (E) -1|x yl" f 34-1
Gil'dlls: (E)
It remains to compute the constant Cd,r which is straightforward for d = 1. For d the map (t, - (t. n f2)
defined on 1-1,1) x g is parametrization of gd-1 The surface measure on gd-1 is then given by
doga-1 = (1
d
see Atkinson & Han 2012 Eq. 1.16). Furthermore, the uniform surface measure Uza-1 reads as
(E) = (1 +2) doga-2
$d-1
where sp-1 is the volume of gd-1, Hence
{ 151 15 d'ddda-- 8d-1
[t] -82)² 222 dtdoga-2(n)
car =
(10)
-g4-1
$d-22
{' (1(1 - 12) di
sd-2
+ d
B
8d-1
8d-1
where B(2 22) is the beta function and we used the integral identity
B 22) 22) 2 S/"² 2 2
Finally, noting that 8a-1
2/4/2 F(g) and 22)
[(a))[(aa) T(a
can be computed as
r
r( (()))) r(
(
Taking the inverse gives the claim.
Remark (Extension to Ps (Rd)). We can extend Theorem to Py (TR.). this end, we first show,
how deduce from MModeste & Dombry 2023 Prop Z.14) that the MMD DR with respect to the extended Riesz kernel K (x, 3)
-11x vll' [ |!y!!" defines metric on Pg (R.). Second,
we will see that Theorem can be extended to Pg (Rd) as well.
(i) By (Steinwart Christmann 2008
Thm 4.26) the MMD
D2 (H,2) /
((xxv)d(u (aad(u-v)(all v)(z)d(p v)(v)
is finte on MR (u M(RR): I (K < oo), where M(R is the space of all signed measures Rdd. Since K(xx)) x) 2||=||", we have that MM {pr M(Rd) I || < oo} such that Mg P(R)) = P. (Rd). Now: inserting K (Modeste &
Dombry 2023 Prop 2. 1.4 states that D is metric on Mg n P(Rd) P. (Rd), where
the assumptions of the proposition are checked in (Modeste Dombry 2023 Ex Ex 3)
and K is named Kn with H 3
16

---

Published as a conference paper at ICLR 2024
(ii) Let be rescaled and k(x, v) -|x yl" + |=|" lul" be the one-dimensional version of the extended Riesz kernel K. With the same calculations as in the proof of Theorem we can show that
K(æ,1) { k((S, (d)
Iga-1
Thus, we also have that SD₃ D on (TR.).
B PROOF OF THEOREM
In the first part of the proof, will use properties of reproducing kernel Hilhert spaces (RKHS), (Steinwart & Christmann 2008)) for an overview on RKHS. For the second part, we will need two lemmata. The first one gives definition of the MMD in terms of the Fourier transform (characteristic function) of the involved measures, where ju(E) (e-xx5, (4) fg. e "((=") Its proof can be found, e.g., in (Székely & Rizzo 2013] Proposition 2). Note that the constant on the right hand-side
differs from (Székely & Rizzo 2013] Proposition 2), since transform and the constant in the MMD.
different notion of the Fourier
Lemma 8. Let μ E (I) and K ((x,)) y)
=-11 yll. Then its holds
r( (i(E) (100-110 2())2
DK((,D)
3d 1|E||=+d de.
2
For the next lemma, recall that the cumulative density functions (cdf) of P(R) is the function Fa R [0, 1] defined by
Fx(x) p((-oo,x)) xi- x(-oc.==) X(-xx= xa(x) = 11, if A, 0, otherwise.
We will need that the Cramer distance between probability measures μt with edfs F2 and F, defined by
(p(u,v)= (1
if it exists. The Cramer distance does not exist for arbitrary probability measures. However, for H, (R) is well-known that
e (14 Wi (H,
(11)
and we have indeed Fμ F, E LR)... The following relation can be found in the literature, see, e.g., ...éeyykely (2002). However, we prefer to add proof which clearly shows which assumptions are necessary.
Lemma 9. Let ML E PI( and k(x.v) --x--y,
y/. Then it holds
(2(H.v) DD(H,DD.
Proof. By Lemma 8 we know that
De(,,)) v) = -- |a y) ()())d((( v)(r) 1
(A(E) DEE)
dg.
(12)
2x
g2
For n, v E (RR), we have F
-F, F, E [R) and can apply Parseval's equality
((w,v)= / -
1 (/((E) (())))
(13)
Now have for the distributional derivative of F,, that DF, p. which can be seen as follows: using Fubini's theorem, we have for any Schwartz function & E ((R) that
(DF,, = --FFn, /
J x(- (') (()) dr ( X(- (-)0,))))) (v) (x))
=- R
d((x)drd((y) = 0()) du(v) = (1,0). Ø).
17

---

Published as a conference paper at ICLR 2024
Then obtain by the differentiation property of the Fourier transform (Plonka al. 2018 that iM(E) = (F) = (E)
Finally,
13 becomes
1
IM(E) ((E))
(2, v) = 2x JR
de.
8²
which yields the assertion by 12
Now we can prove Theorem
Proof. 1. To prove of the first inequality, we use the reproducing property
(J, K (æ, 22n f(x)
of the kernel in the associated RKHS Hg. For any E H with [s S and any E I(, v), we use the estimation
[ f())d(u 2)))))
= /
Ra JRa f(x) f(y)dT(Z,)) - / J 1f(x)) f())ddt()
[ R JR"
/ 11, - K(y, )) (dT(x,))
2 Ja Jud
[K(x,,)) K (v, 11 (2, y):
which is called "coupling bound" in (Sriperumbudrr et al. 2010 Prop. 20). Then, since [[[(r; :) K (v, >ll (x, + K (y, v) - 2K 2K(2,)) = 2|1 yll and using Jensen's 's inequality for the concave function v. we obtain
Ufn (())d(u (x) /2
( / |1 vll (x,
JRs J || /2/e///lle- v)ldT(,,)))
2 (2/L1
2d Jud
By the dual definition of the discrepancy 2D 2Dx(u,, SuP11s $1 Sge fd(p= v), see Novak & Wozniakowski 22100), and taking the supremum over all such f and the infimum over all x E II v), we finally arrive at
2D (H, S Wi (H,
2. The second inequality can be seen as follows: by Bonnotte 2013 Lemma 5.1.4), there exists constant ca > 0 such that
Wi(H, v) S cyR SWi v) गे. CaR (Eewige-: M (Psg u. Pe. w)] a Further, we obtain by (II,, the Bauchy-Schwarr inequality and Lemma Ithat
Eg-idee11 [W1 (PEn u4, Ve = EE-ite-1 [4 (PEN M, PEn w)]
S Ee-ilge (2R) (P's M4 Ps 1)
[Dx Peall PS# 11] S (2R) (Be-llza-1
(2R) 3 E
[D] (Peg P' 1)))
and finally by Theorem that
[(2) V d+11
Be-ehar- [Wi (PEal, u,, PEn v < (2R) (2R)
Dx(,,v).
In summary, this results in
2n())
W (i, s.aa
R Dr (1,, ut
V
Under some additional assumptions, similar bounds have been considered in Chafai et al. (0018) for the Coloumb kernel K(x, y) | 2
18

---

Published as a conference paper at ICLR 2024
C PROOF OF THEOREM 3
Interaction Energy: This part of the proof is similar to (Teuber et al. 2011 Sec. 3). Using that is permutation and by reordering the terms in the double sum, we can rewrite the interaction energy by
B(x) 2N
D3
xz
[o(() - Io(ii
2N
{=1
Σ
Σ To6)
N
N
N
(2i 1)
N -20 (i)
No
To()
2o(1)
N2
To(
N2
Zi.
3=
i=
Since the r,are pairwise disjoint, the sorting permutation is constant in a neighborhood of æ. Hence, E is differentiable with derivative
N +
2x
(1)
22 ((x)
N2
Potential Energy: For any x # y E R it holds
11
if > y,
V x - y/ = x(x,8)) v), where x(x,
-1,
if x <
Thus, we have that
M
V., V(rlg) = MN Σ x(
j=1
(#(je E {1...., M] :3 < - #(j {1, M] yı > x,})
MN
Using that #(i E (1...., M) y/ > x #(i E (1, M) y < x;} the above expression is equal to
1 MN (2 E
8, <2. -M) 2 {l, M} yj < xz} M
MN
D PROOF OF THEOREM 4
In this section, we derive error bounds for the stochastic estimators for the gradient of MMD as defined in (6) for the Riesz kernel with 1. To this end, we employ concentration inequalities (Vershynin] 2018). which were generalized to vector-valued random variables in Kohler & Lucchi (2017). We will need the following Lemma which is the Bernstein inequality and stated (Kohler & Lucchi 2017 Lemma 18).
Lemma 10 (Bernstein Inequality). Let Xi,..., Xp be independent random vectors with mean < o². Then holds for 0 < <t <
zero, [e S μ and [[]] X,
o² that
(- [Ppe
Pt
+ 2 80
P
X > s exp
P
Now use Lemmal 10 to show convergence of the finite approximation to the exact gradient.
Theorem 11 (Concentration Inequality). Let T.,,.. v1...,, B/M RRd. Then, it holds
Pt²
+ 1 2)
P (PPF(()))) - VF > S exp (-32
32 (ca + 112
19

---

Published as a conference paper at ICLR 2024
Proof. Let Ep Uga-1 be the independent random variables from the definition of Vp in (
We set
X = ca Vi (x) U2), (5t. yas)
and define the dNV-dimensional random vector X, (Xi,,," Xi,v). Then, have by 4 that
B| VFa(#ly) 2V y), ... VaN Fl(
By Theorem we know that || 2 I
2ea N B
and by 3 it holds
2
X 1l2 V Fa(a - N
Let X; X; -X,,). Then it holds
2ea + 2 N
|
Xxulla
2ca +
1=1
and thus E [X| [12] 4(ca + 1)2. Since we have [[[x;]
0 for all
L,... P. we can apply
Lemma 10 and obtain
p Ex--
-
P
X
>
<
exp
P
+ 1
32 (ca + 1)2
==1
Since have by definition that
P
Σ
X; V
PFlll))))
P t11
this yields the assertion.
Finally we can draw a corollary which immediately shows Theorem Corollary 12 (Error Bound for Stochastic Gradients). For F1, ..., NN 91. UM E RRd, holds
Ell ( cxp(1/4)v332 32m
2v2 P
Proof. Denote by the random variable
x [)
VFa(=)))1
Then, we have by Theorem IT that
Pt
1
P[X t S exp
32 (ca + 1)2
Thus, we obtain
E / P(X > tjdt S exp(1/4)
/ exp 32 (ca 1)2
P 1
exp(1/4)v33 + +1)
dt -
2vP
where the last step follows from the identity f exp( - -t) y Now we proceed to bound the constant ca in the dimensions. By Theorem I we have that ea
2
Now the claim follows from the bound
2 1 proven by Kershaw
T
1983
20

---

Published as a conference paper at ICLR 2024
E COMPARISON OF DIFFERENT KERNELS IN MMD
Next we compare the MMD flows of the Riesz kernels with those of the positive definite kernels
KG( y) = exp ( || y/1?
(Gaussian),
20²
1
()
V1 y/1² (Inverse Multiquadric),
a vl
KL( = exp
(Laplacian).
The target distribution is defined as the uniform distribution three non-overlapping circles and the initial particles are drawn from a Gaussian distribution with standard deviation 0.01, compare Glaser et al. 2021p. We recognize in Figures 4 and that in contrast to the Riesz kernel, the other MMD flows
heavily depend on the parameters a and e (stability against parameter choice);
cannot perfectly recover the uniform distribution; zoom into the middle right circles see that small blue parts are not covered (approximation of target distribution).
Moreover, in contrast to the other kernels, the Riesz kernel with r = 1 positively homogencous such that the MMD flow is equivariant against scalings of initial and target measure.
Finally, it is interesting that the Riesz kernel is related to the Brownian motion by the following remark, see also Modeste & Dombry (2023) for the more general fractional Brownian motion.
Remark 13. In the one- one-dimensional case, the extended Riesz kernel with 1 reads as
K(x, v) = -- +
|z| lyl = 2 min(æ, y),
which is the twice the covariance kernel of the Brownian motion. More precisely, let (We)esu be Brownian motion and s,t > O. Then, it holds
Cov(W,, W,) min(s, t) = 2 K(
F TRAINING ALGORITHM OF THE GENERATIVE SLICED MMD FLOW
In Algorithm 3 we state the detailed training algorithm of our proposed method.
G ABLATION STUDY
We consider the FID for different number of networks and different number of projections. We run the same experiment in Section 5 on MNIST. Here we choose different number of projections P between 10 and 1000. In Figure 6 we illustrate the progress of the FID value for an increasing number of networks and different number of projections. Obviously, the gradient of the MMD functional is not well-approximated by just using P 10 or P = 100 projections and thus the MMD flow does converges. Once the gradient of the functional is well-approximated. a higher number of projections leads only to a small improvement, see the difference between P 500 and P 1000.
H IMPLEMENTATION DETAILS
The code is available online at https: httpes///gtthu...co////ptrrrt//t
MMD flows We use UNets (+){ P with 3409633 trainable parameters for MNIST and FashionMNIST and 2064035 trainable paramters for CIFARIO. The networks are trained using Adam Kingmma & Ba
modified from
di fiffusi on_tf/modeeeeeoooo s unet.py
21

---

Published as a conference paper at ICLR 2024
Gaussian kernel
0.05
O
0²
0.1
0.5
T=2
T=10
T=100
Inverse multiquadric kernel
0.01
C c=
0.05
0.5
T=0
T=2
T=10
T=100
Figure 4: Comparison of the MMD flow with Gaussian kernel (top) and inverse multiquadric kernel (bottom) for different hyperparameters.
2015) with learning rate of 0.001. All flows are simulated with a step size 7 1. We stop the training of our generative sliced MMD flow when the FID between the generated samples and some validation samples does not decrease twice. Then we take the network with the best FID value to the validation The validation samples are the last 10000 training samples from the corresponding dataset which were not used for training the generative sliced MMD flow. The training of the generative MMD flow takes between 1.5 and 3 days on a NVIDIA GeForce RTX 2060 Super GPU, depending on the current GPU load by other processes. To avoid overfitting, we choose a relatively small number of optimization steps within the training of the networks 4,, which corresponds to an early-stopping technique.
We draw the first initial samples uniformly from [0, 1J Then simulate the momentum MMD flow using P = 1000
MNIST.
20000 target samples from the MNIST training set and 20000
projections for 32 steps and train the network for 2000 optimizer steps with batch size of 100.
After each training of the network,
increase the number of flow steps by min(2 2048) up to
maximal number of 30000 steps, where 7 the iteration of the training procedure, see Algorithm B We choose the momentum parameter
0.7 and stop the whole training after L = 55 networks.
FashionMNIST. Here we draw the first M = 20000 target samples from the FashionMNIST training set and N 20000 initial samples uniformly from [0, 1]d. Then simulate the momentum MMD
22

---

Published as a conference paper at ICLR 2024
Laplace kernel
0.01
o²
0² = 0.02
O
0² 0.1
T=0
T=2
T=10
T=100
Riesz kernel
0=5
1.0
1.5
T=0
T=2
T=10
T=100
Figure 5: Comparison of the MMD flow with Laplacian kernel (top) and Riesz kernel (bottom) for different hyperparameters.
500
P=10
=1000
=P -500
400
-P-0000
350
250
200
100
50
0
20
30
50
60
Network 4g
Figure 6: Illustration of the FID value of the Sliced MMD Flow on MNIST for different number of projections.
flow using P 1000 projections for 32 steps and train the network for 2000 optimizer steps with
23

---

Published as a conference paper at ICLR 2024
Algorithm 3 Training of generative MMD flows (0)
Input: Independent initial samples s ,..., "N from Ho, momentum parameters my E [0, 1) for
1 1= 1, 1,.... L.
Initialize (u....., UN) = O. for = 1, L do
Set (3
(0) (41 (1-1) ,..., NN 11)
Simulate Ti steps of the (momentum) MMD flow:
for 1, = 1..,, Ti do
Update u by
(M,,.,, UN) t- VFa(t 3... s (1,,.... yMd) + mi(u1,.... UN)
Updatee the flow samples:
( (zf ..... NN -N (U)....,uN))
end for
Train i such that (T)
Rs x.
(1:() (0) by minimizing the loss
N
((0))
|| [ (0)) - (5F!0) - -((i))11 M1?
N
1=1
Set (x) ...... x. = (z) (1-1) end for
..... IN (1-1))) (02 ) (1 (z )))
a batch size of 100. After each training of the network, we increase the number of flow steps by min(25+1, 2048) up maximal number of 50000 steps, where I is the iteration of the training procedure. The momentum parameter m is set to 0.8. We stop the whole training after L = 67 networks.
CIFAR. We draw the first M = 30000 target samples from the CIFARIO training set. Here we consider a pyramidal schedule, where the key idea is to run the particle flow on different resolutions, from low to high sequentially. First, we downsample the target samples by a factor 8 and draw N = 30000 initial samples uniformly from [0, 1] A Then we simulate the momentum MMD flow in dimension d 48 using P = 500 projections for 32 steps and train the network for 5000 optimizer steps with a batch size of 100. After each training of the network, we increase the number of flow steps by min 1024) up to a maximal number of 30000 steps, where / is the iteration of the training procedure. The momentum parameter m is increased after each network training by 0.01 up to 0.8, beginning with m = 0 in the first flow step. We increase the resolution of the flow after 600000 flow steps by factor 2 and add Gaussian noise on the particles in order increase the intrinsic dimension of the images, such that the second resolution is of dimension d = 192. Following here the same procedure as before for the second resolution and the third resolution of d = 768,, change the projections in the final resolution of d = 3072. Instead of using projections uniformly projections as in (Du et al. 2023 Nguyen & Ho 2022). The idea is to extract patches of the images x² at a random location in each step k and instead consider the particle flow in the patch dimension. In order to apply these locally-connected projections at different resolutions, we also upsample the projections to different scales. Here we choose a patch size of 7 x and consider the resolutions 7, 14, 21, 28. We stop the whole training after L 86 networks.
sampled from gdd11 we consider locally-comeecte
Note that herewith we introduced an inductive bias, since we do not uniformly sample from [0, 1]4, but empiricially this leads to significant acceleration of the flow. A more comprchensive discussion can be found in (Du et al. 2023 Nguyen & Ho 2022)
CelebA. We draw the first M 20000 target samples from the CelebA training set. Again, we consider a pyramidal schedule as for CIFAR10 10, but we increased the number of flow steps by min(25+1, 8192) up to a maximal number of 100000 steps. We increase the resolutions of the flow after 700000, 900000 and 700000 flow steps by a factor and add Gaussian noise on the particles in order increase the intrinsic dimension of the images. We also use locally-connected projections
24

---

Published as a conference paper at ICLR 2024
Table 2: FID scores of generated samples for training set and test
MNIST
FashionMNIST
CIFARIO
CelebA
training test set
277
10.6
53.0
32.7
3.1
11.3
54.8
32.1
with patch size of 7 x 7 and consider the resolutions 7,, 28 for resolution 32 x 32 and 7,14,20 21, 28, 35, 42, 49 for resolution x 64. We stop the whole training after 71 networks.
ADDITIONAL EXAMPLES
In Figure we show more generated samples our method for MNIST, FashionMNIST and CIFARIO.
In Figure g we compare generated MNIST samples with the closest samples from the training set. We observe that they are significantly different. Hence, our method generates really new samples and is not just reproducing the samples from the training set. In contrast, in Figure 9 we compare the particle flow samples with the closest samples from the training set. Obviously, the samples of the particle flow approximate exactly the training samples. This highlights the important role of the networks: We can interpolate between the training points in order to generalize the dataset.
25

---

Published as a conference paper at ICLR 2024
2
0
4
9
9
/
7
8
4
4
7
9
5
4
8
2
4
8
8
3 6 8
8
5
0
0
3
6
5
/
9
4
3
8
9
2
3
R
3
/
7
3
1
3
9
0
7
7
7
7
4
/
4
5
Y
8
8
/
6
2
5
0
3
6
8
5
2
5
7
9
O
/
9 3
8
2
/
4
0
9
2
9
7
) 8 3
S
7
9
5
9
3
9
8
/
8
6
8
9
X
3 1 4 7
0
6
8
O
4
9
3
6
O
9
8
4
3 8 / / 3
8
O
4
7
9
3 6
3
3
3
0
4
8
7
6 8
7
R
4
4
8
5 3
7
O
5
0
1
8
7
2
1
4
2
3
9
7 5
4
7
6
4
/
7
/
9 2
/
3
/ 9
1
2
6
3
6
4
6
J
O
6
9
6
0
6
3
3
5
2
5
5
2
6
6
2
3
3 4 8
3
9
7
8
6
8
2
0
3
6
8
6
2
2
5
9
O
8
/
7
7
0
7
7
8
4
8
8
6
9
3
4
8 8
O
Y
7
O
O
9
4
2
5
3
8
(a) MNIST
(b) FashionMNIST
CIFARIO
(d) CelebA
Figure 7: Additional generated samples (from top bottom) of MNIST, FashionMNIST, CIFARI0
and CelebA.
26

---

Published as a conference paper at ICLR 2024
2
0
8
7
7
4
8
5
5
6
7
6
8
6
4
3
6
2
5
2
/
0
0
7
7
4
8
5
5
6
7
6
8
/
4
3
6
2
5
/
(a) MNIST
(b) FashionMNIST
CIFAR1O
(d) CelebA
Figure 8: Generated samples (top), Lg closest samples from the training set (middle) and the pixelwise distance between them (bottom).
8
9
5
7
0
5
1
O
4
6
6
4
0
2
5
6
7
3
7
8
9
6
7
0
5
1
0
4
6
6
4
0
2
5
7
3
7
Figure 9: Samples of the particle flow (top), closest samples from the training set (middle) and the pixelwise distance between them (bottom). The mean PSNR between these flow samples and the corresponding closest training images is 82.29.
27


Reviews:
[{"review_id": "22cnQ0L3OH", "reviewer": "Reviewer_e9Tn", "summary": "The work proposed to use sliced MMD with Riesz kernel  to compute MMD gradients for generative model training. The authors introduced the Riesz kernel with its sliced version in section 2 and show that  sliced version is actually the Riesz kernel. Section 3 showed how to compute gradients of sliced MMD in one-dimensional space by its special property of ordering projected data. Section 4 presented MMD flows. The author demonstrated their methods in section 5 with MNIST, FashionMNIST and CIFAR10 datasets.", "questions": "No question", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to follow and read. \nThe proposed method is simple and computational efficient. \nThe experiment results showed an improvement of FID in MNIST and FashionMNIST data sets.", "weaknesses": "All the theory part is quite simple, specially the important theorem 1, which proved that the Sliced Riesz kernel is an equivalent form of Riesz kernel. I have the same impression for the sorting algorithm in 1-D case and results of error bound for stochastic MMD gradient in theorem 4. \n\nThe experimental part is very limited with few experiments. The methods is shown to work with simple data sets like MNIST and FashionMNIST, when they considered a much more complicated-structure data set like CIFAR10, then its FID is quite bad compared to NCSN and WGAN.  \n\nI do not find both theory and application part strong enough for publication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "atnQ3DZBpX", "reviewer": "Reviewer_pBbs", "summary": "The paper is dedicated to the question: how to estimate a gradient of the MMD distance between two empirical distributions w.r.t. points of the first distribution. The MMD distance is a natural distance between distributions, using which one can solve generative modeling problems. So the latter computational problem is quite important for generative modeling based on kernels. It is shown that the MMD distance defined by the Riesz kernel has a very special structure, namely that the so-called sliced Riesz distance coincides with the MMD distance.\nThis allows one to estimate the gradient very precisely because one can take few 1-dimensional projections of empirical points and calculate an averaged sliced MMD. Based on that, authors design a generative modeling algorithm (described in the Appendix as Algorithm 3). Their algorithm performs well, taking into account Table 1, though which part of their algorithm is mainly responsible for such a promising outcome is a non-trivial issue.", "questions": "A natural question: is the Riesz kernel so special, that the MMD distance induced by it leads to successful generative modeling? Or your algorithm for an accurate approximation of gradient is responsible for success?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Major theoretical claims are correct, and proofs seem convincing, though I have not checked all of them.", "weaknesses": "The paper is dedicated to accelerating the computation of the gradient of the sliced MMD with the Riesz kernel. Experiments are dedicated to a new algorithm for generative modeling (Algorithm 3 described in Appendix). A natural question appears: what is responsible for good results on MNIST/FashionMNIST/CIFAR10? Is it the sequential approach to train MMD flows, or the fact that gradients are estimated better, or the fact that Riesz kernel defines such a special MMD, or maybe specifics of architecture of neural networks Ф_1, ..., Ф_L (modified from some previous work)?\n\nFor me, it is hard to make a judgment of what these experimental results really mean. There are too many ingredients there.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m84lbTKsQG", "reviewer": "Reviewer_hi5p", "summary": "This article focuses on MMD flows with a Riesz kernel that is the distance between points raised to a power $r \\in (0,2)$. The main contributions are, firstly, to demonstrate that this kernel is identical (up to a constant) to its 'sliced' version and, secondly, to use this characterization for the efficient computation of gradients in MMD flows.", "questions": "see above\n\n---- AFTER REBUTTAL ----\n\nThe authors have addressed all my concerns.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I find this article to be well-written and its contributions to be interesting. Efficient MMD computation is indeed an important point, not only for MMD flows. The article addresses an important problem and offers an elegant solution for Riesz kernels. However, there are several points that appear to need correction or, at the very least, further elaboration.", "weaknesses": "- Concerning Theorem 2:\n\nTheorem 2 establishes bounds between MMD and Wasserstein distance of order 1. In my opinion, these results are not very sharp, and there appears to be an important missing reference here. Under the same assumptions of compact support, the article [1, Theorem 1] demonstrates that the Wasserstein distance $W_1$ is bounded by an MMD with the Coulomb kernel $k(x, y) = -|x - y|^{2-d}$ but without the power dependency of $1/(d+1)$. Since the measures are bounded, MMD with the Riesz kernel and the Coulomb kernel are related by a constant (dependent on the dimension) on the support of the measures. Therefore, it seems important to mention this result and discuss its relevance.\n\nIn a more general sense, I'm having trouble grasping the significance of these bounds for the current article. They don't appear to be entirely novel, and more importantly, they are not utilized subsequently, neither theoretically nor practically, nor in the discussion.\n\n- Regarding Theorem 3:\n\nTheorem 3 is the main contribution of this article, and I agree that it is interesting and significant. However, there is a minor point that needs clarification: while the function $E$ is shown to be differentiable everywhere in the proof, for the differentiability of $V$, the authors use an argument that doesn't seem rigorous. Indeed, the function $x \\to |x-y|$ is not differentiable at $x=y$. In practice, this may not be very important because this event is almost surely zero if the samples come from measures with densities. Still, it's important to note this limitation.\n\n- Regarding the dimension dependency:\n\nAn important aspect that is not detailed in the article is the dependence on dimension. Indeed, the sliced Riesz Kernel is not exactly equal to the Riesz kernel; it differs by a constant that depends on the dimension. As the dimension approaches infinity, this constant $c_{d,r}$ behaves as $O(d^{r/2})$, and thus the rescaled kernel tends to zero as $O(d^{-r/2})$. This suggests that estimating the gradient with a finite number of projections becomes increasingly challenging in high dimensions, as also observed in Figure 2 and the bound in $O(\\sqrt{d/P})$.\n\nThis point is not thoroughly discussed in the article. It would be interesting to visualize the relative error in the gradient not only as a function of the number of projections but also with respect to the dimension.\n\n\n- About training a sequence of neural networks (Section 4.2):\n\nI am having difficulty grasping the intuition behind the iterative training of neural networks $\\Phi_1, \\cdots, \\Phi_L$ to approximate the generation scheme. Why are these networks needed? Does the flow not work without them? What are the results without these networks on CIFAR10 or MNIST, for example?\n\nFurthermore, it seems to me that training a network per step is very costly; I doubt the feasibility of this method. Is this standard practice?\n\n- Regarding the \"related work\" section in the introduction:\n\nI believe that this section could be improved. The paragraph is somewhat confusing as it introduces a set of articles without providing clear context for the current work or establishing connections between the cited articles. As a result, I find this part to be not very illuminating for understanding the related work.\n\n\n- Additional remarks:\n\n  - Figures 1 and 2: The tables are not informative and hard to read. Since the information is already contained in the plots, I don't see the purpose of these tables. I suggest removing them and using the available space for more details on the points described earlier. Additionally, Figure 1 lacks a legend for the runtime (in ms?).\n\n  - The Riesz kernel also defines a valid MMD for $r=2$. So why restrict it to $r \\in (0,2)$ with 2 excluded?\n\n\n  - Unless I am mistaken, the result about $D_{K}$ as a metric on $P_{r/2}$ in (Modeste & Dombry, 2023) is not mentioned anywhere in the cited article. Can you provide more details on this fact or provide the correct reference ?\n\n  - I'm quite curious to know if the approach presented in the article can be generalized to the case of $r \\in (0,2]$? Particularly, the fast gradient computation.\n\n  - It seems that the reference (Numayer & Steidl, 20201, Lemma 3.3) is incorrect, or at least Lemma 3.3 doesn't state that the two MMDs coincide.\n\n- Refs: \n\n[1] Djalil Chafaï, A. Hardy, and Mylène Maïda. Concentration for Coulomb gases and Coulomb transport inequalities. Journal of Functional Analysis, 275, 2016.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gG4IVdR1mA", "reviewer": "Reviewer_HuoH", "summary": "The paper proposes a non-parametric generative model, sliced MMD Flows with Riesz Kernels. The paper initially presents the concept of \"Sliced MMD with Riesz Kernels,\" which is essentially a sliced variation of MMD with Riesz Kernels. The paper demonstrates that Sliced MMD with Riesz Kernels qualifies as a metric in the space of probability distributions, having non-negativity, symmetry, triangle inequality, and identity. Additionally, it establishes an equivalence between Sliced MMD with Riesz Kernels and the MMD with sliced Riesz Kernels. Furthermore, the paper elaborates on the methodology for calculating Sliced MMD with Riesz Kernels and its gradient, showcasing that this approach achieves nearly linear complexity in relation to the support count of two discrete distributions. Lastly, the paper compares the proposed frameworks with other generative modeling techniques, such as NCSN, WGAN, MMD GAN, SIG, SWF, and more.", "questions": "* Could any methods be used to improve the experimental results on high-dimensional datasets?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper represents a progression in utilizing MMD flows for generative modeling. Through the introduction of an innovative and clever method for calculating gradients of MMDs using Riesz kernels, the suggested approach opens up the possibility of employing MMD flows in generative modeling tasks that were previously considered impractical for these types of functions.\n* The demonstration of the metric property and calculating MMD gradients linked to Riesz kernels is a great contribution. it brings attention to Riesz kernels, which might have been somewhat neglected within the broader landscape of kernel methods.\n* The connection with Wasserstein distance is interesting. \n* The paper archives the best FID score on MNIST and Fashion MNIST.", "weaknesses": "* The FID score on Cifar10 is relatively high to other generative models.\n* The computation of the FID score is only from 1 run without any standard deviation. \n* It seems that the proposed framework is not scalable in terms of dimension since the result from CIFAR10 is quite blurry and noisy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)