Lipschitz regularized gradient flows and latent generative particles
Authors: Hyemin Gu
Panagiota Birmpa
Yannis Pantazis
Markos Katsoulakis
Luc Rey-Bellet
Venue: Submitted to ICLR 2023 (2023)
Keywords: probability divergences
generative models
Lipschitz regularization
gradient flows
autoencoders
particle algorithms

Abstract: Lipschitz regularized $f$-divergences are constructed by imposing a bound on the Lipschitz constant of the discriminator in the variational representation. These divergences interpolate between the Wasserstein metric and $f$-divergences and provide a flexible family of loss functions for non-absolutely continuous (e.g. empirical) distributions, possibly with heavy tails. We first construct Lipschitz regularized gradient flows on the space of probability measures based on these divergences. Examples of such gradient flows are Lipschitz regularized Fokker-Planck and porous medium partial differential equations (PDEs) for the Kullback-Leibler and $\alpha$-divergences, respectively. The regularization corresponds to imposing a Courant–Friedrichs–Lewy numerical stability condition on the PDEs. For empirical measures, the Lipschitz regularization on gradient flows induces a numerically stable transporter/discriminator particle algorithm, where the generative particles are transported along the gradient of the discriminator. The gradient structure leads to a regularized Fisher information which is the total kinetic energy of the particles and can be used to track the convergence of the algorithm. The Lipschitz regularized discriminator can be implemented via neural network spectral normalization and the particle algorithm generates approximate samples from possibly high-dimensional distributions known only from data. Notably, our particle algorithm can generate synthetic data even in small sample size regimes. A new data processing inequality for the regularized divergence allows us to combine our particle algorithm with representation learning, e.g. autoencoder architectures. The resulting particle algorithm in latent space yields markedly improved generative properties in terms of efficiency and quality of the synthetic samples. From a statistical mechanics perspective the encoding can be interpreted dynamically as learning a better mobility for the generative particles. 

Under review as a conference paper at ICLR 2023
LIPSCHITZ REGULARIZED GRADIENT FLOWS AND LA- TENT GENERATIVE PARTICLES
Anonymous authors Paper under double-blind review
ABSTRACT
Lipschitz regularized f-divergences are constructed by imposing a bound on the Lipschitz constant of the discriminator in the variational representation. These divergences interpolate between the Wasserstein metric and f-divergences and provide flexible family of loss functions for non-absolutely continuous (c.g. em- pirical) distributions, possibly with heavy tails. We first construct Lipschitz regu- larized gradient flows the space of probability measures based these diver- gences. Examples of such gradient flows are Lipschitz regularized Fokker-Plancl and porous medium partial differential equations (PDEs) for the Kullback-Leible and a-divergences, respectively. The regularization corresponds to imposing a Courant- Freddr---eeiiiis--eeee numerical stability condition on the PDEs. For empiri- cal measures, the Lipschitz regularization stable transporter/ddisriimmnaooo particle algorithm, where the generative particles are transported along the gradient of the discriminator. The gradient structure leads to a regularized Fisher information which is the total kinetic energy of the particles and can be used to track the convergence of the algorithm. The Lipschitz regularized discriminator can be implemented via neural network spectral normal- ization and the particle algorithm generates approximate samples from possibly high-dimensional distributions known only from data. Notably, our particle algo- rithm can generate synthetic data even in small sample size regimes. A new data processing inequality for the regularized divergence allows us to combine our par- ticle algorithm with representation learning, e.g. autoencoder architectures. The resulting particle algorithm in latent space yields markedly improved generative properties in terms of efficiency and quality of the synthetic samples. From a statistical mechanics perspective the encoding can be interpreted dynamically as learning a better mobility for the generative particles.
gradient flows induces a numerically
INTRODUCTION
We construct new algorithms that are capable of efficiently transporting arbitrary empirical distri- butions to a target data set. The transportation of the empirical distribution is constructed as a (dis- cretized) gradient flow in probability space for Lipschitz-regularizze f- divergences. Samples are viewed as particles and are transported along the gradient of the discriminator of the divergence to- wards the target data set. We take advantage of representation learning concepts, e.g. autoencoders, and make these algorithms efficient even in high-dimensional sample spaces by defining particle algorithms in latent space. Their accuracy is guaranteed by new data processing inequality. One of our main tools is Lipschitz regularized f-divergences which interpolate between the Wasser- stein metric and f-divergences. Such divergences Dupuis & Mao (2022); Birrell et al. (2022a c), discussed in Section provide a flexible family loss functions for non-absolutely continuous dis- needs to build algorithms handle target distributions which
tributions. In Machine Learning are singular, either by their intrinsic nature such as probability densities concentrated on low dimen- sional structures and/or because Q is usually only known through N samples (the corresponding empirical distribution QN is always singular). Another key ingredient our construction that we build gradient flows where mass is transported along the gradient of the optimal discriminator in the variational formulation the divergences. The time discretization such gradient flows for empirical distributions gives rise so-called transporterddicciiiinaoo particle algorithm which transports initial empirical distribution Py toward the target QN. The Lipschitz regularization

---

Under review as a conference paper at ICLR 2023
provides numerically stable, mesh free, particle algorithms that can act generative models for high-dimensional target distributions. Moreover the gradient structure yields a dissipation functional which corresponds to the kinetic energy of the particles (a Lipschitz regularized version of Fisher information) and which can be used to control the convergence of the algorithm. The third new element in our methods is the use of representation learning to reduce the sample space dimension. We construct latent particle algorithms by building a Lipschitz regularized gradient flow in latent space. The fidelity of the latent space particle algorithm guaranteed by a new data processing in- equality for Lipschitz regularized divergence which ensures that convergence in latent space implies convergence in real sample space. The proposed generative approach validated on a wide variety of datasets and applications ranging from image generation to gene expression data integration.
Related work. Our approach is inspired by the MMD and KALE gradient flows from AArbel et al. 2019); Glaser et al. 2222 based on an entropic regularization of the MMD metrics, and related work using the Kernelized Sobolev Discrepancy Mroueh et al. (2019). Furthermore, the recent work of Dupuuss & Mao 2022); ; Birrell et al. (2022a) built the mathematical foundations for large class of new divergences which contains the Lipschitz regularized f f-divergences and used them to construct GANs, and in particular symmetry preserving GANs Birrelll al. (202cc). Lipschitz regularizations (or related spectral normalization) have been shown to improve the stability of GANs Miyato et al.. (2118); Arjovsky et al.] (2017); Gulrajani al. (2017). Our particle algorithms share similarities with GANs Goodfellow et al. (2014); Arjovsky et al. (2017), sharing the same discriminator but having a different generator step. They are also broadly related to the Wasserstein gradient flows Fan et al. (2022) which build a suitable neural method for the JKO-type schemes schemes[ordan (Jodan et al. (998).
Furthermore, our methods are closely related to continuous time normalizing flows (NF) Chen al.
( KKhller : et al. (200); Chen et al. (2018b). diffusion models Sohl-Dickstein et al. (2015) Ho
al. (2020) and score-based generative flows Song & Ermon (2020): [Song et al. (2021). However, the aforementioned continuous time models, along with variational autoencoders Kingma & Welling
(2013) and energy based methods LeCun et al. (2006), are all fikelihood-basel On the other hand, particle gradient flows such as the ones proposed here, can be classified in the same category of generative models that include GANs. Here there is more flexibility in selecting the loss function in terms of suitable divergence probability metric, enabling the comparison of even mutually singular distributions, e.g. |Arjovsky et al. (2017). In Section [ and Section F. we compare further our particle methods other generative particles algorithms such RKHS- based gradient flows and score-matching methods. Gradient flows in probability spaces related to the Kullback-Leible (KL) divergence, such as the Fokker-Planck equations and Langevin dynamics Roberts & Tweedie 1996);
Durmus & Moulines (0077) or Stein variational gradient descent Liu & Wang (2016): Liu (011))
Lu el. al.01) form the basis of a variety of sampling algorithms when the target distribution Q has a known density (up to normalization). The weighted porous media equations form another family of gradient flows based on a-divergences Markowich & Villani (2000n)(2000)) Otto (200T); Ambrosio et al.) (005);; DDolbcuult et al.] (2008): V Vázquez (2014) which are very useful in the presence of heavy tails. Our gradient flows are Lipschitz-regularzzee porous media equations), see Appendix B for PDE and numerical analysis perspective on such flows. Finally, deterministic particle methods and associated probabilistic flows of ODEs such as the ones derived here for weapchitt-reuuariizeee gradient flows for (f. divergences, were considered in recent works for classical KI -ivvierggsssss and associated Fokker-Planck equations as sampling tools Maoutsa al. (2020); Boffi& & Handen-Eijnder (2022). for Bayesian inference Reich & Weissmann (2021) and as generative models SSog et al. (2021). Our latent generative particles approach is inspired by latent diffusion models using auto-encoders Rombach et al. (202l) and by autoencoders used for model reduction in coarse-graining for molecular dynamics, Vlachas et al. (2022): Wang & (2019): Stieffenhofer et al. (202T).
of such classical PDE's (Fokker-Planck and
2 LIPSCHITZ-RFGULARIZZI S-DVEEEEEEDDDIIIIEEEEGENNE
In the paper Dupuis & Mao 2022 continuing with Birrell al. (2022a) new general class of divergences has been constructed which interpolate between f-divergences and integral probability metrics and inherit desirable properties from both. In this paper we focus on one specific fam- ily which we view as a Lipschitz regularization of the KL-divergence (or f-divergences)) or as an entropic regularization of the 1- Wasserrstei metric. We denote by P(R") the space of all Borel prob- ability measures on RRd by Pi (Rd) {P E P(Rd): : (x/)P(x)) < oo). We denote by C& (IR)) the
2

---

Under review as a conference paper at ICLR 2023
bounded continuous function and by Fz {f: R R: If(ee) [ - L for all 2, y) the Lipschitz continous functions with Lipschitz constant bounded by L (note that al = Toi).
f-divergences. If [0, oo) R is strictly convex and lower-semiiconiimouss with f(1) 0 the f-divergence of P with respect Q is defined by ()(P)] [ (2)) if P Q and be too otherwise. We have the variational representation (see e.g. Birrell (202aa) for proof)
D)(P)1Q)=
sup (ECs(RR)
Epiø] inf {u + Eolf m)]}
where (s) = suPrEz {st - } } is the Legendre-Fenchel transform of f. We will use the KL- divergence with fKI = log and the a-divergence: fo a(a-I] with Legendre transforms (v) ex-1 and fa b (see the Appendix). For KL the infimum over be solved analytically and yields the Donsker-Varadhan with log EQ |e*] term (see Birrell al. (2022b) for more on variational representations).
Wasserstein metries. The I-Wasserstein metries W integral probability metrics, see Arjovsky et al. (2017). By keeping the Lipschitz constant as regularization parameter set
(P,Q) Q) with transport cost |ar y is
WT&(P,Q) sup [Eplol Eold]} Φetz
and note that we have WT =
Lipschitz- Lipschitz-regulattee regularized f f-divergences. The Lipschitz regularized f-divergences are defined rectly in terms their variational representations, by replacing the optimization over bounded contin- uous functions in ( by Lipsehitz continuous functions in Lz.
DJ (PII)) == sup
[Pp(o] inf (v+ Fols"(6- (1000-11) v11b- PEB
dETz
Some of the important properties of Lipschitz regularized f-divergences, which summarizes results from Dupuis & Mao ) Birrell et al. (2022a) given in Theorem 2.1 It is assumed there that f is super-linear (called admissible Birrell et al. (2022a)), that is lim,-+oc f(s)/s = +oo. This excludes the case of a-divergences for < for which the existence of optimizers more delicate problem, but parts of the theorems remain true.
Theorem 2.1. Assume that f superlinear and strictly convex. Then for P,Q E Pi (Rd) we have
Infimal Convolution Formula: (PI()) = inf (P= In particu- -EP(()
lar we have 0 < D (PIIQ) S min min (PP(())),, (FL((,Q))
Interpolation and limiting behavior of DJ (PI[Q):
= (y())) and lim 1 (P(()) = (P)(..) L
lim L
(4)
Optimizers: There exists optimizer the Ti, unique up constant in supp( Usupp(()) Remark 2.2. The optimizer -n in the infimal convolution formula exists, is unique and d'L L,+ x (MY( (see Birrell et al. 2022a for details). For example for KI., di el dQ.
LIPSCHITZ-REGGULARIII GRADIENT FLOWS AND THEIR TRANSPORTERRDISCCIMINNNT REPRESENTATION
Background. Gradient flows in latent spaces (for example in GANs) or in the space of probability measures (for example in diffusion models and score matching algorithm play central role in

---

Under review as a conference paper at ICLR 2023
generative modeling. It is well-known Jordan al. (1998) that the Fokker Planck equation (FPE) can be thought as the gradient flow of the KL divergence
8 8DL (p)l))))) div Pr VeVlogg log
Oept = div
8p:
where Pt and the densities time t and the stationary density respectively. A similar result relates weighted porous media equation and gradient flows for ʃ divergences OOto (2001). From a generative model perspective where Q is known only through samples (and may not have density in the first place as Q is concentrated on low dimensional structure), one cannot use such flows without further regularization. Score matching and diffusion models start by regularizing the data by adding small amount noise to the data (see SaceSSohl-iiksteee et al. (2013); et al. (2020) ancSong & Ermon (220); Song et al. (202Tt). Next, we propose a different and complementary approach by regularizing the divergence in (5) directly. We refer to Section A and Section F. for further connections between these different approaches and the last Example Section 6
gradient flows. Given a target probability measure Q, we build an evolu- tion equation for probability measures based on the Lipschitz regularized f-divergences Dj (PIIQ)
by considering the PDE
SD (
P,Pr = div P.V
Po = PEP(R)
(6)
8Pr
where (P(I) is the first variation of 2 (PIIQ) (to be discussed below in Theorem 3.1 An advantage of the Lipschitz regularized f-divergences is its ability to compare singular measures and so (6) is to be understood in the sense of distributions (integrating against test functions). For this reason use the probability measure Pt notation in (6). instead of density notation Pt as in the FPE G). In the limit L + oo and if P < Q, yields the FPE (5) (for KL divergence) and the weighted porous medium equation (for a-divergences) Otto 2001): Dolbeault al, (2008). see Appendix B
The following theorem was first proved in Dupuis & Mao 2022) for KL and can be generalized to the f-divergences considered in Theorem 2.1 (see the proof in Appendix C
Theorem 3.1. Assume f superlinear and strictly convex and P, Q E Pi (R.).. Then we have
(PII())
(P) = =
6P
In more details, let p be a signed measure of total mass O and let p= p+ p- where P1 E (R()
are mutually singular. If P + ep Pi (Rdd) for sufficiently small |el then D7 (P + <p|(Q) differentiable e 0 and
lim c+0
(PP(1))))))))) = @4irpp.
(8)
Combining Theorem 3.1 with 6 leads to new class of PDEs:
TransporterDDierrimnatoo PDE:
a.P. = div where @ L.+
arg max (Ep, [o] Rel(((0)))
ΦET,
Remark 3.2, (a) The transpotter/disciiinnato PDE O9 makes sense when P and Q are replaced by their empirical measures Pvv, QN based on N IID samples. This will be the basis of our numerical algorithm in Section (see Algorithm []. (b) Also ( makes sense P and Q are mutually singular (e.g. when Q is supported ization classical PDEs which allows particle-base based approximations based on data. In particular, the Lipschitz condition ó rz enforees finite speed of propagation of at most L in the trans- port equation in (9. This is in sharp contrast with the Fokker Planck equation given in Appendix [ which is diffusion equation, Appendix B.2 for more details and practical implications.
low-dimensional structure). We can view (9) as a Lipschitz regular-
4

---

Under review as a conference paper at ICLR 2023
4 LIPSCHITZ-REGULAAIEE GENERATIVE PARTICLES
In this section we build a numerical algorithm to solve the transporter/diseriinnnaoo gradient flow when N IID samples from the target distribution are given. For map T R2d R and P P(RR), the pushforward measure is denoted by T (i.c. T+P(A) = (A)). The forward- Euler discretization of the system yields:
Euler method for the TransporteDDiscriinnato PDE:
Pa+1 (T-AVV
Pa.
where & arg max {Ep,, [g] (ol)((0]] (10)
ΦET2
When only N IID samples (X JN of the target distribution are available we build a particle system by considering N IID samples J from some initial measure P (M * samples are also possible) and (TO) becomes
Lipschitz regularized generative particles:
Em ((Y N
Emm (((X())
y() y AtV (Y P arg max DETL
(11)
N
The empirical measure PN = N i=l 6 0y10: built from gives a solution of the system (10)
we use as target the empirical measure QN N Σ measure PN - EN 8y Finally we note that formulation of (9) i.e. the ODE/variational problem
ox and as initial condition the empirical
is time-discrttization of the Lagrangian
dt Y = -ott (Y,,t), where the = -argmax dEIr
Ep, 1ol inf {v+ (o -wll]] (12)
VER
Algorithm 1: Lipschitz regularized generative particles algorithm
Require: f defined in (2) and its Legendre conjugate f", L: Lipschitz constant, 1:: scalar parameter for optimizing divergence,
T: number of updates for the particles, 7: time step size, N: number of particles
Require: W [W parameters for the neural network é Rd + R, D: depth
of the neural network, d: learning rate of the neural network, TNN: number of updates
for the neural network.
Result: [ hi=1
Sample [X ~ Q, batch from the real data
Sample (
lim1 P. P. a batch of prior samples Initialize / < 0, W randomly and W + LT/ W /1|
for 0 to (T 1) do
for m = 0 to TNN - do
gradw.r + Vw N-1 ΣN ((Y, ; W) N--1 (O(X)); W) -v) -v)+
W W + 8 gradw, = 8 grad,
W + 1// W W
end 10
(
(1) VoL W), i= 1,..,,
end
// The height of & is adjusted by the optimization over v. This keeps o(xx") W) values to reside in the domain f and allows to avoid the degeneracy.
Remark 4.1. (a) The transport mechanism given by is linear. However, nonlinear interactions between particles as introduced via the discriminator L are created due to the velocity field VoL..* which depends all particles that comprise P.N and Q each step n. (b) Computationally, the
5

---

Under review as a conference paper at ICLR 2023
discriminator optimization (over Lipschitz continuous functions) is implemented, for example, via spectral normalization for neural networks architectures. Moreover the gradient of the discriminator is computed only at the positions of the particles. (c) The Lipschitz bound L on the discriminator
space implies a pointwise bound [V (Y:"*) S L and thus the particle speed is bounded by L. Hence the Lipschitz regularization imposes speed limit L the particles, ensuring the stability of the algorithm for suitable choices of L. This implicit grid is reminiscent of the Courant, Friedrichs, and Lewy (CFL) condition for the stability of discrete scheme. These are fundamental features for the performance and the stability of Algorithm| [ derived from (11) (see Sections 66) and Appendix
Kinetic energy of particles: The gradient structures implies, see Theorem B.1 that, for (9) d
the derivative of the regularized divergence satisfies de D = - - where
(P-IQ) = Ep. 13] which is interpreted as a regularzze Fisher Information. As I, + oc one recovers for example the Fisher Information used for the Fokker-Plance equation. For the Algorithm the Lipshitz-regularizer Fisher information
(P((PNINN)) = Vol. (dr)
1 N
Vox (X
(13)
is equal to the total kinetic energy of the particle since Voli (Y)))) is the velocity of the hd particle at time Clearly when the total kinetic energy 1 2 (PN (12) is zero, the algorithm will stop.
5 LATENT GENERATIVE PARTICLES: GRADIENT FLOWS IN LATENT SPACE
A standard paradigm of machine learning is that target measures are often supported on low di- mensional structures. We leverage this insight, in the form of an auto-encoder, to construct particle algorithms in latent, lower dimensional space. The resulting latent particle algorithms more accurate and efficient, even in high- high-dimensional sample spaces, and their performance is guar- anteed by new Data Processing Inequality in Theorem 5.1 Assume Q = Q* is supported some low dimensional set S ) Re encoder map & : y - Z where 2 Rdd d' < and decoder map D Z - y are invertible in S, i.c. D o E(S D(Z) S. We denote by Q² the image of the measure Q" by the map &, i.e. for A C Z, we define Ey ("((()))) likewise for Dy P2. The following theorem expresses how information remains controlled under encoding/decoding and guarantees the performance the approximation Dæ P2 in the real space. The latter is achieved by posteriori estimate (14), in the sense of numerical analysis, where the approximation in the tractable latent space Z will bound the error in the real space y
both
(E)) and
Theorem 5.1. Suppose that
1. Perfect encoding. For Q the encoder & and the decoder 'D such that D Q Q²
2. Lipschitz decoder. The decoder is Lipschitz continuous with Lipschitz constant ap.
Then, for any p2 E P2) we have
DJ (D 10 s DJ (P²
(14)
This theorem and more general versions thereof for other representation learning tools beyond au- toencoders, is proved in Appendix D.2 The proof consequence of new, tighter data processing inequality derived in Birrell et al. (2022a) that involves both transformations of probabilities and discriminator spaces
Remark 5.2. In practice an autoencoder is trained on data using the empirical measure Q and suitable loss function and neural network architectures. Assumption in Theorem 3.1 can easily be enforced using e.g. spectral normalization. Assumption is a reasonable, but somewhat idealized, version of the requirement that the autoencoder captures adequately the features of the dataset Q. In particular the dimension of the latent space Z needs be selected carefully (see Section 6).

---

Under review as a conference paper at ICLR 2023
Gradient flow in latent spaces. If o is the discriminator in latent space leading to the gradient flow ( a,P,² = div(P, Vof) then, in the particle algorithm, each particle is transported follow- - -of (2t), as in Section 4 The Algorithm 2 can be
ing (the time-discrtizzation of) the ODE found in the Appendix D.3 Upon decoding we find the transport ODE in real space is
0D
T 2, - Oz (x1)
(3)
@DD Oz (zt)
0D Oz
J1 =
(15)
where 8D (2x) is the Jacobian of DD at the point on and the reconstructed discriminator x² is given by o² e² o D. Using 15 we can therefore interpret the encoding as learning a mobility μe = 85
85
(x) te
8D (z+), i.e., learning a better geometry in real space. This leads to gradient flow in real
space with non-trivial mobility, cf. (9)
O.P = div (HPPPVO?)
(16)
We note that the mobility concept is well-known in computational materials science where it is used
model kinetics of species and interfaces, see for instance Cahn (1965); [Zhu al. (1999); Wang
et al. (2020). Finally, we note similar computation to (15) in Mrouch et al. (2019) regarding the
interpretation of GAN's as a gradient flow. The differences (and similarities) between (Lipschitz- regularized) Generative particle algorithm (GPA) and GAN are summarized in Figure 4 and Table
where the latter we also include comparison between mobilities.
GPA
GPA latent space
GAN
Discriminator
Lip())) (ly AIV ) P
d² Z))
Lip()> Go(z), ~ (O, 1z)
Generator
(oo(1 A(o
P
Partieles
Particles
Generator parameters
Updates
y E y
2 Z
O R|
Mobility y 16
ly
0D
0z (2t)
7 aD
0= (=))
ag(o..:) :) T 06(0..=)
89
Table 1: Comparison of features in GPA, GPA in the latent space and GAN. Note that GPA consists of one neural network for the discriminator while GAN has two neural networks for the discriminator and the generator, respectively. We use the notations y Fd and Z E()) for distinguishing the real space and the latent space. A schematic diagram in Figure 4 shows how the Latent GPA and the GAN interact between the latent and the real spaces.
EXPERIMENTS
We present three types of experiments: (1) generating MNIST images from small number of data, see Figure D (2) merging gene expression data sets which are very high-dimensional (and thus require latent space description) but also have low number samples the order of the low hundreds since they correspond to patients, see Figure 2 (c) generating heavy-tailed distributions where KL maximum likelihood- based methodologies will necessarily fail since f-divergence is required, see F Overall, we found that GPA perform best in learning from small number of samples, learning distributions with heavy tails, and exhibit enhanced learning when latent space is available.
1. Learning MNIST from few data. Our (fKL, I1) --PA is found to perform well in generating images using a small number of samples while GANs struggle with limited data. We stress from the digit-conditional MNIST data generation example in Figure [ that (fkr., T.P-GPA could generate images from ten digit labels out of 200 samples. We compare the performance with (JKI, [1 J-GN and the well-known Wasserstein GAN. FID values and the improvement of our result with latent GPA can be found in Appendix F2
2. Merging of microarray gene expression data sets Using GPA, we can transport arbitrary souree data to arbitrary target data even if a relatively small number of samples is available from the target. Furthermore, combining our algorithm with representation learning, we introduce a bioinfor- matics application of our algorithm in the analysis of gene expression data. Gene expression datasets

---

Under review as a conference paper at ICLR 2023
0
0
6
(a) (fKL.
T
GPGA 200 samples
(b) (/kL, (i),TN))-GN,, 200 samples
(c)
Wasserstein 200 samples
000
0
0
0
a
0
DGD
{
C
f f
2
2
2
2
2
3
2
2
2
2
2
Q
2
7
3
5
3
7
3
8
6
3
3
3
8
4
9
M
4
4
44
4
4
S
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
G
6
6 7
7
7
7
7
7
7
7
77
77
7
7
8
C
8
8
0
6 9
8
8
R
8
8
9 9
9
9
9
7
T
(d) (fKL, GPA. 2000 samples (c) (fKL, -1)-GAN, 2000 samples (f) Wasserstein- GAN, 2000
ples
Figure I: MNIST; Learned digits by different generative models (column) with different num- ber of training data (row). The (fKL, I1)L.P1)-PPA G-PA was able to learn digits from a small data set, while the other methods failed. Using sufficiently large training data, GANs outperformed in capturing the scale, which can be observed by the more intense color contrast between a digit and its background. See FID scores in T
are not only high-dimensional but also small-sized thus is crucial to increase the sample size by in- tegrating together all available datasets from the disease. However, this is not straightforward process since it is well known that gene expression datasets may have different statistics even when they target the same disease; phenomenon referred to "batch effects"" TTan et (2020). We pro- pose to mitigate batch effects via the latent generative particle algorithm and match the statistics of the two datasets. Figure presents the results on applying our algorithm between two breast cancer datasets from the Gene Expression Omnibus (https: www. ncbi mnmmm ni h. gov geo/) More technical details and experiments can found the Appendix F.3 F.3
3. GPA and Porous medium equations for heavy-tailed targets Similarly to score-based meth- ods (sce equation 22 in Section A) developed in Boffi & Vanden-Eijnden (2022) to solve the FP equation our proposed methods will allow develop new particle systems algorithms for solving porous medium equations with steady state probability measure Q and density q:
))
Pt (1)
Oep: div
pr V
(17)
For the special case where the f-divergence is a-divergence with fa(x) ~ x², obtain power law in equation 17 and ultimately the well-known porous media equations, e.g. [Boforre al. (2010;; Otto 2001): Dolbeault al. (2008), used for applications to actual porous medium flow, typically in dimension 3. However, here we propose porous medium equations and associated particle algorithms as statistical learning tools for pdfs with heavy tails. For instance, score-based methods, are KL-based, see Song et al.) and the discussion in Section A hence they are not suit- able for heavy tailed distributions: we refer the collapse in the algorithms that minimize KI. divergence observed in Figure in Birrell et al. (2022a), in stark contrast to the stable behavior of suitable (f, -'ddiivergesces for heavy tails. For this reason, we propose GPA algorithms based on the porous medium equation equation| 17 and its more stable Lipschitz-regullrizze particles-based solution given by the Algorithm [ a new efficient and reliable way for generating samples from heavy-tailed targets Q. See the example in Figure 3 Our methods are also partly theoretically grounded on of asymptotics and related functional incqualities results for porous medium equations,
e.g. Dolbeault et al. 2008); BBnforte al. (2010).
8

---

Under review as a conference paper at ICLR 2023
+ HM4DESOUTED recon) -+- HHeiitageet recon)
MMD
(a) Particles in the latent space
(b) Particles in the real space
Figure 2: (Gene expression data; Merging Breast Cancer datasets). We merged gene data using our latent GPA in significantly lower dimensions. Two distinct gene data sets but from the same disease decrease their dimension from 54,675D into d' 2, 5, 10. 20, 50, 100, 200 using normalized PCA. Then, latent particles are transported using GPA. blue: source (206 samples), red: target (245 samples), black: transported (206 samples). (a) Latent particles in the RRd with d' 20 which are encoded by the PCA. (b) Transported samples are reconstructed to the real space. The 2D visualizations are obtained using the UMAP algorithm Mclnnes et al. (2018). (c) The MMD distance Gretton et al. (2012) between the reconstructed datasets. blue: MMD PY), red: MMD(Q" PF). ). T 25K. The transported distribution has smaller distances from the target distribution when d' = 5, 10, 20, 50, 200.
Sanget oudgut
output
20-
-10
-10
-20
-20
-30
-20
20
(a) 0 reppschit-rregularizzdd GPA
(b) Score matching and annealed Langevin dynamics
Figure 3: (2D Mixture of Student-t) (fa, GPAA with 2 for heavy tailed target and comparison with score based model. 200 target samples from Student-t(p) with v = 0.5 are provided to transport 500 particles which uniformly distributed in the plotted region time t t Blue: target, Orange: output. (a) The choice of divergence Ja with a = 2 and propagation of particles through the (Ja, I-PAA captures the heavy-tailed target. (b) Noise conditional score network (NCSN, score based model Song et al.) evolves particles by learning the vector field, the score log (xx). from data. However, mixture of disjoint distributions makes hard to learn the score where the data is sparse. NCSN tackles the problem by injecting different levels of noise on the data and learning the scores of noise-injected distributions. Then propagates particles through annealed Langevin dynamics using sequence of scores 8, with different noise levels a. When the level of injected noise gets smaller as much as < 1, score-matching of the perturbed data for Student-t was extremely hard. In addition, particles transportation through (annealed) Langevin dynamics might not lead to the convergence to the heavy tailed distribution. similar comparison for mixture of Gaussians can be found in Figure

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
Each figure contains main parameters such as divergence-peecffying f. Lipschitz constant L, and dataset parameter. In Appendix Section E experimental setting for the experiments the main text and appendix are described in these aspects:
Data sets
Neural network architectures
Computational resources.
In Supplementary material, source code, all dependent libraries, and documentation (README.md) are attached. README.me specifies the required open-source libraries and the entire parameters set including random seeds for reproducing individual experiments.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media, 2005.
Michael Arbel, Anna Korba, Adil SALIM, and Arthur Gretton. Maximum mean discrepancy gradient flow. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. 314a5ae3483ed5ccoe wab.e1lbbcccbb 7942a279 Paper. pdf
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial net- works. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Con- ference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp- 214-223, PMLR, 06-11 Aug 2017. URL https: https::/ https://proceedinnge. m wtt.a/ssss.......//100.0
sEjovsky17a.html
Jeremiah Birrell, Paul Dupuis, Markos A Katsoulakis, Yannis Pantazis, and Luc Rey-Bellet. (f- d)-divergences: Interpolating between f-divergences and integral probability metries. Journal of Machine Learning Research, 23(39):1-70, 2022a.
Jeremiah Birrell, Markos A. Katsoulakis, and Yannis Pantazis. Optimizing variational representa- tions of divergences and accelerating their statistical estimation. IEEE Transactions on Informa- fion Theory, pp. 1-1, 2022b. doi: 10. 109/TIT.2022.3166655
Jeremiah Birrell, Markos A. Katsoulakis, Luc Rey-Bellet, and Wei Zhu. Structure-preserviin gans. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Bal- timore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 1982- 2020. PMLR, 2022c. URL https: http:tts/ppcceeed.gggg mlr http:///.w.........r..e///////// wtt..perss///le//rreeeeeee html
Nicholas M. Boffi and Eric Vanden-Fijnden Probability flow solution of the Fokker-Planck equa- tion. arXiv e-prints, art. arXiv:2206.04642, June 2022.
M. Bonforte, Dolbeault, G. Grillo, and J. L. Vazquez. Sharp rates decay of solutions to the nonlinear fast diffusion equation via functional inequalities. Proceedings of the National Academy of Sciences, 167939):1445--1666 2010. doi: 10. 10.1073ppas. 1073//paa.10003921000 URL https:/ 1073/ /paa.. C03972107
www. www.pppas..cmm . wrtp://.r/////////////
John W. Cahn. Phase separation by spinodal decomposition in isotropic systems. The Journal of Chemical Physies, 42(1):93 99, 1965. doi: 10. 10.1063/1.1695311. URL https:://o......
10. 10.1063/1..695731
José Carrillo, Jean Dolbeault, Ivan Gentil, and Ansgar Juengel. Entropy-energy inequalities and im- proved convergence rates for nonlinear parabolic equations. Discrete and Continuous Dynamical Systems-Series B, 605: 2006.
José Antonio Carrillo, Yanghong Huang, Francesco Saverio Patacchini, and Gershon Wolansky. Numerical study of particle method for gradient flows. Kinetic and Related Models, 10(3): 613-641, 2017.
Changyou Chen, Chunyuan Li, Liqun Chen, Wenlin Wang, Yunchen Pu, and Lawrence Carin Duke. Continuous-tim flows for efficient inference and density estimation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 824 833. PMLR, 10-15 Jul 2018a. URL www.sppressseemm wtw.s///0//ssseeeeeepe chen 8d html
fttt::://proceedinee
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Masa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- httpo//ooo..uuutpp//////cooooaa.
ciates, Inc., 2018b. URL ht

---

Under review as a conference paper at ICLR 2023
Jean Dolbeault, Ivan Gentil, Amaud Guillin, and Feng- Yu Wang. I'-functional inequalities and weighted porous media equations. Potential Anal., 28( 59, 2008. ISSN 2601- doi: 10. 9066-0
URLhttps::/aoii http://otoo.ogo.////// org/ 111.:1..1000111111111
Paul Dupuis and Yixiang Mao. Formulation and properties of divergence used to compare prob- ability measures without absolute continuity. ESAIM Control Optim. Calc. Var, 28:Paper No. 10, 38, 2022. ISSN 1292-8115 8119.
doi: 10. 00.1011/cooc//222200.. URLhttps: / org/10.
1051/cocv/202002
Alain Durmus and Eric Moulines. Nonasymptotic convergence analysis for the unadjusted Langevin algorithm. The Annals of Applied Probability, 27(3):1551 1587, 2017. doi: 10.1214/
16-AAP1238.
Jaajiao Fan, Qinsheng Zhang, Amirhossein Taghvaei, and Yongxin Chen. Variational Wasserstein gradient flow. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 6185-6215. PMLR, 17-23 Jul 2022. URL https: htt.tttss./ppooceeeeeeeeeee http://pocceeeeeeeeeeeeeeeeeecee mlv wtt.://ww.p...s..ee.eeeeeeeeeeee
62 fanddd.nn2dddii. html
Pierre Glaser, Michael Arbel, and Arthur Gretton. Kale flow: A relaxed gradient flow for prob- abilities with disjoint support. In M. Ranzato, A. Beygelzimer. Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.). Advances in Neural Information Processing Systems, volume 34, pp. 8018 8031. Curran Associates, Inc., 2021. URL https: http::/::////ppppeeeeeeeeoe neurips cc/ pdf
lan Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. J. Mach. Learn. Res., 13:723-773, March 2012. ISSN 1532-4435.
URL http://dl. acm. wttp://ttiioo....iitcieoo 2188410
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httptattt/ppppeeeeeeeeeee /ttp://///./peeeeeg...... nwwrips....... ea/ smoothing splines. Communications in htt.:e2223336222cc0c066666833333 wtw. Paperr.appeeeeee pdf 1989.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian and Computation, 18(3):1059-100 1076,
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. J. Mach. Learn. Res,, 6:695-709, December 2005. ISSN 1532-4435. URL http: //dp: acm. org/
citation. www.:onni.mmmmmm00mm00m0 108169990.18886666 1088696
Richard Jordan, David Kinderlehrer, and Felix Outo. The variational formulation of the fokker- planck equation. SIAM journal on mathematical analysis, 29(1):1-17, 1998.
Diederík P Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv e-prints, art. arXiv:1312.611 December 2013.
Jonas Köhler, Leon Klein, and Frank Noe. Equivariant flows: Exact likelihood generative leamn ing for symmetric densities. In Hal Daumé and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learn- ing Research, pp. 5361-5370. PMLR, 13-18 Jul 2020. URL URL htttps:/pprocedd momeeediing.mmmm. mmr.
press/v119/koaler200 html
Yann LeCun, Sumit Chopra, Raia Hadsell, Mare' Aurelio Ranzato, and Fu-Jie Huang. A tutorial on energy-basee based learning. In G. Bakir, T. Hofman, B. Schölkopf, A. Smola, and B. Taskar (eds.), Predicting Structured Data. MIT Press, 2006.
12

---

Under review as a conference paper at ICLR 2023
Randall Le Veque Finite Difference Methods for Ordinary and Partial Differential Equations: Steady-State and Time- -eppendend Problems (Classics in Applied Mathematics Classics in Applied Mathemat). Society for Industrial and Applied Mathematics, USA, 2007. ISBN 0898716292.
Qiang Liu. Stein variational gradient descent gradient flow. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL httptt::/ppooeeeeeeee neur ps. www.eeerppps.cpppprr/////.. cc.ppager/201//fiee pdff
1 0tbiaaadd22599002744662222223333
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Asso- ciates, Inc., 2016. URL /ttps::/proccedanss htwp://www.eeiiioo.e.i.orrooooor b 222aa8leee2222227773300333888823
pdf
Jianfeng Lu, Yulong Lu, and James Nolen. Scaling limit of the stein variational gradient descent: The mean field regime. SIAM Journal on Mathematical Analysis, 51(2):64-66) 671, 2019. doi: 13.1137/18 187611. bRtps::::///oi. http:///oo..rr/..../// 01137/1MM118661
Dimitra Maoutsa, Sebastian Reich, and Manfred Opper. Interacting particle solutions of fokker- Baakerr-panck equations through gradient-log density estimation. Entropy, 22(8), 2020. ISSN 1099-4300. doi: 10.3390/e22080802. URL https:////// /ww. mdpi. com 1099-43009922
8/802
Peter A Markowich and Cédric Villani. On the trend to equilibrium for the fokker-planck equation: an interplay between physics and functional analysis. Mat. Contemp, 19:1-29, 2000.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossherger. Umap: Uniform manifold
approximation and projection. Journal of Open Source Software, 3(29):861, 2018. doi: 10.
100600sss.0086.. URL httph::///// http://ooo.oggg//// 2ee.10..211//ooo joss. D0861
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. 02 2018.
Youssef Mroueh, Tom Sercu, and Anant Raj. Sobolev descent. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2976-2985. PMLR, 2019.
Otto and C. Villani. Generalization of an inequality by talagrrdd and links with the logarithmic sobolev inequality. Journal of Functional Analysis, 173(2)3611-00,, 2000. ISSN 0022-1236. doi: hrtp://doo..or//0...100000 http://jjjannn.999999 1999. 3557. URL https::///// http:///wwww.ceeeeeeeeeeetttttte com/
Felix Otto. The geometry of dissipative evolution equations: the porous medium equation.
Comm. Partial Differential Equations, 26(1-2):101-17,, 2001. ISSN 0360-5302. doi: 10.1081/
PDE-100002244 htth:tt:: wtt.0//0/000..1////02...2. SOODE-10002233
Sebastian Reich and Simon Weissmann. Fokker-planck particle systems for bayesian inference: Computational approaches. SIAM/ASA Journal on Uncertainty Quantificatiin, 9(2) 482,,
2021. doi: 10. 137/19911300 303116 162. URL httliitps:///dd htt///oi.org//.// www.///..... 1137/1 9M1 3031 62
Gareth O. Roberts and Richard L. Tweedie. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, 2(4):341 363, 1996. doi: bj/1178291835. URL
https:://di..rg//
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. CoRR, abs/2112.10752, 2021. URL
https::/arxiv..or
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution, 2020.
13

---

Under review as a conference paper at ICLR 2023
Yang Song. Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score- based diffusion models. Advances in Neural Information Processing Systems.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp. 574-584. PMLR, 2020.
Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ArXiv, abs/201 1.1456,, 2021.
Mare Stieffenhofer, Tristan Bereau, and Michael Wand. Adversarial reverse mapping of condensed- phase molecular structures: Chemical transferabilityy. APL Materials, 9:031107, 03 2021. doi:
Guiseppe Toscani and Cédric Villani. On the trend equilibrium for some dissipative systems with slowly increasing a priori bounds. Jourmal of Statistical Physics, 98(5):1279-1309, 2000.
Hoa Thi Nhu Tran, Kok Siong Ang, Marion Chevrier, Xiaomeng Zhang, Nicole Yee Shin Lee, Michelle Goh, and Jinmiao Chen. A benchmark of batch-effect correction methods for single-cell sequencing data. Genome Biology, 21(1):12-12, 2020. ISSN 1474-760X.
Juan Luis Vázquez. Barenblatt solutions and asymptotic behaviour for nonlinear fractional heat equation of porous medium type. Journal of the European Mathematical Society, 16(4):769-803, 2014.
Pantelis Vlachas, Georgios Arampatzis, Caroline Uhler, and Petros Koumoutsakos. Multiscale sim- ulations of complex systems by learning their effective dynamics. Nature Machine Intelligence, 03 2022. doi: 10.445550/rXiv.2006.. 13431.
Feng-Yu Wang. A generalization of Poincaré and log-Sobolev inequalities. Potential Anal., 22(1): 1-15, 2005. ISSN 0926- doi: 10. 11101//111111000111111111
(RLhhtts:://oii orq/
10.1007/1111--004-4000
Feng-Yu Yu Wang. Orlicz-poincaré inequalities. Proceedings of the Edinburgh Mathematical Society, 51(2):2995543 2008
Qiao Wang, Geng Zhang, Yajie Li, Zijian Hong, Da Wang, and Siqi Shi. Application of phase-field method in rechargeable batteries. npj Computational Materials, 6(1):1-8, 2020.
Wujie Wang and Rafael Gömez--ombbrelli. Coarse Coarse-graining auto-encoders for molecular dynam- ics. npj Computational Materials, 5:125, 12 2019, doi: 10.1038641522-019-22611..
Jingzhi Zhu, Long-Qing Chen, Jie Shen, and Veena Tikare. Coarsening kinetics from a variable- mobility cahn-hilliard equation: Application of semi-implicit fourier spectral method. Physical Review E, 60(4):5564 1999.
14

---

Under review as a conference paper at ICLR 2023
A CONNECTIONS OF GPA WITH SCORE GENERATIVE METHODS
In this Section we discuss connections of GPAs with diffusion-type generative models in particular with score-based based models which seem to be the most closely related. Score-based generative mod- eling (SGM) relies closely on concepts and methods related to Langevin samplers, e.g. Durmus & Moulines 2017 Given R--valued samples [ Ji=1 from an unknown probability Q with distribution 4, we want to produce more realizations from Q. In SGMs, the score of the unknown distribution, V log ((v) is learned from the training set. An optimization problem to learn the score is defined follows Song & Ermon (2020): we search for : Fd + Rd in a function space F parametrized by 0 (typically neural networks),
min L(0) min 1
2 / [|s(y: V log (()))l(()))d
(18)
The key observation in (18) is that the loss functional can be estimated via available samples from Q without any evaluation of the density q (of Q) or V log q. |Hvärinenn (2005). Indeed, by expanding
the square and integrating by parts in 18 we arrive at equivalent loss function J(f):
argmingL(0) argmin, (0)) argming Ei
1 2
(s(y:))) I) + Vy s(y: 8)
(19)
Since computing the divergence of s(x; 0), when it is expressed some neural network is quite pensive, the complexity of this last loss functional (19) can be further reduced by using a Hutchinson- type randomization Hutchinson (1989) for the efficient evaluation of V,, sg: 0). by developing the so-called "sliced score matching method Song et al. (020).. Lastly, we remark that it may initially appear that the selection of the loss functional (18) is somewhat ad hoc as in many other successful machine learning algorithms. However, using the Girsanov Theorem it can be shown that (18) has rigorous cnermattiontheooetti underpinnings because it is related to the minimization of the KL divergence on path-space, [Song et al.
After learning the score from the minimization of (1). namely s(y) = s(y; 0*) V log g(y), there are several directions that can be taken towards generative modeling. The most naïve option is to simulate trajectories of the overdamped Langevin dynamics using the learned score:
dy; d(t) + 22W,, where 8(y) ~ V log ((v)
(20)
As discussed in Song & Ermon 2020). naïvely applying this approach is not practical: (a) Langevin dynamics will not sample from the true data distribution when the data lies on a lower dimensional manifold; (b) the score cannot be accurately estimated for regions with little data; (c) Langevin dynamics mixes poorly, and therefore generates poorly, especially for multi-modal systems. To overcome these challenges, Song & Ermon| 22020); Song et al. (222)) propose to add noise to the available data from Q in a systematic fashion. In [Song& & Ermon (2020). the authors propose to build a noise-conditional score network, in which sequence of noisy datasets are generated by adding different noises to the given dataset. In the end, we simulate an annealed Langevin dynamics analogue to 20
dY, = o² (t) log (YY) + (2)
(21)
with the annealing schedule (t). The distribution of Y, at some finite cut-off time T is supposed to approximate the data distribution q. Indeed (21) is shown to mix better, and estimate the score better in regions of low probability, /Song& & Ermon (2020). In Sonn et al. (02I), the author propose an alternative method by considering a forward-back backward formulation However all these Langevin dynamics approaches require tuning- - mainly since the annealing schedule or the forward model need to be user-pres eser-preseribed and is problem-depender like all annealing methods.
Connections between Score-based methods and GPA. The distribution pt of the solution Y, of the SGM (20) is the solution of classical Fokker Planck (FP) equation (5), given a target distribution q. The FP is of course a diffusion equation, but it can also be viewed as transport equation taking
the form O.p: = V (w(x,, t)):), with velocity field v(x, f) = V LK lg) In analogy to 12 we
ópc
can write the solution of this transport formulation as the density of particles evolving according to its Lagrangian formulation
S Yr v(Y,,)) V Vlog V Po.),, where Y, ~ Pt.
d
(22)
di
15

---

Under review as a conference paper at ICLR 2023
In Song et al. (2021), the authors proposed the deterministic probability flow (2 alternative
to generative stochastic samplers such as 20 and (21) due to advantages related to better statistical estimators.
In the perspective of GPAs, we can write (2) in an equivalent transport/variatiinaal problem that employs the variational formulation of the KL divergence, 1). First, we note that o" (z) log (x(z)
q(x)
is the maximizer (discriminator) of this variatonal representation, [Brrelll et al. (2022b). Next, no- tice that (22) can also written in the equivalent particle/variatioaal form 12 which for the KL. divergence, and due to 6* (z) = log p=(a)) takes the form q(a)*
X, v(X,()x (X,0) where e- argmax Ep, [6] inf {u Eole di
VER
(23)
evident that (23 mathematically equivalent to (22 since (0))) = log p=(at However,
g(ar)
the particle/variational problem (23) (23)-through its generalization (T2)-allows us to use much wider
variety of divergences that-unlike KI. used in score-based methods-ard suitable for probability dis-
tributions supported at lower dimensional manifolds, are singular such as empirical distributions, or
have heavy tails. These are precisely the featues of the GPA developed in in this paper, based on (f. diverggessss.
B PDE, CONVERGENCE & LEARNING RATES
Here, we discuss how PDE perspectives and tools provide insights for the analysis, stability and convergence for the proposed generative particle algorithms. We focus on continuous time for con- venience. By recalling Theorem 2.1 part 4. and 2.4 + P as + oo (if P is absolutely continuous with respect to Q) in Remark 2.2 becomes Lipschitz-reguulrriedd f-divergences gradient flow (with its limit as + 00), i.c.
(
dP: dQ
0, = div P.VS'
O = div P.V
(24)
dQ
L-x00
Lip. regulurized f-divergence
f-divergence
The right hand side of (24) nonlinear operator which encodes the Lipschitz regularization in the discriminator space. This defines a new class of PDE gradient flows where absolute continuity between P. and Q for every / 2 0, is not required, contrary to gradient flows of f-divergences (ob- tained L co.) We discuss these connections next in the context of two well-known PDBs, the (linear) -lanckk and the (non-linear) porous media equation. Rewriting the limiting equation in terms of the density hg = dPt we have
dep
1. Lipschitz-regularized Fokker-Planck.
For the KI, f(x) = x llg(()) we obtain
dhi t..*
O.P: = div
P.V log
0, (A + V log(g) V)
(25)
dQ
Lipschite-regulariied Weighted Porous Medium equation (WPME). For the a-divergence obtain regularization of the porous media equation Otto (2001)
with (xx = a
Dolbeault et (2008)
a-1
o,h, 1 (A ( logq V) K
0.p a div
Pt
V
(26)
I-+00
B. CONVERGENCE TO EQUILIBRIUM AND FUNCTIONAL INEQUALITIES
Functional inequalities are fundamental methods for guaranteeing the convergence of gradient flow PDE to their equilibrium states and therefore are natural tools for studying convergence properties for the corresponding particle-based algorithms. A first step is to compute the rate of change of the divergence along solutions P; of (9)
16

---

Under review as a conference paper at ICLR 2023
Theorem B.1. [Lipschitz regularized dissipation] Along trajectory of smooth solution {Pi)i20 of with source probability distribution P we have the following rate of decay identity:
D (P.IQ) = -1+1 0o
(27)
where we define the
Fisher Information as
(PI())))) - (VLL+PP((d) / = Ep | (()har) dQ )
(28)
Consequently, for any T 2 0, we have (P:)))))
So I (P.lQ)dss
Remark B.2. (a) For the generative particles the Lipschitz-reegularized Fisher Information can be in- terpreted as their total kinetic energy, see Paragraph| 12 (b) When f fKL. as + oc, we recover which is used to prove convergence
the usual Fisher information 1 (PI[)) = Ep V log
2
to the equilibrium state for the Tokker-Plande equation
25
Functional inequalities, such as the classical Poincaré and the Logarithmic Sobolev-type inequal- ities, and many generalizations thereof see Markowich Villani (2000); Otto & Villani (2000);
Toscani & Villani (2000); Dolbeault et al. (2008); Wang (2005) are powerful to prove convergence
to equilibrium (e.g exponential or polynomial convergence), building on dissipation estimates such as Theorem B.1 For example, if for some À > 0, a Sobolev inequality DJ (PIIQ) < 7
holds (true when Q is sub- uub-Gaussian), then we obtain exponential convergence to Q for any Pp: (P.I)) S e At DD (PollQ). There exits various results, e.g- Carrillo et al. (2006); Dolbeault al. 2008 Markowich & Villani (2000); Wang (2008), reviewed in the Section B3 B.3] on func- tional inequalities when L = оo which have been used to prove convergence to equilibrium (at exponential or polynomial rate) for Fokker-Plande and/or the porous media equation when the tar- get distribution are Gaussian distribution, stretched exponential and Student-t type distribution. The existence of functional inequalities for Lipschitz-reegllarzed gradient flows is an open question.
B.2 NUMERICAL ANALYSIS FOR LIPSCHITZ REGULARIZED PDES AND GENERATIVE PARTICLES
For a Lipschitz-regullrrzzed gradient flow 24 the transpotterdiscriinnator representation 9 im- plies that the domain of dependence is determined by the velocity fields Vol"** whose norm is bounded by the Lipschitz constant L. Therefore the domain of dependence of the solution is finite
and is contained in a cone of slope L that emanates from any point (z, t) back to the time plane t O.
From a numerical analysis point of view, 110 is an explicit numerical scheme Pl++1 - Pr At div (pr. (x)). For corresponding spatial discretization schemes there is an abundance of nu- merical methods which we can use to get some numerical analysis insight into our particle schemes (10). In particular, the Courant, Friedrichs, and Lewy (CFL) condition for stability of discrete schemes asserts that a numerical method can be convergent only if its numerical domain depen- dence contains the true domain of dependence of the continuous PDE, Le Veque (2007). In our context, the CFL condition means sup,, /V (x)) At Cmaa where Ciaa 1 for such explicit schemes, (207). Clearly, the Lipschitz regularization enforces a CFL type condition with learning rate At proportional to the inverse of L. It remains an open question how to rigorously extend this CFL analysis to particle- based algorithms where the spatial discretization grid Ar is known only implicitly as noted in Remark 4.1.b), see also related questions in Carrillo et al. 2001 Nevertheless, in our experiments we explore the inversely proportional relation between L and At suggested by the CFL. analysis.
B.3 POKKER-PLANCK EQUATION, AND ITS CONVERGENCE TO EQUILIBRIUM STATE
Generalized Fokker Planck as the gradient flow of /ddivergences. Let p: is the density of Pt. The associated gradient flow is given by the generalized Bokker-Plannde equation
0.pr = V. (p. (P QQ) (pe)
V (DeV V Pt
(29)
8P
q
17

---

Under review as a conference paper at ICLR 2023
The Fokker Planck as gradient flow of KL. When f fKL. obtain the known Fokker-Plann equation
Pr Vq
Oep. Ap: + V.
O.
B.3.1 EXPONENTIAL DECAY WHEN e AND V X-CONVEX
In this section for simplicity that the probability densities of both source and target distributions exist and are denoted by We consider the Cauchy problem of the Fokker-Planck equation given Section B with
p(l 0,,) p 0 and f pp=1. 1.
(30)
The next theorem in Markowich & Villani (2000) gives us the conditions that probability measure satisfies order to logarithmic Sobolev inequalities and consequently exponential decay.
Theorem B.3. Let q E I. (R44 and V be X-convex (i.e. D²V(2)) 2 for all x E RR4), where La is the identity matrix dimension d. Then, q satisfies logarithmic Sobolev inequality with constant X, iee. (pl(p)))) S (pllq)) and the solution of the homogeneous Fokker-Planee equation goes to equilibrium in KL divergence, with rate e -2M at least.
Typical examples that satisfy the conditions of Theorem B.3
(
f --12/ for Rd with
822
(31)
When B = 2, the target probability distribution with density is the Gaussian with variance and zero mean, i.e.
q(x) (2 24/2
(i.e. = Lef By applying Theorem B.3 we get that for any initial probability distribution P which is absolutely continuous with respect Q.
DKL (P: (14) S DRL (pollg)e 21/a
(32)
where we have also used that the Stam- Gross Logarithmic Sobolev incquality, i.c. (KL(p)lg) - 5 (pllq)). see formula (14) in Markowich & Villani (2000).
B.3.2 POLYNOMIAL DECAY WHEN q e AND V DEGENERATELY CONVEX AT INFINITY
We consider a potential V E W2.00 such that J = and degenerately convex at infinity, i.e.
U(u) S (u) < (uu + 6
(33)
where a, b are nonnegative constants and U is convex degenerate, i.c.
D²U(w) 2 c(1 + /ul)a-2 <> 0 and B E (0,2)
(34)
Without loss of generality we assume that U takes its unique minimum We further assume that for some b, Co >
(u( clu Co typical potential that satisfies (3). 34 and 335 is V = |x] with 0 B < Before we state the next theorem in Toscani & Villani 2000), we further define the following quantities
(35)
2
Mp)p)
|x( (2x
2 and 8 2(2
E0.,
(36)
3)+ (s
Theorem B.4. Let V be a potential satisfying assumptions 33
34 and (35). Let po be a proba-
bility density such that DKL(Pullg) < s0, M.(Po) < given in 36 for 2. Let also {pr hpch be a (smooth) solution of the Fokker-Planck equation with potential V and with initial datum po- Then, there is constant C depending on Drt.(pollg) M.(P)). and such that for all 1. 0.
C (x
1- 25
-
DKL(Pelle (V)
with K
(37)
8 2 3
where 5 given (36).
Note that as 3 + 2, one recovers the usual logarithmic Sobolev incquality as discussed in Sect. B.3.T We summarize the said examples in the following table.
18

---

Under review as a conference paper at ICLR 2023
Table 2: Rate convergence to equilibrium state q x e in KL divergence
Examples of g x xe-
Rate of convergence in KL divergence
9 g 3 323
at least e -2c at least - with À
Special case: (0, σ)
g x ex <<
O(1 H. as (3
B.4 WEIGHTED POROUS MEDIUM EQUATIONS AND THEIR CONVERGENCE TO EQUILIBRIUM STATE
B.4.1
WEIGHTED POROUS MEDIUM EQUATION
The gradient flow of f-divergences for likelihood ratio. One may rewrite (29 in terms of the likeli- hood ratio denoted by h, and defined as
dp
h4
(38)
dg
By using the operator identity (q being the multiplication operator by the function q). i.c.
Vy = vV + V log4)
we have that
Pt
< V Vl
9
and thus we can rewrite
29 as
(x), (V + Vlo h4 V (h+)
(39)
Moreover if we denote 4 the adjoint of V on (g) we have V = -VV + Vlog g) and thus (39)
has the form
0che(x) = V
(40)
Let now (1) x²-1 a) we rewrite
AVha-1= =
3 Vo = -== h0-1 and h = e-- B a
and thus we obtain
0zhr(z)= = (A V log q 4)
(41)
for t 2 and x E Rd corresponding to non-negative initial condition h(x, 0) = ho (I), called weighted Porous Medium equation. For existence and uniqueness, see Dolbeault 20088
Rd
Remark B.5. The formula for fa is given by
(a-1)
y 1y30 + a(a-1)
a > 1
(v)
(42)
00 o01 y20 +
y>0
al-aa
1 1950
a (0,1)
a(1-a)
Remark B.6. For completeness, we discuss a related gradient flow known as granular media equa- tion. To be precise, the 2- Wasserstein gradient flow of F(p) MMDD(p, ql² where MMD[p. ql is the Maximum mean discrepancy (MMD) Gretton et al. (201 2). By recalling 2 MMD is defined as
MMD(p, ql
sup (0.1
{Eq g] - Eplg]}
and its maximizer (²)
(2) k( 2))(()))( (z) - k(x, 2)p(x)dr k p()) ((2)
is called witness function between the probability densities and p. In fact, g" is the difference between the mean embeddings of q and p which finally makes MMD be re-written as the RKHS norm of the unnormalized g", i.e.
MMD/p, q] = 1100 1|
(43)
Then the gradient flow equation associated to F leads to the granular media equation, i.c
@pp((x) diu (pV (I: * p k q)) = díu (p. V
(44)
19

---

Under review as a conference paper at ICLR 2023
B.4.2 FUNCTIONAL INEQUALITIES FOR THE WEIGHTED POROUS MEDIUM EQUATION
In this section, we apply Theorem 4.5 in Dolbeault (0088) Weighted Porous Medium for the likelihood ratio h1 E and we prove polynomial decay in KL and x²-divergence. Before we state the result we first define the L -Poonccr inequality and L" -logarithmic Sobolev inequality (see also Dolbeault et (2088).
Definition B.7. Let be probability measure Riemannian manifold (M, g). Then the entropy defined as follows: for any smooth function f E (M)
Ent = / fog (Tran) dg
(45)
while
Var,())= f
fdy
dq
(46)
Definition B.8. Let be a probability measure on Riemannian manifold (M, Let also be a positive measure on (M,g) g). We assume that q E (0, 1]. We say that (g. v) satisfies I PPinnare inequality with constant Cp if and only if, for any nonnegative function f E C² (M)
[Var, (Prr) s SCp /IVJPa
(47)
We say that (g, v) satisfies I If"-logrithmmic Sobolev inequality with constant CLS if and only if, for any nonnegative function f E (M)
Ent, (r S SLs
/
(48)
Theorem B.9. If (q. q) satisfies L2/3_ Poincaré Sobolev inequality, for some constant Cp > 0, then for any non-negative initial condition ho En E L²),), we have for every t 2
(pell) & (1x2000110) 1/2
9 Cpt)
8
(49)
Reciprocally, if the above inequality is satisfied for any 9o. then (q. 9) satisfies T2/3 -Piincre Sobolev inequality with constant Cp > O.
Theorem B.10. Let > 1. If (g. 9) satisfies T.A/--lgarii Sobolev inequality, for some constant OLs > 0, then for any non-negative initial condition ho such that (KL(Pollg) < co, we have for every 2 0
DKL(Pell)) & (PDLL(oOll)) + 4(xx 4(xx-1 ((x=-1) -1ssttt
11((a-1)
Clst
(50)
Reciprocally, the above inequality is satisfied for any go. then (g. 9) satisfies a I, I.
Sobolev inequality with constant Cis > 0.
Next we discuss two examples of probability distributions satisfy I'- P'-Poina inequality and L"- logarithmic Sobolev inequality:
Let E (0, and B E 1). 1). The probability measure
dq
1=1 dz, EED R
(51)
2r 1+ 8
satisfies L J'--Poincaré inequality and L L'-logarithmic Sobolev inequality. Let E [1/2,1), 1), then for dq = (1 + (2) 1+ dx, I
2r T-p the probability measure
8
E IR
(52)
satisfies I."-Poincaré inequality and I logarihhiii Sobolev inequality.
20

---

Under review as a conference paper at ICLR 2023
Table
Rate of convergence to equilibrium state & x e in x diveergee
Examples g x e
Rate of convergence X divergence
1
0<rsl, <1
given in (9)
(1+
q = (1- [+) 1/25r<1, 82 1-r
given (9)
C FIRST VARIATION OF REGULARIZED DIVERGENCES
In this section we prove the following theorem
Theorem C.1. Assume superlinear and strictly convex and P,Q E (R()..
1. For æ & supp(P) n supp(Q) define (1) suPzesupp(() (x)+*(x) L/x y/} then
Lbn is Lipschitz continuous on TR...
2. dh. sup(h( L, h(y) (y)* every E supp(Q)}
3. Let p be signed measure of total mass O and let p+ - p- where p+ E Pi(K) are mutually singular. If P cp E Pi(K) for sufficiently small [c] then D cpl(Q)
differentiable at e and we have
(DF
DJ (P(()) = @hitdp.
lim -00
other words we have
(PIII) (P) = =h
5P
Proof. The proof of is straightforward by using the triangular incquality of norms. For 2., since h E LT, we have that h(x) < h(y) + ||z - yll. This implies that for supp(Q) and x & supp(Q), h(x) S inf (h(y) + I y|]} inf " (v) ||x yll -h. "z"*(2). Since (1)(1) E l', this concludes the proof. For 3., we use the variational formula epllQ) where we suppose that P + ep E Pi (I
for (P +
DJC + (pl())
Epteplø]
- inf {v + (ol HER
= sup SEt'.
2 / 1zt d( d(P+tpp))) +c))
{ 4 (
inf
wER
c 0/ DD. (P1(()
(53)
Thus
lim inf (P+ D)- > / 0r +-
For the other direction: Let us define F'(e) (P <pllQ). By Theorem 18 and 71 in Birrell et al. (2022a). F(c) is convex, lower semicontinuous and finite on [0. eo). Due to the convexity of F. F is differentiable on (0, eo) except for a countable number points. Let E (0, eo) such that F is differentiable and > small. Also, let o- I be the optimizer of DJ. (P + <pllQ) satisfying
.. ( 0 so that
inf 4 / (o(oti--)ddd }
DJ (P DP =ppl()))=) = o. d(p +p)
wER
By using the same argument as before in the proof, we have that
-P00-=00+0000)0))))(+++000001110 26 do 000'0)0000'000
(54)
21

---

Under review as a conference paper at ICLR 2023
and
-8 - dp
(55)
which gives us that
0. dp
lim 8-00 d
DJ (P <pll4))
(c)
lim
(P) (0 (0))))))22
8-0
/
0.
(56)
Consequently,
Fc(c 07
(57)
Let FL (0) lim \ (F(c) - F(0)). By convexity, for any sequence (GnINEN such that En + 0, we have
>
F (0) lim F'(ex) 2=000
lim n-
/
dr
By applying the Arzelá-Ascoli to o,,4, and then doing diagonalization argument, there exists
a subsequence of (na)220 {n)n20, such that &: converges pointwise to function o5 Lip (TRd). For simplicity, from now on we denote n the convergent subsequence.
At this point, we recall that for any E (0, (0) = 0. For any (x) 6,-( -
L||alla which implies that
(x)) S T||
Thus by the dominated convergence theorem
/ dp- dgdp n2+00
F4(0)
lim
By the lower semicontinuity of DJ.t (-11Q), we have
D (PI())
lim DJ (P (xpl()
n=000
inf
Eptcno - inf {u+ E0ls" -1]} DER
400
lim Ep Fpp 1 lim inf {u+ Foll - -v)}} n VER
a0000
Eplo - inf {
V Eqlf" wi]}
WER
(PI())
(58)
where for the second inequality we use the dominated convergence theorem, 57 and that by Fatou's lemma
/ " dQ 2
/ 10000
lim sup 22000
d Se dQ
lim inf n-000
Since both sides of the inequality coincide, 00 must be the optimizer. By Theorem 3.1 part 1. and
Theorem C.1 part 2., have that "
(xx) I rh for all x. Thus
(0) / ordp < f 0d
which concludes the proof.
22

---

Under review as a conference paper at ICLR 2023
D DETAILS ON LATENT GENERATIVE PARTICLES
D.1 GENERALIZATION OF ENCODER AND DECODER FUNCTIONS
In the Section 5 we built Theorem 5.1 based on an encoder map & + Z (for instance y 12
Z Rd d' << d) and decoder map D Z + y. For the generalization purpose, identify these mappings as dirac kemels Kg (B, dz) = de(u) (dz) and Kp (b(=)(dy))
1. A pullback function induced by the kernel Kg given
Kolfl()) = - (()))
(59)
for f E MM(3),,E) Ey
2. A push-forward measure induced by the kernel Ke maps P(Y) - P(Z) and defines a push-forward measure which is given
Ke(PI(B) f /Kelv. () P Po
(60)
for P E P(Y) and Z-measurable set B.
Likewise, the kernel Ka induces pullback function and push-forward measure in the opposite
direction. In the previous formulation, the Q³ -perfect encoding property E Q² = Q" can be rewritten as Q' = Kp [g[Q*)
[[ Given any p> E 'P()), we have the latent probability mea- sure p2 Kg [ E P(z) and the reconstructed probability measure py KD[Ke [PJ]] P(D(2)) where D(3) y Rd. In general, PV + p>
Transition probability kernels are defined in the form of conditional distributions: Kp (2) = p(dzly) from 2 and K, dy) g(dyl=) from to Y. The kernel-induced pullback functions y) or [=]g]() f ( [2) =
kyl[(()) = [==')p(dzz)) (u) (2
= 2] are interpreted as conditional expectations. In addition, the ker- nels induce push forward measures p2 (dz) p(dzly) (ddy) for pv P(RR) (dyy)
Ey (=11====((do)))0(()))
x(dy)=)R2 (R=((dd) (dy) for R² P(RR)). For the Q-perfect encoding property, require these kernels to satisfy dQ (dy) a(dy))))(()) y)dQ (dy).
D.2 DATA PROCESSING INEQUALITY AND PROOF OF THEOREM 5.1
The proof of Theorem 5.1 a consequence of a, new, tighter data processing inequality derived in Birrell et al. (202a)) that involves both transformations of probabilities and discriminator spaces Г. Theorem D.1 (Data processing inequality for (f.I) -divergences). Given a real valued con- vex function f. P.O. Q E P()),, and probability kernel K from (2, to (N, if C N is nonempty, then
(K([]]IK(()) < DJ (PII))
(61)
Proof. From the variational formulation of divergences, we have
()( sup
((() dy) ((.))))))) /
u) () Q())
sEr.EER
(62)
Since f is convex, Jensen's inequality gives
/ ((((o)) u)) f (f(olo) -)) v)K((, do)
(63)
for all x E Q. Hence. D(((PP]]((Q))
sup
Ep[K[ø] - vi - Eolf"(K(o) - -v] = (PII))
(64)
ΦETwER
23

---

Under review as a conference paper at ICLR 2023
Now state and prove the generalized version of the Theorem 5.1
Theorem D.2. Suppose that
1. Perfect encoding. For Q" the encoder & and decoder are such that KolKe(Q"1] Q²
2. [T] Czz real function space.
The pullback functions induced by the decoder kernel is included in the
Then, for any p2 P(Rx") we have
DJ (Kol_"1111))) = (P=I|KE(0"))
(65)
Proof. Since the encoder & and the decoder D perfectly reconstruct Q?'
DJ (K=( DJ (K (K (1)
From data processing inequality,
DIP [K < DD (p2
By the assumption that KD CTz,
KolT>> (22
D.3 LATENT GENERATIVE PARTICLES ALGORITHM
Algorithm 2: Latent Lipschitz regularized generative particles algorithm
Require:
defined in (2) and its Legendre conjugate f", Lipschitz constant, scalar parameter for optimizing f divergence, T: number of updates for the particles, ? time step size, N: number of particles
Require: W ) parameters for the neural network á Rdd TR, D: depth of the neural network, 5: learning rate of the neural network, TNN: number of updates for the neural network.
Require: E Rdd + IR''; trained encoder, D IRd
Rd: trained decoder.
Result: {Yg ( Ji=1
Sample (X = E(X()) E RR IN where (x() } ~ Q is a batch from the real data
Sample (Y, (4) = E(r()) E IRR N where (Y ~ Po P is a batch of prior samples
Apply Lipschitz regularized generative particles algorithm on (1) and Y, (4)
Reconstruct y,( D(Y (3)
E EXPERIMENTAL SETTING
Neural network architectures. We use the discriminator á (compared to GAN setting) which is implemented using neural network. In Table we provide the architecture of the neural networks used to produce the experimental results. The Lipschitz constraint on Φ is implemented by spectral normalization (the weight matrix in each layer of the D layers in total has spectral norm ||W||2 = T,1PP) which interpreted as a hard constraint. Imposing a gradient penalty term the loss (soft constraint) tried, but there were some problems: required additional tuning for the initial weight scales, and did not successfully restrict particle speeds bounded by L. Therefore we keep imposing the hard constraint in the entire paper.
Data sets and important parameters. See Table material README.md.
More details can be found in Supplementary
24

---

Under review as a conference paper at ICLR 2023
max BG, oct
Yn ~
YN ~ PN
Ga. (2)
(2, ()))
(2)
(2, Ps
Zy
max Eeppoo o Eesoy (+) gell
2ui1
(2,)
0o
0₂
On
0N
argmin max (2) 0- Eqf
Z
Z
(a) GPA in latent space
(b) GAN
Figure 4: Workflow of different generative models. green: real space, yellow: latent space, blue: parameter space
CNN Discriminator
5 x Conv SN, 2 x 2 stride (1 chi)
leaky ReI.U
FNN Discriminator
5 Conv SN, 2 stride (ch1 - ch22)
W E daxt with SN, E R
leaky ReLU
ReLU
x 5 Conv SN, 2 stride (ch2 - cluz)
W E B with SN, D2 E R
leaky ReLU
ReLU
Flatten with dimension E3
W R with SN, b R ReLU
W E Rea with SN, E Rd
ReLU 5 R with SN, 6 E R Linear
W E
R R with SN, 84 E R
Linear
(b) Low dimensional data with dimension d
Image data (MNIST, CIFARIO)
Table 4: Neural network architectures of the discriminator & R IR
Computational resources. Low dimensional examples are computed in the tensorflow-CPU en- vironment: tensorflow-gpu-s with CPU model Intel(R) Core(TM) 15-10210U CPU 1.60GHz 2.11 GHz. Image generation examples are computed in the tensorflow-GPU environment: tensorffow-ggu=22..00 with 1-GPU model Tesla K80 in Google cloud platform.
F ADDITIONAL EXPERIMENTS
F.1 COMPARISON WITH OTHER GENERATIVE DYNAMICS
We compare the GPA and other generative dynamics such as RKsed-based methods (Figure 5) and score-based methods by examples of 2D Mixture of Gaussians (Figure 6
F.2 ADDITIONAL LATENT GENERATIVE PARTICLES EXAMPLE
We applied (fKL: (1) generative particle algorithm in the latent space and reconstructed to the high dimensional image data. In the high dimensional space, first sampled initial particles from the logistic distribution and target data in [0, 1] 28222 for MNIST and in [0, 1|32x32x3 for CIFAR1O. For each of MNIST and CIFAR10, autoencoder with 128d latent dimension were trained. Then GPA was done in the 128d latent spaces. The number of training samples are N 200, 2000.
F.3 MICROARRAY GENE EXPRESSION DATA
The flcxibility on the choice source distributions and the small sample size regime enable our generative particles algorithm be used for medical data-procsssing purpose. In addition, using
25

---

Under review as a conference paper at ICLR 2023
data parameter
NN structure
learning rate
Dataset
f
At
No
200,
MNIST
KL
conditioned
CNN (128, 128, 128)
0.001
0.5
2K
d = 2, 5, 10
FFN (32, 32, 32)
Gene data
d 20, 50,
KL
FFN (64, 64, 64)
0.1
5.0
245
100
d = 200
FFN (128, 128, 128)
2D
Student-t Mixture
a 2
0.5
FFN (32, 32, 32)
0.005
1.0
500
conditioned
200,
MNIST
KL
FNN (256, 512, 256)
0.001
0.5
d' 128
2K
conditioned
200.
CIFARIO
KL
FNN (256, 512, 256)
0.001
d' 128
2K
2D
I, 10, 100,
Gaussian
KL
σQ = 0.5
FFN (32, 32, 32)
0.005
1.0
200
Mixture
None
2D
Gaussian
KL
σQ 1.0
FFN (32, 32, 32)
0.005
0.5
500
Mixture
Table 5: Data sets and important parameters
Sample size
200
2000
Sample size
200
2000
(JKL, Tn -PGA
4571.98
5143.55
MNIST
1107.57
1048.87
88.4
(fKL. D1 -AGN
5603.55
1270.13
CIFARTO
87.34
75.10
Wasserstein-GAN
5653.20
1879.18
(b) Final FID of MNIST and CIFARIO (latent GPA).
(a) Final FID MNIST: GPA and GANs.
Table 6: Conditional image generation performance summary. See Figure
latent generative particles scheme, we can effectively handle high-dimensional data, such as gene expression data, typically in the dimension of 10 depending the probes in signif- icantly reduced dimensions. We suggest batch normalization/ddat merging as an application of our algorithm.
Dataset. We tested on publicly available gene expression data sets from Gene Expression Omnibus (https: :/ h/www//wwt//////////////////// nobi. lammmmmhhh gov/geo/)
Breast cancer: Accession number GSE47109 (206 samples), GSE10843 (245 samples)
The former forms the source dataset, and the latter forms the target dataset. The source and the target data lie in the same dimensional space 544667
Auto-encoder. We applied PCA on the combined matrix for the source data and the target data which have been firstly normalized to mean zero and variance one. The normalized PCA can be interpreted as a linear auto-encoder. The PCA decoder is Lipschitz continuous with = vdd.
(Let y dd' E V.. Ed' z\V; Rdd. The decoder D(z) Σ 2,V, satisfies I|D(z)
Dl ||| = 11 (zs v/)vvll < Z z' E 12 = Vd'lz z' by Cauchy-Schware inequality.
Outputs See Figures 8 for the transported particles the latent space for varying d' = 2, 5, 10, 20, 50, 100, 200 and the reconstructed space.
26

---

Under review as a conference paper at ICLR 2023
CMO
OMQ
12
0850
(a) Trajectory of (fKL, -geenerrratvee particle algorithm
(b) Divergences of (fKa, PL)- generative particle algorithm
(d) Trajectory of KL ULA Durmus & Moulines
(c) Trajectory of KALE flow Glaser et regularizations)
(2021 (Different 2017). MMD flow Arbel et al. (2019
Figure 5: (2D Mixture of Gaussians 1) Comparison with RKHS based generative dynamics. (a) (fKL, -')geeerrrte particles algorithm with different values for L. The particles are transported to the 4 wells faster as gets larger, however for large the algorithm become unstable (L > 100). Learning rates are chosen as y 1.0, 8 = 0.005. (c) KALE flow can be compared with (fKL. Iz)- generative particles algorithm in the sense of being different regularization technique. The KALE gradient flow regularizes the RKHS norm of o² while (/KI. L.))ggenrratvve particles algorithm reg- ularizes the norm of VØ" The KALE flow Glaser al. (2021) fails to capture the wells reasonable amount of time. Here Gaussian kernel with
0.5 is chosen for the RKHS kernel. Learning rate is chosen as 0.001. (d) Bottom: MMD Arbel et al. (2019) gradient flow (without extra noise). A Gaussian kernel with gradient flow trained with the unadjusted Langevin algorithm (ULA) Durmus & Moulines (2017). The comparison with KL ULA and MMD flow suggests that the of regularization enables the convergence without further techniques such as adding noise.
0.5 is used for the RKHS. Top: For comparison KI.
27

---

Under review as a conference paper at ICLR 2023
20
30
(a) KL Lipschitz-regularized GPA
(b) Score matching and annealed Langevin dynamics
Figure 6: (2D Mixture of Gaussians) (SKL. D G-PA and the score based model (Noise con- ditional score network, NCSN). 200 target samples from Mixture of Gaussians with øq 1.0 are provided to transport 500 particles which are uniformly distributed in the plotted region at time t O. Blue: target, Orange: output. (a) The choice of divergence fKL and propagation of parti- cles through the (fKı,, J-G-PA captures the target. tractable using NCSN, Compare with a heavy-tailed target example in Figure
(b) shows learning mixture of Gaussians is
O
0
0
0
0
O 1 2 3
O /
0
0
0
0
0
0
O
/
/
1
1
1
1 2
1
1
1
0
1
2
3
2
2 3 4
2
2
2
A
7
2
2.
5
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
7
4
4 5 6
4
4
1
4
4
4
4
4
4
5
5
5
5
6
5
5
5
5
5
5
5
5
5
5
5
5 6
6
5
6
6
6
6
6
6
6
6
6
6
6
6
6
7
7
7
7 8
7 8
7 8
7
7
P
/
8
8
8
P
8
8
8
2
8
4
9
5
7
9 9
9
9
9
7
(a) MNIST (fKL, P1) L-GPA, 200 samples, 128d(b) MNIST (fkL, r1) L-GPA, 2000 samples, 128d
latent dimension
latent dimension
(c) CIFAR10 (fKL, P1) L-GPA, 200 (amples,(d) CIFAR10 (fKL, [1) L-GPA, 2000 samples, 128d latent dimension
128d latent dimension
Figure 7: GPA in the 128d latent space. 200 samples are generated from 200, 2000 training data. See FID values in table 6
28

---

Under review as a conference paper at ICLR 2023
soure lirget
HOUCE Earget
(a) dim
(b) dim
12
12-
14
(c) dim
(d) dim 20
12-
(e) dim 50
(I) dim 100
JMP
30-
MMD(source, recon) MMD(target. recon)
351
(g) dim 200
(h) MMD
Figure 8: (Gene expression data, BreastCancer) Latent samples. blue: souree, red: target,
black: transported. (h) The distance between the latent distributions. blue: MMD(PF . red: MMD(Q2 ( PZ), black: MMD 22 Q²) with T = 25, 000.
29

---

Under review as a conference paper at ICLR 2023
Aeppu sagat
11.
(a) dim
(b) dim 5
15
MOFCE target
34
11-
10-
12-
(c) dim
(d) dim 20
13
13
11
12-
Wage
(e) dim
(f) dim 100
32'
MMD(source, recon)
MMD(target. recon)
351
12-
337
12"4
(g) dim 200
(h) MMD
Figure 9: (Gene expression data, BreastCancer) Reconstructed samples. blue: source, red: target, black: transported. (h) The distance between the reconstructed distributions. blue:
MMD P P?), red: MMD(Q", PP)), black: MMD(P)) Q²) with T = 25, 000.
30


Reviews:
[{"review_id": "mTz7V-Pbdy", "reviewer": "Reviewer_VqE2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers a new class of f-divergences that incorporate Lipschitz continuous functions in the variational representation of the f-divergence. These new class of divergences interpolate between the 1-Wasserstein metric and f-divergence when the Lipschitz parameter of the function class varies from 0 to infinity. This class also allows one to define a gradient flow and transportation equation from one distribution to another. Using this theory, the authors introduce a scheme to learn a generative model of a data distribution by learning a particle flow in the latent space of an auto encoder. This structure leverages the fact that many datasets of interest have intrinsic low-dimensionality. Experimental results are showcased on small synthetic and toy datasets such as MNIST.", "summary_of_the_review": "Overall, I think that there are interesting mathematical ideas presented in this work, and the experimental results showcase some potential in the applications of such ideas as well. However, there are certain points about the core contributions on the theoretical side and puzzling properties about the model that I would like the authors to comment on before increasing my score.", "strength_and_weaknesses": "**Strengths:**\n- The proposed methodology of the paper is mathematically grounded, with some nice results on the approximation side showcasing that the error between the two distributions can be quantified.\n- The method appears to perform at its best, and fairly well in low-data regimes.\n\n**Weaknesses:**\n- The motivation for the approach could be strengthened. In particular, it is unclear what the benefits are in terms of utilizing this Lipschitz regularized formulation versus the original variational formulation over continuous and bounded functions. More motivation and discussion along these lines would improve the readability of the paper.\n- Some experimental results show that the generative capabilities do not improve with more data. For example, the MNIST generation results do not appear to improve with more samples both visually and quantitatively (as given by the FID score).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bvjxSK4cFo4", "reviewer": "Reviewer_hEpA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses Lipschitz regularization (in the context of the dual for f-divergences) to construct gradient flows for sampling from target distributions. These distributions can be empirical (i.e. discrete), and the construction can be applied to build generative models. Experiments show improved performance for generative modeling in low-data regimes.", "summary_of_the_review": "Overall, I lean (slightly) to accept the paper. This is mostly due to the theoretical niceness of the paper (including it's many connections with gradient flows, f-divergences, and PDEs). What's stopping me from being fully supportive is the experimental section, which includes relatively mixed results on small toy datasets. However, given the theoretical nature of the work, I do believe that the technical developments outweigh experimental shortcomings.\n\nI would also ask the authors to include a section on diffusion models and the connections therein if possible.\n\nUpdate\n--------\nSeveral of my questions have gone unanswered, which is rather concerning. Furthermore, in the rebuttal, the authors referenced \"engineering issues\" or \"full optimization of algorithms\" when talking about toy results (MNIST) or algorithmic running time. Given that there still seems to be a lot of necessary changes to be made, I believe it is better to revise and resubmit the paper and have thus changed my score to leaning reject.", "strength_and_weaknesses": "Strengths\n-----------\n* The method is very nicely developed with a good interplay between f divergences, lipschitz regularization, and gradient flows. By using these three perspectives, it is able to introduce a unified perspective and build a real technical model.\n* The presentation is very clear and is able to present the mathematically technical details without it being overly cumbersome. In particular, I appreciated the usage of notation.\n* Additionally, the overall mathematical construction does not seem to have any flaws.\n\nWeaknesses\n---------------\n* The experiments are somewhat weak. First, the experiments only show results for MNIST and Gene Expression (student-t seems more like a proof of concept than an experiment). These datasets are pretty toy (MNIST is rather low dimensional and easy to solve with simpler methods, and the Gene Expression dataset was just introduced to show merging).\n* Furthermore, for MNIST in particular, it is very disconcerting that, while the method does perform well in the low-data regime, increasing data from 200->2000 samples doesn't improve the method at all (either visually or numerically). This seems like a potentially big drawback for the purported applications. In particular, if the comparison is on the low data regime only, the authors should compare with other methods that also look at this.\n* The MNIST results also should be compared with modern methods. In particular, Wasserstein GANs are around 4 years old, and modern methods such as diffusion.\n* There should be a connection with continuous diffusion models (in particular diffusion also has a similar connection with PDEs like the Fokker-Planck Equation) that the authors should elaborate on and include.\n* It seems like the method might be pretty computationally expensive (e.g. Table 2 (b)). Could the authors comment more on the training time, as GANs are reasonably fast to train compared with most differential equation methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UQvbJFe6r4", "reviewer": "Reviewer_Ds4C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies Wasserstein gradient flows for a certain class of objective functionals. These functionals form are an approximation of the f-divergence defined using its variational form. In particular, the optimization domain in the variational formulation is restricted to the class of functions with bounded Lipschitz constants. As the bound on the Lipschitz constant grows, these approximations become exact. The gradient flow is constructed with the objective of generating new samples from a target distribution (similar to the objective in GAN). Another presented idea is to consider generating the gradient flows in the latent space with an encoded-decoder mechanism in order to take advantage of the low-dimensional structure present in the target distribution.   ", "summary_of_the_review": "The paper is well written in general and contain original ideas. However, it needs more motivation for considering Lipschitz constrained class. Also, it discusses two very nice and independent ideas in one single paper. Gradient flows in the latent space is independent of Lipschitz constrained gradient flows. And it is not clear that which of these two ideas are responsible for the improved numerical results compared to GAN. \n\n####\nI thank the authors for their response. But I am still not convinced about the motivation for considering Lipschitz constrained since most gradient flows are Lipschitz under mild assumptions on the initial and target density. After reading other reviews I am changing the score to 5.  ", "strength_and_weaknesses": "Strength: \n- The idea of considering gradient flows for Lipschitz constrained f-divergences is new to me\n- Also, considering the gradient flows in the latent space seems interesting and valuable\n- Numerical experiments are complete and informative\n- the comparison Table 2 is very nice, though I wished to see more discussion. \n\nWeakness: \n- Not enough motivation for constraining in terms of Lipschitz functions, since neural networks are not a good representative of all Lipschitz functions with a certain Lipschitz constant. Why not gradient-norm penalty?\n- The claim that Lipschitz constant makes the particle system \"stable\" should be made more precise. Stability in what sense? \n- The infinite-speed of diffusions is true, but diffusion is not the only way to implement Fokker-Plank. It can also be implemented deterministically using \\nabla \\log (density) which is Lipschitz under standard assumptions on density. \n- Not in depth discussion of the numerical results, conclusions, and limitations.  \n\n\n   ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "I44lwUnuLH", "reviewer": "Reviewer_VduJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a particle flow algorithm that approximates a target empirical distribution with particles. The suggested algorithm updates the particles using the Wasserstein gradient of Lipschitz-regularized $f$-divergences, which is shown to exhibit a variational formulation as a supremum over Lipschitz continuous functions. An additional autoencoder can be used to flow the particles in a latent space. Various experiments are done to demonstrate the performance of the proposed method compared to GANs. \n", "summary_of_the_review": "Overall I don't think the paper has enough novelty or has justified its application prospect well enough. The writing could also use a lot of improvement. As a result, I'm leaning toward rejection.", "strength_and_weaknesses": "## Strengths:\n* The idea of facilitating Wasserstein gradient flow by optimizing a learnable neural network from which the Wasserstein gradient can be obtained is a very interesting one.\n* The proposed algorithm is simple and works for all kinds of $f$-divergences.\n\n## Weaknesses:\n* I found the novelty of the present paper limited. To me, the main novelty is (6), although I think the derivation is more or less identical to that of Dupuis & Mao (2022) from the KL case.\n* The application prospect of the proposed algorithm is questionable. Without using the autoencoder, to me, the proposed algorithm is just to subsample the target empirical distribution. I think there is a lot to be explored here. For instance, if we use a very small number of particles (compared to the number of samples in the target distribution), can we still represent the target distribution faithfully? If so can we use the proposed method as an alternative to k-means?\n* I don't think comparing with GAN is reasonable. GAN is capable of generating endless streams of new samples, whereas for the proposed method you must fix the number of particles ahead of time (it is also unclear to me how many particles were used in the experiments). A good baseline to compare would be, say minimizing the Wasserstein-1 (or W2) distance between particles and the target distribution and comparing the resulting particles. As a result, I do not find the experimental results convincing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject