The Score-Difference Flow for Implicit Generative Modeling
Authors: Romann M. Weber
Venue: Accepted by TMLR ()

Abstract: Implicit generative modeling (IGM) aims to produce samples of synthetic data matching the characteristics of a target data distribution.  Recent work (e.g. score-matching networks, diffusion models) has approached the IGM problem from the perspective of pushing synthetic source data toward the target distribution via dynamical perturbations or flows in the ambient space.  In this direction, we present the score difference (SD) between arbitrary target and source distributions as a flow that optimally reduces the Kullback-Leibler divergence between them while also solving the Schrödinger bridge problem.  We apply the SD flow to convenient proxy distributions, which are aligned if and only if the original distributions are aligned.  We demonstrate the formal equivalence of this formulation to denoising diffusion models under certain conditions.  We also show that the training of generative adversarial networks includes a hidden data-optimization sub-problem, which induces the SD flow under certain choices of loss function when the discriminator is optimal. As a result, the SD flow provides a theoretical link between model classes that individually address the three challenges of the "generative modeling trilemma"—high sample quality, mode coverage, and fast sampling—thereby setting the stage for a unified approach.

Published in Transactions on Machine Learning Research (07/2023)
The Score-Difference Flow for Implicit Generative Modeling
Romann M. Weber
Disney Research:
Studios
Reviewed on OpenReview: htips: // openreview. net/ form?? id= dpCSNLUCzu
Abstract
Implicit generative modeling (IGM) aims to produce samples of syuthetic data matching the characteristies of larget data distribution. Recent work (e.g. score-malching networks, dif- fusion models) has approached the 1GM problem from the perspective of pushing synthetic source data toward the target distribution via dynamical perturbations or flows in the am- bient space. In this direction, we present the score difference (SD) between arbitrary target and source distributions as a flow that optimally reduces the Kullback-Leiblee divergence between them while also solving the Schrödinger bridge problem. We apply the SD flow to convenient proxy distributions, which are aligned if and only if the original distributions are aligned. We demonstrate the formal equivalence of this formulation to denoising diffusion models under certain conditions. We also show that the training of generative adversarial networks includes a hidden data-oplimization sub-problem, which induces the SD flow un- der certain choices of loss function when the discriminator optimal. As a result, the SD flow provides a theoretical link between model classes that individually address the three challenges of the generalive modeling brilemma high sample quality, mode coverage, and fast sampling -enpeengthhereby setting the stage for unified approach.
1 Introduction
The goal of implicit generative modeling (IGM) is to create synthetic data samples that are indistinguishable from those drawn from target distribution. variety of approaches exist in the literature that address this problem from the perspective of the dynomics imposed upon the synthetic data during sampling training. In particle-pplimization sampling methods such as Langevin dynamics (Bussi & Parrinello, 2007 Welling & Teh 2011p, Hamiltonian Monte Carlo (MacKay) 2003), and Stein variational gradient descent (Liu & Wang 2016), synthetic particles are drawn from a source distribution and perturbed over a number of steps until they resemble particles drawn from the target distribution. Separately, parametric models have been developed that either perform these perturbations implicitly under the hood, as in the case of normalizing flows (Rezende & Mohamed] 2015 Papamakarios et al.) 2021), or do so explicitly during inference, as in the case of diffusion models (Ho et al. 2020 Nichol & Dhariwal 2021
This would seem to contrast with the training of parametric models for single-step generation by learning a mapping between input drawn from a noise distribution and synthetic output resembling the target data. Perhaps the most famous example of such a model is a generative adversarial network (GAN) (Goodfellow et al. 2014), which leverages a separately trained discriminator to guide the generator toward more target- like output. However, we will show that GAN training contains a hidden sub-problem that induces flow on the generated data that is completely determined by the loss being optimized. Consequently, this and various other IGM methods can be understood in terms of the dynamics imposed upon the synthetic data.
The question then becomes one of asking what the optimal dynamics might be for the IGM task. In this direction, we present the score difference (SD) the difference in the gradients of the log-densities of the target and source data distributions with respect to the data as the flow direction that optimally reduces the KI. divergence between them at each step. We then argue that we can sidestep directly working with the source and target distributions in favor of operating on convenient proxy distributions with common support, which we show are aligned if and only if the original source and target distributions are aligned.

---

Published in Transactions on Machine Learning Research (07 (07/2023)
We derive the score difference from the analysis of the dynamical systems that govern probability flow. But we also show that the score difference is hidden within or relates to various other IGM approaches, most notably denoising diffusion models and GANs, under certain conditions. We also outline a flexible algorithmic approach for leveraging SD flow when working with any source and target distributions, with no restrictions placed on either distribution.
Our aim provide an accessible treatment of complex topic with important connections various areas of generative modeling and variational inference. The paper is organized around its key contributions as follows:
1. In Section 2 derive the score-difference (SD) flow from the study of probability flow dynamical systems and show that SD flow optimally reduces the Kullback-Leibler (KL) divergence between the source and target distributions and solves the Schrödinger bridge problem.
In Soction 3 we consider modified proxy distributions for the source and target distributions, which have common support and are generally easier manage and estimate than the unmodified distri- butions. We outline method for aligning these proxies and show that this alignment occurs if and only if the unmodified distributions are aligned.
3. In Section we draw a connection between SD flow and denoising diffusion models and show that they are equivalent under certain conditions. However, unlike diffusion models, SD flow places no restrictions on the prior distribution.
4. In Section 5 show that GAN generator training is composed of two sub-problems, particle- oplimization step that induces flow determined by the loss being optimized and a model- optimization step, in which the flow-peruurbed particles are fit by the generator via regression. We then show that the SD flow is induced in GANs in the particle-optimizatoon step under certain conditions and choices of loss function.
In Section 6 we present flexible algorithms for applying SD flow to both direct sampling (particle optimization) and parametric generator training (model optimization).
6. In Section we report experiments on our kernel-based implementation of SD flow, including comparisons to maximum mean discrepancy (MMD) gradient flow and Stein variational gradicnt descent (SVGD) under a variety of conditions.
We conclude in Section S
In the appendices we provide supplemental information, discuss theoretical links MMD gradient flow and SVGD, and report additional experimental results.
2 Probability Flow and the Score Difference
2.1 Derivation from Stochastic Differential Equations
Consider data æ E IRd drawn from base distribution 4o- We can describe a dynamical system that perturbs the data and evolves its distribution qu - 9t over time by the stochastic differential equation
dæ ju(x,t)d + (t)dw,
(1)
where 1 x TR Rd is drift coefficient, (t) is diffusion coefficient, and dw denotes the standard Wiener process (Song et al. 2020
(tt)² When ((x t) 2 V logp(x), the diffusion in equation 1 describes Langevin dynamies, which for a
suitable decreasing noise schedule att) can be shown to produce samples from target distribution p as (Bussi & Parrinello, 2007 Welling & Teh 2011 We indicate this convergence by writing qe p.
2

---

Published in Transactions on Machine Learning Research (07/2023)
As the data points are perturbed over time t, the distribution qu evolves to 9:- This evolution is described by the Fokker-Planck equation
(Risken 1996),
0²
Σ
da, (2,)()))(()))
Σ
Diy((e (2)(
01
1=1
j=1
at))
V [1, t))((())]
V2 ((
2
(2)
a(1) Vn (x) Vn log [x)]
2
a(1)²
Vn (x) (V, logp(w) Vn (a)g)
2
where (xc, is isotropic diffusion lensor with elements Day( al02 if i = and (sy(ar 1) 0 2
otherwise, 2 V V represents the Laplacian operator and (t) and f(xxt) are defined as above. When dr P, equation vanishes, and the evolution of dr stops. When the drift term uıxxt) is defined above, namely as the gradient of potential Jordan et al. (1998) show that the dynamics in equation prescribe a direction of steepest descent, on free-energy functional with respect to the Wasserstein metric.
A result due to Auderson [1882) shows that the forward dynamics equation can be reversed, effectively undoing the evolution of q to p. These reverse dynamics are given by
dæ [1(a, v log (()) a()d@
where now df is negative time step, and w is time-reversed Wiener process. The reverse of diffusion process is therefore another diffusion process.
Remarkably, there is a deterministic process with the same marginal densities as those prescribed by equa-
tion 3 The corresponding dynamics are given by the probability flow ODE [Maoutsa et al. 2020 Song et al. 2020
att) V log qe (x)
dx [(xxt))
(4)
If we substitute the Langevin drift term u(xx = a(t)² v logp(x) from above, equation becomes
at)) 2
dæ
V logp(x)) - Vae log q (x)) dt.
Since dt is assumed to be negative time step in the reverse process, equation 5as written provides the dynamics of the forward process pushing gr toward the target distribution when dt is positive, Equation 5 represents the score-difference (SD) flow a probability distribution qe evolving toward (or away from depending on the sign). Note that this is no longer a diffusion but rather defines deterministic trajectory.
Combining equation 2 and equation yields
dg. Va (x( di
dæ
(6)
at
This is the Liouville equation Ehrendorfer 1994 Maoutsa et al. deterministic systems analogously to how the Fokker-Planck equation describes stochastic systems.
2020 which describes the evolution of
Many authors denote the Laplacian by but have reserved use for discrete-time differences. Heee the patential given by U(x) log p(z).
3

---

Published in Transactions on Machine Learning Research (07 (023))
2.2 SD Flow Optimally Reduces KL Divergence
A continuous flow dæ f(a:) can be approximated by defining transformation T(x) x of for some small step = > 0 If x ~ q and x' = T(æ) ~ 9/2)) then Liu & Wang (2016) show that the Kullback-Leibler (KL) divergence between 1/7) and p.
(prpll)) Exwaon [log (()(() log px))]
(7)
varies according to its functional derivative,
VelDa..ygrlllll -
-Eawgr [Tr (App(()))
(8)
where
Apf(x) = Va log)(x))(x>) + (.)(
(9)
is the Stein operator Gorham & Mackey
2015
By applying Stein's identity
Stein
1981 Lin et al. 2019 to equation
we obtain
Eawgy Tr (Apf(z))
Eaway
V logp(æ) f(x)]
E Bawqrry
[ = log (oy) (2) f(x)]
(10)
E Eawarry
log p(x)
V log (a:)
f(x)
which is the inner product of the score difference and the flow vector f(x) Maximizing the reduction in the KL divergence (equation [ corresponds to maximizing this inner product. Since the inner product of two vectors is maximized when they are parallel, choosing f(x) to output a vector parallel to the score difference
will decrease the KI. divergence as fast as possible. We can also see from equation the decrease in the KL divergence is then proportional to the Fisher divergence,
g and equation 10 that
Dr (T llp) = Ex-qp
[l/V logp(æ) - Va log qury (=))])
(11)
which, never being negative, shows that moving along the SD flow in sufficiently small steps will not increase the KL divergence.
We note that the SD flow also naturally emerges from variational gradient flows defined over f-divergences (Gao et al. 2019 Feng al. 2021 Gao al. 2022). Specifically, Gao et al. (2019) show that the negative gradient of the first variation of the f-divergence functional evaluated q defines flow that minimizes the divergence. When that f-divergence is the KL divergence, (KI (p]p), the first variation is given by oF = log ((x) log)(æ) + 1, whose negative gradient is the SD flow. Dynamics closely resembling the SD flow also emerge in the study of optimal interventions for constraining stochastic interacting particle systems in Kullback-Leibh control problems (Maoutsa & Opper 2022
2.3 SD Flow Solves the Schrödinger Bridge Problem
The Schrödinger bridge (SB) problem considers the most likely evolution between two marginal densities q and p over time E [0, T] (Chen et al. 2015). Specifically, the problem solves
p = arg min DKI. (PIQ),,), PeN)(P.)
(12)
where [[ p) is the collection of densities with marginals Po 9 and P, = and Q is a reference diffusion with initial density that causes the solution to be unique (Winkler et al. 2023
Chen et al. (2021) show that solutions to the SB problem correspond to the following forward and backward SDEs:
dx = [1( t) + o² (t) log (zx t))dt (t) dx (1(x,t) x² (t)V log ((x t))dt at))d
(13)
(14)
31f e is sufficiently small, then the Jacobian of T is of full rank, meaning that the transformation is bijective. WWe assume the mild condition that f(x)p(x) 0 and f(x)grr) (x) ||ж||| co.
Note that we are considering the diffusion from q to p for consistency with the Langevin example Section 2.1
4

---

Published in Transactions on Machine Learning Research (07/2023)
where V and T are non-negative potentials (or Schrödinger factors) such that (a t) t) t) qu (xx), and all other notation is as defined in Section 2.1 If we set (ax, L) = and (ar, t) (a)). then the necessary
conditions on the potentials are met, and the backward SDE 114 matches equation 3 from which the probability flow ODE in equation | and the SD flow in equation directly follow. Therefore, SD flow defines
a Schrödinger bridge between the source and larget distributions q and p.
3 Applying SD Flow to Proxy Distributions
One of the difficulties in applying Langevin dynamics other score-based methods the requirement that have access to the true score of the target distribution, V log p(x), which is almost never available in practice. It is also the case that when operating in the ambient space of æ E IRd the score may not be well defined in areas of limited support if the data exist on a lower-dimennsionl manifold, which is generally assurned for a variety of data lypes of interest, such as image data. A large literal has emerged that is dedicated to the estimation of this score or the design of training procedures that are equivalent to estimating Song & Ermon 2020 Song Kingma, 2021 Karras et al., 2022).
it (Hyvärinen & Dayan 2005;
Applying SD flow would appear to be al least twice difficult, since instead of one score to est estima,e, now we have two. The distribution qu is also changing over time, so even reasonably good estimate at one time would have to be discarded and re-estimated at another. Our approach will be to essentially ignore p and qe and work instead with modified praxy distributions that are casier to estimate and manage. Importantly, aligning these proxy distributions will automatically align the unmodified source and target. distributions.
3.1 Aligning Proxy Distributions
We can assess the alignment of two distributions q and p by computing a statistical distance between them Although this quantity not always a true "distance" in a strict mathematical sense, it will have the two key properties that (1) D(allp) 2 0 for all distributions p, p.q and (2) [[(llp)) DDpll)) 0 if and only if q- Perhaps the best-known statistical distance is the KL divergence, (xl(())p) (equation 7 although it can diverge to infinity if and have unequal support.
One way to equalize the support of two distributions is to corrupt their data with additive noise defined over the whole space Rdd Let us assume a Gaussian noise model. The distribution of z = x + ae, with æ ~ p and e~ E~N(0, I), given by the convolution p = p (O0,
x(z) = p(=;0( = [
(15)
Exep [(z;
with q(z) ((2;0) Eg [x( defined analogously. Although DKl(īll)) < (gl()) and (pl((p)))) as oc only if 9 p (Zhang et al. 2020 As result, aligning the proxy distributions and p is equivalent to aligning the generative distribution q with the target distribution p.
(Sripcrumbudur et al. (017),, it is casy to show that (gl( 0 if and
We have shown that moving parallel to the score difference optimally reduces the KL divergence, so we define an SD flow between q and ğ to align g with p. The score corresponding to p(z) = p(z;o) 15 is
V_ logp(z;or) = V=r(2;0)
o(z;o)
Enep [ (2N(zz
(16)
Enep [N 0²
Enyp
0² 0 T]
Exep W (z; 02]]
We will suppress the time index on o here convenience.
5

---

Published in Transactions on Machine Learning Research (07/2023)
The score for ((z) g(2; o) is derived in the same way for = y + oe, with y ~ q. This leads to the following expression for the score difference:
1 Bexp [K (z, )x
Bynu [Ko 8)8)
Vz (zlog((z;0) - V= V=loga(2;0)==
(17)
E Rgw Ko (2, x:) Bynug [K (z y)
where exp
1== 2 20
is the Gaussian kernel
If we set the noise level according to the schedule (t) = then the variance term cancels from equation leading to the update
A= ?) Eapp Ko (2, x))))] Eynu [Ko (2o y)y
(18)
2 Exeu [o x)] Bynu (2. y)]
for some 1j > 0 defining the step size.
We treat the dynamics of y ~ q(y) as being the same as the dynamics of z ~ 7(2) and sct Ay = Az under the following rationale: At time L, we draw Y/ ge(y) and 6 ~ NO,, I) to form Y toe = 2ς ~ ύ(z). We then perturb 2 according to equation 18 forming Z₁ + Az, Z++1 (2)1 which is closer to the distribution p(z) than the point 2N was. However, there is a second way to obtain the point ž++1 (z)1 which is to first create y: + Ay: = Do+1 ~ (1) via the (unknown) update Aye and then corrupt it with noise to form yo+1 + ae = z++1 (z) If we think of our random noise e as coming from a queue 2 then the value of e the same in both scenarios. The only difference is that e is drawn from the queue at time in the first scenario and is saved until time + 11 in the second. As a result, we can equate the two definitions of zt+1 and write
yt + Ay. +OE yt + oE + +
(19)
M++1
which, when we cancel like terms, leads to Ay: Azy 2. In this case the update Ay does not necessarily equal the score difference of the "clean" distributions, V, log (ogp(y) Vy log q(y), but is rather the perturbation to the clean data required for optimally aligning the proxies q and p. Recalling that the alignment of the proxy distributions implics the alignment of the original distributions, the update Ay still serves to align with p.
3.2 Limitations and Alternative Formulations of SD Flow
In the limit of infinite data, equation 17 is exact. But applying this formulation in a large-data setting can be computationally expensive, and estimates using smaller batches may suffer from low accuracy, especially in high dimensions. Recent work by Ba et al. (2021) shows that SVGD, which has a kernel-based specification similar to our description SD flow in Section 3.1 tends to undorestimate the target variance unless the number of particles is greater than the dimensionality. However, those authors note that MMD gradient descent (Arbel al. 2019), which is closer in formulation to SD) flow (see Appendix[B.3 does not have this limitation. Additionally, recent work by Wang et al. (0022) proposed Wasserstein gradient descent method essentially identical to the SD flow formulation in Section 3.1 albeit operating in a projected space, which the authors claim scales favorably to high dimensions.
In any case, it is useful to consider each term of equation 17 as a module that can be swapped out for another estimate, depending on the problem setting. For example, we can rewrite equation 17 in the equivalent form
1
V logp(z;0) Vz logg(2;;))
E | E[y/|2]]
(20)
o²
1
Dz(2;0) D2 (z:0))
(21)
where (2) and D (2; are the optimal denoising models for the distributions p and (t, respectively, when corrupted by Gaussian noise at level σ. simple derivation of this result is possible by rearranging
This is possible because the normalization constant of the normal distribution cancels from the mumerator and denominator. 8 Here we assume constant noise schedule, (t)) for all 9hii is the case with random number tables well our computers' pseudorandom number generators.
6

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Tweedie's formula [Efron 2011 10 but provide separate proof of optimality in Appendix B.1 This formulation is particularly well-suited to high-dimensional applications, as it admits the use of specialized U-net architectures that form the backbone of modern denoising diffusion models (Karras et al. 2022).
As practical consideration, the denoiser corresponding to the target data would need to be trained only once, while the denoiser for the generative distribution would, at least principle, need to be retrained after each step along the flow. This is not in itself major limitation, since the iterative fine-tuning of the sencrativo-distriutttoi denoiser Dye is more onerous than the standard practice of training GAN generator and discriminator in alternating steps, although it does create potential burden resources by requiring a second denoiser to be loaded in memory. However, since we actually observe y q: before corrupting it to form z + de, we can replace Dx, (z;0) = E[y|=] with in equation 21 11 leading to the update
y - (1-p'y + (2;(2;0)
(22)
for soine small step size p. In Section 4 show that this approxinate formulation of SD flow is equivalent to the reverse process in denoising diffusion models.
4 Relation to Denoising Diffusion Models
In diffusion modeling, data from the target distribution is corrupted in a forward diffusion process by Gaussian noise under the scale and noise schedules 0e = (t) and az = (t), respectively. Then for zo æ ~ p, the conditional distribution at time relative to that at time s < t is given by
p(z(|z,) N(ar T
where atls a4/a, and o? o? -0-- Kingma et al. 2021
of
The hard part is inferring the reverse diffusion process, p(z,(z)), 2.). which is intractable unless also conditioned on 2g = a: p(z, 1 2e; x) (Neee o2),) where Majt = (a))o?/o?)2z + (1.09//0 of) j and o = = o Tn practice, æ is replaced by D(ze; ox), the output of denoising model
If we let a, = for all s,t, then
o² +
2, 2:
1
D(z(: o
(23)
0i
(1 -p)p + + + vpa,e,
(24)
for 6, ~ I) and - 1 / Recalling that in our framework, yr orE, (((z)o))) for all t, equation 24 becomes
2, (1 p)y, + (z(; + Vpo,e, + p1,
(1 p)ye + PD( 0.) + v po? + (1 - -1-
2
(25)
p)y + Dz(z
o2 +
of
ye
ys + d.C.
where E, E; I) and y. follows from equation 22
In other words, the updating process under SD
flow is equivalent to the denoising diffusion reverse process under the substitution described in Section 3.2
WTTweedi''s formula states that E[z|2]] = o'V, log)(=;0).
This an approximation, since D (2) E[y|a] will not necessarily equal y and may be closer local mean for large 12 alternative but equivalent implementations, the error between a and zę predicted by a parametric model c=(s)))).
7

---

Published in Transactions on Machine Learning Research (07/2023)
5 Implicit Flows in Generative Adversarial Networks
5.1 Decomposing Generator Training into Sub-problems
When training a generative model ge, define loss L, which is scalar function that quantifies the discrepancy between the current model output and the target distribution. We typically treat this loss as function of the parameters 0 E I2" and then optimize 0 to minimize L via gradien descent at some learning rate 1j > 0:
0' - 1)
(26)
00
However, the loss L is also function of the generated data y - go(E) IR,, which is ilself a function of either & E Ie or 0 IR", depending on our perspective. This perspective can be made explicit by decomposing the derivative of the loss via the multivariate chain rule,
OC
Oc 0y
(27)
00
dy
00
18n
dxn
This allows us to consider each component of the decomposition as corresponding to its own sub-problem. In the first sub-problem, we perturb the generated data y the direction of the negative gradient,
ac dy
T
y A1
(28)
where A is some small step size. Intuitively, the perturbod data y' corresponds a potential output of the generator that would have a lower loss, but we can also interpret it as resulting from gradient flow on the synthetic data. In GANs, this flow serves to approximately "invert" the discriminator (Weber) 2021
In the second sub-problem, update the generator parameters 0 by regressing the new, perturbed target y' on the original generator input & via the least-squares loss,
1 J 1 [l90(()) 2'112 - l1 1'1
(29)
Putting the pieces together leads the parameter update
(0J
0'=0 0 Aa
(30)
00
(go(E)
0
Aa
(go(E) - -y)
(31)
00
Oy
T
0
Aa
(y - 3²)
(32)
30
Oy
T
0C T
0
X1d2
(33)
20
Oy
0
A1de
(34)
00
which is is equal to the standard gradient update of under the original loss L (equation 26 with step size
77 = A1d2. Here equation 33 follows from equation 32 by rearranging and substituting equation 28
while
equation 34 follows from equation 33 via equation 27
We treat the gradient as the transpose of the derivative.

---

Published in Transactions on Machine Learning Research (07/2023)
Although this decomposition is a direct consequence of gradient descent, it shows that hidden within gen- erator training are two sub-problems with separate control options (their learning rates, for instance), each of which may be easier to conceptualize and handle than the original problem. In particular, we see that the model-optimization step of generator training is proceded, at least implicitly, by a particle-optimization step that prescribes a flow in the ambient data space Rd, regardless of the overall loss being optimized. This suggests that we can treat this particleoptiimzztion step as a target-generation module that can be swapped out in favor of other procedures, such as SD flow. We note that this interpretation is consistent with recently proposed flow-based methods for training parametric generators (Gao et al. 2019 2022). Furthermore, it suggests that wide variety of generative models can be understood in terms of the dynamics imposed on the generated data during training.
5.2 SD Flow in GANs
Many GAN generators employ the widely used non-saturating loss (Goodfellow 2016 given by
1
[())= = -Pp [log (go s
log f(y),
(35)
B
BEBruqe
where & E R' is a random noise input to the generator drawn from prior distribution po and f IR + (0, 1) is a separately trained discriminator that estimates the probability that its argument is real data coming from target distribution p (in which case f R: 1), opposed to fake data coming from the generator distribution qi (in which case 0). We note that in every practical case the expectation over finile batch of generated data y go( collected in the set B. Intuitively, the aim of the loss given by equation 35 is to tune the parameters 0 to mazimize the discriminator's assessment of generated data as real.
working with an empirical estimate of
Tt can be shown that, if the prior probabilities of coming from either p or at are equal, the Bayes oplimal classifier fe is given by
p(y)
f((y) p(y) (3)
(36)
where we have included the time subscript to indicate the optimal discriminator's dependence on the changing distribution dt- An optimal discriminator is often assumed in the analysis of GANs but almost never holds in actual practice.
When implemented as a neural network, the discriminator fi usually terminates with a sigmoid activation,
f( = + (By))
(37)
where (3) is the pre-activation output of the discriminator ft. Equating equation 36 and equation 37 we see that in the case of an optimal discriminator, h+(y) = p(y) - log d (v), whose gradient is the score difference,
V Vy) = Vy log V. y log qe (y)-
(38)
When trained using the non-saturating loss (equation
35
the gradient flow induced a point y is
Vyl((y)
Vyl((y')
VyC
B fe(
B
f((e))
D'EB
(39)
B (1 - f(())) (,)
x (1 - f((x)) VV log p(y) - V, log gr (y)).
Since [1 - fe(g)) > for all y, taking the results of Scction 5.1 into account, we see that standard GAN training with an optimal discriminator consistently pushes the generated data toward the target data in a direction parallel to the score difference (equation 38
9

---

Published in Transactions on Machine Learning Research (07/2023)
We can also consider alternative to the non-saturating loss that focuses the sum of the discriminator's pre-activation outputs for generated data,
Cult
he(y):
(40)
WEB
which induces flow exactly equal to the score difference when the discriminator is optimal:
V y Calt
-y)+(y)+ + Σ
Vyli(y))
V'EB'U
(41)
Vy logp((y V log (y))
6 Algorithms
In this section we provide pseudocode algorithms for two main applications of SD flow: (1) direct sampling via particle optimézation, which transports set of particles from source distribution to match target distribution, interpolating between the distributions in the process, and (2) the model optimization of a parametric generator by (a) progressively perturbing generator output toward the target distribution and then (b) regressing those perturbed targets on the generator input. In Section 5 we showed that the training of GAN generator can be decomposed into steps (a) and (b). This creates the opportunity to replace a scparately trained discriminator with another target-generant
method, such as SD flow, in step (a).
The algorithms reflect that SD flow has based specification (Section 3.1) derived from considering noise-injected proxy distributions; there is the denoiser-bused specification (Section 3.2 which exploits link between SD flow and diffusion models; and there is the density-ratio specification (Section 5 as estimated via discriminator, such as one would use in GAN training. Other specifications and estimation methods are possible, especially considering the fundamental role played by the density ratio in machine learning applications (Sugiyama et al. 2012
than one representation or specification. There is the kernel-
6.1 Particle Optimization
A Algorithm Particle optimization with SD flow
Input: Target data {x p, base (prior) data {y, M ~ qo, noise schedule o(t), step schedule r7(t) repeat
Draw data batches x ~ p and y ~ d
Draw noise E E~ NN, I) and perturb data: z= y + o())
# Calculate SD (equation 18
21 or 38
Az X Ex-p (z,)) Eynn Koco (z.M)
Fe (Kro)
Faenu (2.y)))))))))
()(=)((t) - Dú (z:o(t)) & vzh((z)
# Move (clean) data in SD direction.
y - + (t) until Terminated
In the particle-optimiatte application (Algorithm sample is generated by perturbing a single batch of "base data," " much like one would via Langevin dynamics or Hamiltonian Monte Carlo. Here we interpret the base data as being drawn from qu and evolving to the distribution qt ~4 p through iterative perturbation.
6.2 Model Optimization
Ordinary regression problems involve paired training data ((E, y)) implicitly defining the mapping + to be learned. the case of generative modeling, where the aim is to map data from prior noise distribution target distribution, no priori pairing exists. This requires the mapping be learned indirectly. With GANs, the sub-problem interpretation of Section 5.1 is that the discriminator provides this pairing by associating noise input with perturbed output y that then serves as regression target.
10

---

Published in Transactions on Machine Learning Research (07/2023)
Algorithm 2 Model optimization with SD flow
Input: Target data {x; p, noise input for generator Po, initialized generator go (0 E R"), noise schedule at)), step schedule n((), learning rate schedule A(t)
repeat
Draw data batches a apand Generate sample y (E)
and 5 Po
Draw noise c cnd (0, and perturb data: 2 y + o(t)
# Calculate SD (equation 18
21 or
38
Be (enn(Krte)(s,0)))
Bym K (2.1)))
A= x Baup (ant
D)(=;((t)) - Dr (z: & Vzh((z)
Ey-or (Kco
# Move (clean) data in SD direction.
f (tt
# Update generator parameters via regression.
0 - 0 Velloo))) yyl²
until Terminated
The model-opliinzzaion application (aliiionnnAlllla (Algorithm 2 trains a generalor with unpaired data via regression on perturbed targets that move progressively closer the target distribution. Here we interpret qo the distribution of the output of the generator ge before training. As the generator regrossos perturbed
largets, its output distribution changes to 4 at time slep L as governed by the Liouville equalion W When
the choice of SD flow is the gradient of the pre-activation discriminator output, V=h+(z) (equation
38 this
algorithm is equivalent to GAN training with the alternative loss described in equation 40
7
Experiments
Here we report several experiments on toy data. We focus in this section on particle-optimizatin applications using the kernel-based definition of SD flow (Section 3.1. This allows us to compare the performance of SD flow against two other kernel- besnel-based particle-pptimizaton algorithms, namely MMD gradient flow (Arbel et al. 2019) and Stein variational gradient descent (SVGD) (Liu & Wang 2016). Consistent with those works, focus low-dimensional data, although present additional results moderately high-dimensional data in modelloptiiizattie application in Appendix C.4
We leave a thorough exploration of our alternative specifications of SD flow (Section 3.2) on high-dimensional image data to follow-up work, since beyond the considerable computational and data demands in that setting, there are many architectural design choices and training tricks behind the current state of the art in image generation, which fall outside the scope of this paper. Nevertheless, several suggestions for applying SD flow to high-dinensional data, such images, are given in Section 3.2 The general procedures given in Algorithms and 2 still apply in the sigh-dimensional case.
7.1 Experimental Conditions
For the experiments reported in this section, we vary the conditions below, which describe along with the labels used in Tables and 2 All methods were tested with default learning rate of r = 0.1.
ADAGRAD:
The published implementation of SVGD [Liu & Wang 2016) relies on the adaptive gradient opti- mization algorithm AdaGrad (Duchi et al. 2011).. We tested all algorithms using AdaGrad versus standard stochastic gradient descent (SGD).
BATCH:
The moderate size of our toy data sets allows us to test both batch-based and full-data versions of the algorithms. Batch sizes of 128 were used in the batch condition.
CONST:
In Scction 4 we showed that an approximation of SD flow is equivalent to denoising diffusion under & decreasing noise schedule. However, we also showed in Section 3.1 that we can work entirely
14 Section 2.1 and Appendix B.2
11

---

Published in Transactions on Machine Learning Research (07/2023)
with noise-corrupted proxy distributions, which allows for constant noise schedule. Both MMD gradient flow and SVGD can be interpreted in the constant-noise setting, we tested all algorithms with both constant and varying noise schedules. When using a constant noise schedule, we used the method described by Liu & Wang (2016), which determines the kernel bandwidth as a function of the median squared pairwise distance among the source (base) data and the number of points [3 In the non-constant setting, we used modified cosine noise schedule to interpolate between a maximum and minimum variance: (1) Fmxx (Tt.) for E [0, [max], with max = 2/TT cos (/ /
ANNEAL:
Although the kernel-based realization of SD flow is motivated by considering noise-annealed proxy distributtins (Section 3.1 annealing is not necessary for SD flow to converge. Further, since SVGD does not anneal data during training, and MMD gradient flow introduces annealing only as a regu- larization technique, we tested all algorithms in both anncaled and non-annealed conditions.
OFFSET: In our experiments in R we introduced the condition of initializing the base distribution either away from the center of the target distribution (offset) or centered at the target distribution's mean.
A discussion of the relationship between SD flow and MMD gradient flow is given in Appendix B3 and between SD flow and SVGD in Appendix B.4] That discussion provides additional context behind some of the experimental conditions described above.
7.2 Results
To measure distribution alignment, we define mean characteristic function distance (CFD),
K
(CF()))9)) = K Boo [exp(iw] x)] - Eyrrr [exp(iww) v)] I,
where == /-1 is the imaginary unit. (pr(p)))) is the mean absolute difference between the empirical characteristic functions of p and qi evaluated at K frequencies (K 256 in case) We IR drawn from normal distribution. For the reporting in Tables 1 and 2 we establish convergence in the following way: We compute the CFD for two independent random samples of the target distribution and record the mazimum value over 1000 trials, which provides a threshold for the CFD estimated from finite data when the distributions are equal. We say that an algorithm has converged the CFD between the synthetic and target distributions falls below this threshold.
7.2.1 Fitting 25-Gaus Grid
For our first particle-optimizatie experiment, created target distribution of 1024 points drawn from mixture of 25 spherical Gaussians arranged on grid in R² and initialized 1024 points from spherical Gaussian base distribution at a large distance from the larget distribution but closest to its lop-left compo- nent. (See Figure
- = 10 and of ! For the non-constant condition, we used a cosine noise schedule (described above) with 0.5. The results of this experiment are summarizod in Table Note that SD flow is the only algorithm to converge in every condition and that each algorithm consistently either converges or fails to converge in each condition.
7.2.2 Fitting Gaussian Mixture
In this experiment, our target "mystery distribution" p is a mixture of 30 Gaussians in R arranged in the shape of a question mark. (See Figure 2) In addition to the experimental conditions tested in Section 7.2.1 we also tested initializing the base distribution offset from the target distribution versus centered at the target distribution's mean. For the non-constant condition, we used a cosine noise schedule (described
above) with - and oi = 0.5. The convergence results are given in Table 2 Once again, SD flow is the only algorithm to converge in every condition.
15,2 x median[D? // log(N +
12

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Table 1: Convergence probabilities (final three columms) for the SD flow (SD), MMD gradient flow (MMD) and Stein variational gradient descent (SVGD) algorithms after 1000 iterations over five independent trials on the 25- 25-Gassiian particle-ptámization problem in IR2 under the experimental conditions described in
Section 7.1 Convergence is defined obtaining minimum characteristic function distance less than threshold empirically determined by comparing independent copies of the target distribution (see text). SD flow is the only algorithm to converge under all conditions.
ADAGRAD BATCH CONST
ANNEAL
SD (OURS)
MMD
SVGD
N
N
N
N
N
N
N
Y
1
N
N
Y
N
1
N
N
Y
Y
N
Y
N
N
1
N
Y
N
N
Y
Y
N
N
Y
Y
N
Y
N
N
Y
N
N
Y
Y
Y
Y
Y
Y
1
Y
Y
Y
Y
Y
Y
1
7.2.3 Discussion of Experimental Results
Our experiments show that there are conditions under which all tested algorithms successfully and consis- tently transport particles from source distribution to target distribution. However, only SD flow showed itself to be robust to all experimental conditions. SVGD converged in every condition in which AdaGrad employed in our experiments, where its performance was extremely close to that of SD flow. However, SVGD consistently failed to converge in any condition in which AdaGrad was not employed, while SD flow did converge in these conditions. MMD gradient flow fared worse in general and exhibited a tendency to cause a subset the synthetic data to diverge We note, however, that noise plays a different role in our setup than it does in the work of Arbel al. (2019] where it serves a regularizing purpose.
The convergence results reported in Tables and were determined by comparing the minimum CFD achieved by an algorithm after 1000 iterations to threshold empirically determined by multiple comparisons of independent random draws of the target distribution. Failure to meet this threshold could mean that the algorithm failed to steer the synthetic data toward the target distribution at all or simply failed to do so in the allotted number of iterations. For completeness, we provide the average minimum CFD values corresponding Tables and in Appendix C.1 Additional experimental results are also reported Appendix C
8 Concluding Remarks
In this work, presented the score-difference (SD) flow optimal trajectory for aligning source distribution with target distribution. We also showed that this alignment can be performed by working entirely with proxy distributions formed by smoothing the data with additive noise. As a result, while the SD flow defines a deterministic trajectory, its application to noise-injected points adds a stochastic component.
16gee Remark Appendix R.3
13

---

Published in Transactions on Machine Learning Research (07/2023)
Iniiiilizaiinn
Step 100
Stop 260
Step 500
Stop 1000
Thaggl
Syntheee
Syrthrio
Systhett
SD Flow
Initialization
Step 100
Step
Step 500
Step 1000
Tarpet Brithekk
Tagge Systh
Targer Syrtholl
Target Berthert
Target Syithette
MMD Flow
23
Initialization
Step 100
Step 250
Stop 500
Stop 1000
Tarpet Settwrts
Target Systeet
Target Surthikg
Target Rerthrno
Syttwet
SVGD
Figure 1: Evolution of synthetic data points from an offset base distribution toward the target distribution of 25 Gaussians over 1000 steps of SD flow (top row), MMD gradient flow (middle row) and SVGD (bottom row) in the no-AdaGrad, full-data, batched, and annealed condition (corresponding to the second row of Table D. Only SD flow converged in this condition.
SD flow can be used direct sampling technique, in which case a sample of base data is converted to a sample from the target distribution via the flow. It can also be used in the training of parametric generative models, in which case generator output is perturbed by the flow to provide the generator with regression targets guaranteed to be closer to the target distribution than the previous output. Unlike most other score- based methods, there are no restrictions on the choice of base or prior distributions. Consistent with this advantage, we have shown that SD flow forms a Schrödinger bridge (Schrödinger 1932 De Bortoli et al., 2021) between source and target distributions.
We have shown that SD flow emerges in both GANs and diffusion models under certain conditions. The conditions for GANs include the presence of an optimal discriminator, which is often unattainable when training with finite, categorically labeled data. By contrast, diffuusion models have the comparatively easier task of learning to denoise an imago, a task for which the ground truth is more readily represented in the training data. Modern neural denoising architectures that employ allention provide another edge, since they have shown themselves to be extraordinarily capable of capturing patterns in data due to their error- correction and pattern-retrieval characteristics reminiscent of Hopfield networks (Ramsauer et al. 2020).
SD flow supplies a link between IGM techhods-nameee GANs and diffusion models- that collectively per- form well on all three desiderata. of the so-called generative learning trilemma (Xiao et al. 2021): high sample quality, mode coverage, and fast sampling. Inserting SD flow as an alternative, non-adversarial target-generation module within generator training 17 replacing the need for an optimal discriminator, could lead to the development of "triple threat" " models that produce high-quality, diverse output in a single in- ference step. The various alternatives we have described in this paper for representing SD flow namely those based on kernels, denoisers, and densily ratios suggest variety of opportunities for integrating this approach into a number of IGM frameworks. We look forward to further developments in this direction.
17 See Section 5.1 and Appendix C.4
14

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Table 2: Convergence probabilities (final three columns) for the SD flow (SD), MMD gradient flow (MMD) and Stein variational gradient descent (SVGD) algorithms after 1000 iterations over five independent trials on the "mystery distribution" " particle-optimization problem in I under the experimental conditions described in Section 7.1 Convergence defined obtaining minimum characteristic function distance less than threshold empirically determined by comparing independent copies of the target distribution (see text). SD flow is the only algorithm to converge under all conditions.
ADAGRAD BATCH CONST ANNEAL
OFFSET
SD (OURS)
MMD
SVGD
N
N
N
N
N
N
N
Y
N
N
N
Y
N
N
N
N
Y
Y
N
Y
N
N
N
Y
N
N
Y
Y
Y
N
Y
N
Y
Y
Y
N
Y
N
Y
Y
Y
Y
N
N
Y
Y
Y
Y
N
Y
Y
Y
MANNNN
N
N
Y
N
Y
Y
N
N
Y
Y
N
N
Y
N
Y
Y
Y
N
Y
Y
Y
Y
Y
N
N
Y
Y
N
I
Y
Y
Y
Y
INNI
Y
N
Y
Y
N
Y
Y
N
Y
Y
Y
Y
Y
Y
Y
Y
Broader Impact Statement
Implicit generative modeling in the image and text domains has matured to the point of producing output with unprecedented level of realism, with other modalities not far behind. It is getting increasingly difficult to tell real data from fake, which is exciting when one reflects on how far the field has come in the past few years but also disturbing when one considers the consequences of advanced IGM methods in the hands bad actors. We promote the responsible development and use IGM not only with respect to its deployment but also its training, which should only use data with the proper usage rights secured. We also support continuing research into detecting generated manipulated data in the effort of counteracting the misuse of this technology and minimizing the societal effects of any misuse.
15

---

Published in Transactions on Machine Learning Research (07/ (272))
in
Step 100
Step 250
Stop 600
Step 1000
Synthwtt:
Systhate
Syntwetc Tarpe
Synthete
SD Flow
in
Step
Stop 250
Step
Step
Systhers Torpet
Syerete Target
Syrthil Toiget
Sytthano Torger
Synthoo
Tarpet
MMD Flow
Initielleation
Step 100
Step 250
Step 500
Step 1000
Target
bagat
Target
Targe
SVGD
Figure 2: Evolution of synthetic data points from an offset base distribution toward the target distribution of 30 Gaussians over 1000 steps of SD flow (top row), MMD gradient flow (middle row) and SVGD (bottom row) in the ad-Adara,,, full-data, cosine-noise annealed condition (corresponding to the fourth row of Table 2 Only SD flow converged in this condition.
References
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-322 326,, 1982.
Michael Arbel, Anna Korba, Adil Salim, and Arthur Gretton. Maximum mean discrepancy gradient flow. Advances in Newral Information Processing Systems, 32, 2019.
Jimmy Ba, Murat A Erdogdu, Marzyeh Ghassemi, Shengyang Sun, Taiji Suzuki, Denny Wu, and Tianzong Zhang. Understanding the variance collapse of svgd in high dimensions. In International Conference on Learning Representations, 2021.
Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft. When is "nearest neighbor" mean- ingful? In Database Theory ICDT'99: 7th International Conference Jerusalem, Isruel, Jarvary 10 12, 1999 Proceedings 7, pp. 217-235. Springer, 1999.
Giovanni Bussi and Michele Parrinello. Accurate sampling using langevin dynamics. Physical Review E, 75 (5):056707, 2007.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018
Tianrong Chen, Guan-Horng Liu, and Evangelos Theodoron. Likelihood training of schrödinger bridge using forward-backward sdes theory. In International Conference on Learning Representations, 2021.
Yongxin Chen, Tryphon T Georgion, and Michele Pavon. Optimal steering of a linear stochastic system final probability distribution, part i. IEFF Transactions on Automatic Control, (50:11188-1693 2015.
16

---

Published in Transactions on Machine Learning Research (07/2023)
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34: 17695 17709, 2021.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106 (496):1602 1614, 2011.
Martin Ehrendorfer.
The liouville equation and its potential usefulness for the prediction of forecast skill. part i: Theory. Monthly Weather Review, 122(4):703 713, 1994.
Xingdong Feng, Yuan Gao, Jian Huang, Yuling Jiao, and Xu Liu. Relative entropy gradient sampler for unnormalized distributions. arviv preprind orXiv:2110002755 2021.
Yuan Gao, Yuling Jiao, Yang Wang, Yao Wang. Can Yang, and Shunkang Zhang. Deep generative learning via variational gradient flow. In Internalional Conference on Machine Learning, pp. 2093-2101. 2101. PMLR, 2019.
Yuan Gao, Jian Huang, Yuling Jiao, Jin Liu, Xiliang Lu,, and Zhijian Yang. Deep generative learning via euler particle transport. In Mathematical and Scientific Machine Learning, pp. 336 368. PMLR, 2022.
Tan Goodfellow. Nips 2016 tutorial: 2016.
Generative adversarial networks. arXin preprint arxi: 17010001100000
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farloy, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXie: arXie:1406.c0 meeec110..6660 2014.
Jackson Gorham and Lester Mackey. Mcasuring sample quality with stein's method. Information Processing Systems, 28, 2015.
Advances in Neural
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXi::1810.01367, 2018.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-775 -7733 2012.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural
Information Processing Systems, 33:6840 33:6840-6851, 2020.
Aapo Hyvärinen and Peter Dayan. Estimation of non-normn non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the fokker fokker-planck equation. SIAM journal on mathematical analysis, 29(1):1 17, 1998.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basod generative models. orXiv preprint arXiv:2206. 00364, 2022.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696 21707, 2021.
Wu Lin, Mohammad Emtiyaz Khan, and Mark Schmidt. Stein's lemma for the roparameterization trick with exponential lamily mixtures. arXiv preprinl 1rXiv:1910.13399 2019.
Qiang Liu and Dilin Wang. Stein variational gradicnt descent: A general purpose bayesian inference algo- rithm. Advances in nearal informalion processing systems, 29, 2016.
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-fi fit tests. In International conference on machine learning, pp. 276-284, PMTRR 2016.
17

---

Published in Transactions on Machine Learning Research (07/2023)
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
Dimitra Maoutsa and Manfred Opper. Deterministic particle flows for constraining stochastic nonlinear systems. Physicol Review Research, 4(4):043335, 2022.
Dimitra Maoutsa, Sebastian Reich, and Manfred Opper. Interacting particle solutions of fokker-plande equations through Fradient-lgg-dnnse estimation. Entropy, 22(8))802, 2020.
Henry McKean Jr. A class of markov processes associated with nonlinear parabolic equations. Proceedings of the National Academy of Sciences, 56(())1907-1911, 1966.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1) 22()):2177-2660 2021.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprinl vxXix:2008.0 022117, 2020.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International con- ference on machine learning, pp. 1530-1538 PMLR, 2015.
Hannes Risken. Pokker-planek equation. Springer, 1996.
Erwin Schrödinger. Sur la théorie relativiste de l'électron et l'interprétation de la mécanique quantique. In Annales de I l'institut Henri Poincaré, volume 2(4), pp. 269 269-310, 1932.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 0411238--1445, 2020.
Yang Song and Diederik P Kingma. How to train your energy-based models. arXic preprint
arXiv:2101.03286 2021.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishck Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXin preprinl
arXiv:2011.13466, 2020.
Bharath Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Aapo Hyvärinen, and Revant Kumar. Density estimation in infinite dimensional exponential families. Journal of Machine Learning Research, 18, 2017.
Bharath K Sriperumbudur, Kenji Fukumizu, and Gert RG Lanckrict. Universality, charactcristic kernels and rkhs embedding of measures. Journal of Machine Learning Research, 12(7), 2011.
Charles M Stein. Estimation of the mean of a multivariate normal distribution. The annals of Statistics, pp. 1135-1151, 1981.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Densily rulio eslimation in machine learning. Cambridge University Press, 2012.
Yifei Wang, Peng Chen, and Wuchen Li. Projected wasserstein gradient descent for high-dimensionall bayesian inference. SIAM/ASA Journal on Uncertainty Quantification, 10(4):1513 1532, 2022.
Romann M Weber. Exploiting the hidden tasks of gans: Making implicit subproblems explicit. arXiv preprint arXiv:2101.11863, 2021.
Max Welling and Yoe W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICMI--11), pp. (681-88 2011.
18

---

Published in Transactions on Machine Learning Research (07/2023)
Ludwig Winkler, Cesar Ojeda, and Manfred Opper. A score-based approach for training schrödinger bridges for data modelling. Entropy, 25(2):316, 2023.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. ar preprint arXiu:2112.0780 2021.
Mingtian Zhang, Peter Hayes, Thomas Bird, Raza. Habib, and David Barber. Spread divergence. In Inter- national Conference Machine Learning, pp. 11106-11116. PMLR, 2020.
A Guide to the Appendices
In Appendix B. we show that the score difference corresponds to the difference between the outputs of optimal denoisers corresponding to the target (p) and current synthetic (q:) distributions. In Appendix B.2, describe the cvolution of the generative distribution of a GAN under any loss. In Appendix [B.3] draw connection between the kernel definition of SD flow and maximum mean discrepancy (MMD) gradient flow (Arbel et al. 2019). In Appendix B.4, we draw connection between SD flow and Stein variational gradient descent (SVGD). Additional experimental results are reported in Appendix
B Supplemental Results
B.1 The Score Difference the Difference of Optimal Denoisers
We follow approach similar to that found in Appendix B.3 of Karras et al. denoiser for We define the denoising loss by
2022 to derive the optimal
E(Dpio) = Ee [12p(x E; - Ex ((m,001 (z) = ==
Bx Baap (1;p))))) x=|²]
(42)
[ Expp [M(z;a, o2) ae dz.
8(Dniz,))
We can optimize E(Dpio) by minimizing the integrand E(Dpi 2, o) pointwise. The optimal denoiser is then given by
D)(2;0) = arg min E(Dpi D,(2;))
(43)
which defines convex optimization problem. We can then find the global minimum by setting V ((pp: r) to zero, leading to
Egep [N o² V, (p(z;)) x| =0 =0
(x) [V o²1]] = 2E [x
(44)
D)(2;0) Exxp [M 1)
Excp ((x;2,0211
the final term of which is equal to the first term inside the brackets in equation kernel. The derivation for D (z) is identical, which leads to the result.
18 when Kr is the Gaussian
B.2 GAN Dynamics Under General Losses
Since the negative gradient of the GAN loss defines flow on the generated data y, which the generator fits via regression, we can track the evolution of the synthetic distribution qe within the context of normalizing Papamakarios et al. 2021). In particular, the dynamics induced by
flows (Rezende & Mohamed 2015
19

---

Published in Transactions on Machine Learning Research (07 (07/2023)
generator loss constitute, in the limit of arbitrarily small steps, continuous normalizing flow whose effect the synthetic (generated) data distribution is governed by the Liouville equation (see equation 6 Section 2.1
( δt
Vv. V,
(45)
continuity equation with solution
qe (: (o(30) exp (L Tr (r((H((U)) dr) dr) -oo(oo) exp (f Ldrr)
(46)
where Hc(U+) is the Hcssian matrix of the loss L evaluated Yr, whose trace is the Laplacian V 5.0/Oy Taking the logarithm of both sides of equation 46 yields the solution to the inslanlaneous change of voriables differential equation (Chen et al. [0188] Grathwohl et al.) 2018). Note that this evolution holds for any generator loss L and does not make any assumptions about an optimal discriminator.
B.3 Relation to MMD Gradient Flow
In the study of reproducing kernel Hilbert spaces (RKHS), the Gaussian kernel K=(z,x) is known characlerislic kernel (Sriperumbudur et al.j 2011). This means that the mapping <pp(=) = (xaplKa(z,) æ)] is injective, and (p(()== = (*) for all æ if and only p p= = q. This forms the basis the maximum mean discrepancy (MMD), which is equal the Hilbert space norm [Mallll, where
(z) pz(z) pp(z) g K (z, y) Ex [Ko (z, x)]
(47)
is known the witness function Gretton et al. 2012 Arbel et al.
2019
In the theory of optimal transport, we wish to efficiently transport "mass" from an initial distribution to a target distribution p, which we can do by defining flow from to p via intermediate distributions d-- One such flow is defined by the solution to
dg.
V [q: WWW.
(48)
ôt
another instance of the Liouville equation with dynamics
6 45 that defines a McKean- Vlasov process (McKean Jr 1966
dæ, v. Wpane (Z+) dt
(49)
(Bm~pl V (Ze, æ)] EBy V Ko B)l) df,
where 20 proportional to V2.((z(:) - V2. For the Gaussian kernel, we have
The results of Section 3.1 suggest that, in the limit of infinite data, this direction is
x zz
V., Kz( = K,(21, x)
and for discrete time and finite data we can write equation
49
as
Az
Ko Ko xe:)
2
21
1
Ko (Zt, 9i)
y 21
(50)
M
N
M
M
N
w
(9)
ar;
w
yj
+
w
(gc)
(p) w
Zt.
(51)
This process defines the MMD gradient flow (Arbel et al.] 2019) The kernel version of SD flow (equation 18 can also be written in the form
where wp{p) K2 1g, x; ) 2Noo) and w/ (a)
(z Hi) 4/)/(Mo
20

---

Published in Transactions on Machine Learning Research (07/2023)
of equation 51 by setting w/() KKa(2 wt)/ Σi= Ko (2)) and wy (n) (Zt, wi)/ Ei=n Ka (21, (z); which causes the & term to vanish.
There are practical consequences of this difference in weighting schemes between methods, which put the MMD gradient flow at a disadvantage in some conditions, as discussed in the following remark.
Remark 1 p and qi are far aport, for a point z1 = y + De, with y ~ qi and a small kernel bandwidth o, equation 51 shows that under the MMD framework Σ, (a+) = 1/(Mo²) and E, w = 0. This suggests that under these conditions, the MMD gradient flow direction AZ will be nearly parallel to 2: - y ge, which would have the effect of increasing the variance of q: while not necessorily pushing it toward p. Evidence of this variance-erploding effect emerged in experiments and also apparent in the authors' original paper (e.g. Appendir: G.2 therein). Normalizing the weights in equation 51 to sum. to one resolves this issue, however, and MMD gradient flow and SD flow then become equivalent.
Remark 2
Our melhod prescribes that we inject noise at a level equal to the kernel bandwidth in order to sample from the noise-smoothed prory distribution. In controst, with MMD gradient flow the noise level is separate parameter that essentially controls a regularization effect. In that case, this added noise is typically at a level far greater than that of the kernel bandwidth, which remains fixee during training.
B.4 Relation to Stein Variational Gradient Descent (SVGD)
Another statistical distance that measures the discrepancy between distributions p and is the Stein dis- crepancy (Gorham & Mackey 2015),
(p(l)) Eu [r (Apf(z))),
(52)
where Ap is the Stein operator defined in equation D and a function that vanishes on the boundary of the support of p or (oquivalently) behaves such that f(=))((=) - and f()))g(a)) 0 as ||>||| >> x..
The discrepancy 52 vanishes if and only if p q, which can be seen by expanding out equation 52 in
equation 10
Exzn (T (Apf(())) = Ea-g (V l(g)( - Va logg( T f(xz)
(53)
since for nontrivial functions f, the above vanishes only when Va logp(æ) V log q(z). When is re- stricted to bounded functions within reproducing kernel Hilbert space (RKHS), one obtains the kernel Stein discrepancy (Liu et al. 2016).
As reported in Section 2.2 Liu & Wang (0166) establish a link between the kernel Stein discrepancy and the variation in the KL divergence, which leads to their derivation of Stein variational gradient descent (SVGD) as an optimal direction for reducing the KI. divergence between q and p when operating a RKHS:
((z) = Exeing V log logg(o)) + V v=K(z,
(54)
= Exnn
( logp(x) - V log (())) [)z,z)],
which shows SVGD to be the keinel-weighted average of the score difference. separate analysis of the connection between SVGD and MMD gradient flow.
Ba et al. (0021) provide
C Additional Experimental Results
C.1 Average CFD Values for Particle-Optimizatio Experiments
In Sections 7.2.1 and 7.2.2 we reported convergence probabilities for SD flow, MMD gradient flow, and SVGD under various experimental conditions. Convergence was defined as achieving a minimum characteristic function distance (CFD) below a threshold empirically determined by multiple comparisons of independent copics of the target distribution (0.0651 for the 25-Gaussian grid problem in R² and 0.0612 for the 30- GanSsss "mystery distribution" " in R In Tables 3 and 4 we report average CFD minimum values achieved by the algorithms after 1000 steps over five separate trials.
21

---

Published in Transactions on Machine Learning Research (07/2023)
Table 3: Average minimum CFD (final three columns) for the SD flow (SD), MMD gradient flow (MMD) and
Stein variational gradient descent (SVGD) algorithms after 1000 iterations over five independent trials on the 25 Gaussian particle-optimization problem in I under the experimental conditions described in Section
ADAGRAD BATCH
CONST
ANNEAL
SD (OURS)
MMD
SVGD
N
N
N
N
0.0007
0.6129
0.8495
N
N
N
0.0007
0.6111
0.8521
N
Y
0.0007
0.5816
0.7878
N
N
Y
0.0009
0.5837
0.7978
N
Y
N
N
0.0374
0.9382
0.7944
N
Y
N
0.0357
0.9365
0.7863
N
Y
Y
0.0371
0.9230
0.7454
N
Y
Y
0.0389
0.9249
0.7494
N
Y
N
N
N
Y
0.0006
0.1584
0.0033
Y
N
0.0006
0.1622
0.0031
Y
Y
0.0006
0.1566
0.0029
N
Y
Y
Y
0.0006
0.1553
0.0028
Y
Y
IND
N
0.0359
0.1884
0.0412
Y
Y
N
Y
0.0399
0.1895
0.0414
Y
Y
Y
N
0.0381
0.1892
0.0377
Y
Y
Y
0.0385
0.1902
0.0359
C.2 Data-Set Interpolation
Standard score-based generative modeling is designed such that the end of the forward process is Gaussian distribution. While this has the advantage of defining a prior that is easy to sample from for the reverse, generative process, it limits the flexibility of the method. Recent work on approximating Schrödinger bridge between source and target distributions (De Bortoli et al. 2021 relaxes this limitation, but the method itself is relatively complicated.
SD flow, on the other hand, (in our opinion) a simpler and more intuitive method that also solves the Schrödinger bridge problem (see Section 2.3 and places no restrictions on the distributions p and q. It is therefore also capable of performing interpolation between arbitrary data sets. The results of one such interpolation experiment are shown in Figure 3 The figure actually shows two interpolation experiments: The first evolves 1024 points of the "Swiss roll" data toward the "mystery" distribution (Section 7.2.2) in R3, while the second evolves from the "mystery" distribution to the "Swiss roll." The same cosine variance schedule as in Section 7.2.2 was employed.
C.3 Nearest- Neeeht- Analysis
It is important to note that SD flow does not cause the synthetic data to collapse to nearest neighbors in the target distribution. In Figure 4 we show the distribution of distances from points in the synthetic distribution to their first nearest neighbors in the target distribution (shaded in green) for the particle- optimization experiment described in Section 7.2.2 Note the overlap with the distribution of distances between target data points and their first nearest neighbors (excluding themselves) in the target distribution. Overfitting to the target data would result in large concentration of mass near zero for the synthetic data.
C.4 Model Optimization
Although we do not rum experiments on high-dimensional image data in the present work for the reasons described in Section 7 we report here an experiment using the moddlopptimization application (Algorithm) 2 "high"=da "ddimesioonl data in 5500 Here the scare quotes acknowledge that this dimensionality is far lower
22

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Table 4: Average minimum CFD (final three columns) for the SD flow (SD), MMD gradient flow (MMD) and Stein variational gradient descent (SVGD) algorithms after 1000 iterations over five independent trials on the "mystery distribution" particle-otimizztion problem in I2 under the experimental conditions described
in Section 7.1
ADAGRAD BATCH CONST
ANNEAL OFFSET SD (OURS)
MMD
SVGD
N
N
N
N
0.0002
0.1020
0.2407
N
N
N
0.0026
0.7050
0.6994
N
N
Y
N
0.0004
0.1020
0.2407
N
N
Y
Y
0.0020
0.7050
0.6995
N
N
Y
N
N
0.0005
0.1186
0.2761
N
N
Y
N
0.0006
0.5953
0.7006
N
N
Y
Y
N
0.0005
0.1185
0.2762
N
Y
0.0006
0.5953
0.7006
Y
MNNA INNN N
INN
N
0.0146
0.1689
0.1804
Y
0.0136
0.8250
0.6060
Y
0.0157
0.1734
0.1725
Y
Y
Y
0.0172
0.8231
0.6051
Y
Y
N
N
0.0159
0.2086
0.2410
Y
Y Y Y INME MANNINA MANNNNNN
N
Y
0.0157
0.8348
0.6168
Y
Y
N
0.0167
0.2045
0.2359
0.0168
0.8404
0.6265
N
N
0.0039
0.0029
0.0059
N Y
0.0038
0.1945
0.0057
Y
N
0.0037
0.0033
0.0059
Y
N
Y
0.0042
0.1938
0.0060
Y
Y
N
N
0.0056
0.0039
0.0077
N
Y
N
0.0056
0.0847
0.0081
N
Y
Y
0.0056
0.0041
0.0082
N
Y
0.0056
0.0841
0.0083
Y
INN
N
0.0154
0.0972
0.0175
Y
Y
N
Y
0.0163
0.3141
0.0185
Y
Y
Y
N
0.0148
0.0972
0.0176
Y
Y
Y
0.0162
0.3117
0.0164
Y
Y
N
N
0.0146
0.1122
0.0242
Y
N
0.0162
0.2915
0.0228
Y
Y
Y
0.0165
0.1111
0.0222
Y
Y
Y
0.0149
0.2922
0.0238
than the thousands millions of dimensions typical in high-resolution image data, but is high enough exhibit the problematic, intuition-callln challennnnn characteristics of high-dimensional data in general.
Specifically, it is well known that as data dimensionality grows, the ratio of the distance of point to its farlhest neighbor, D Diax, and the distance to its nearest neighbor, Dmin, tends toward unity. The ratio D Dmin drops precipitously in lower dimensions before leveling off at around 30 dimensions and very slowly approaching an asymptote of one afterward (Beyer et al. 1999.. We therefore chose R as reasonable setting to challenge our approach in high dimensions.
We generated a ground-truth target distribution by randomly populating 50 25 matrix B with values drawn from (0, 0.25I) and 50-vector with values drawn from (/10, I). Target data samples from N(,, BB were then generated 18 by drawing samples IO, E 25 and forming B μ. The
18Techiccllly this is completely defined normal distribution, since BB of full rank.
23

---

Published in Transactions on Machine Learning Research (07/202) 2023)
Step
Iritslzzdoon
Step
Step
Stop 200
Stop 600
Figure 3: Top: Data-set interpolation via evolution of 1024 points from the "Swiss roll" distribution to the "mystery" distribution in R3. Bottom: The reverse interpolation, from the "mystery" distribution to the "Swiss roll distribution.
Target Synthetic
Density
0.6
1.5
05
Distance
Figure 4: Distribution of distances from synthetic (blue) and target (red) data points to their first nearest neighbors in the target distribution.
model parameters to be learned were 50 25 matrix B, initialized from 0.00., and a 50-vector p. initialized to all zeros. This model can be interpreted a single-layer linear neural network, but it is most important to note that it exactly matches the capacity of the data-generating model.
If we retained the input vectors & E E for the outputs x E X, then the task of learning the parameters would be fairly straightforward in the context of a regression problem on paired data {(5+)). But in the general IGM problem, we have only unpaired data to work with, so we assume that all information about the target data inputs is unavailable.
We performed 1000 steps of SD flow using Algorithm with constant noise schedule of times the average distance of the initial synthetic (base) distribution to first. nearest neighbors in the target distribution
24

---

Published in Transactions on Machine Learning Research (07/202) 2023)
Denelty
Earget
Synthelic Meer
Synthetic Verance
Distance
Figure 5: Model optimization results in R500 using a constant noise schedule. SD flow allows a parametric model to be learned that very closely matches the target mean (14 versus fu, left panel) and the elements of the covariance matrix (BB* BB center panel). Diagonals are included for reference. Nearest-neighhor analysis showed no overfitting of the data (right panel) but showed a slightly lower average distance to nearest neighbors in the target set than exhibited by the target data relative to itself.
with batch size of 1024, an SD flow step size of 1, and a regression learning rate of À = 10-3, Other than brief experimentation to set reasonable values, no effort was made lo optimize these hyperparameters.
(corrosponding to o² > 700) 19
The results of this experiment are shown in Figure Despite (or perhaps because of) a massive and constant injection of noise, SD flow successfully fit the target distribution. Analysis of nearest neighbors once again showed that SD flow did not overfit to the target distribution, although there was very slight shift toward lower distances between synthetic data and their nearest neighbors in the target distribution as compared with the target data's 's mearest-neighhon distances relative to itself.
This experiment provides a basic proof of concept for a much more general procedure, one that can benefit from more sophisticated model architectures specifically suited to the problem at hand. For instance, for image generation, the kernel- based specification of SD flow from Section 3.1 can be exchanged for the denoising-based specification from Section 3.2. allowing to take advantage of attention-equipped U-net denoising architectures. Furthermore, methods can be mixed and matched: A denoising model can be employed for the score component while kernel-basee estimate can be used for the 4. score component, for example.
19 found that using higher amount of noise somewhat improved convergence profile of the algorithm.
25


Reviews:
[{"review_id": "pnK0jQovFm", "reviewer": "Reviewer_XeUD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- Well written manuscript provides a good overview of the recent contributions for sampling intractable distributions and connects the proposed framework with existing literature.\n\n- The advantage of this approach compared to usual Langevin dynamics that admit a sampling with probability flow ODEs is that the target distribution in this setting does not need to be analytically defined (e.g. in terms of a potential), but can rather be represented by samples as in the numerical experiments in the Appendix. \n\n**Weaknesses:**\n\n- Although the authors connect the score difference flow to diffusion models, they do not provide the straightforward connection to the gradient flow dynamics of Fokker-Planck equation.\n\n- There may be practical/numerical limitations in the practical uses of the framework for high-dimensional data generation, but this by itself is not a problem. However it may be useful for the authors to mention which parts of the proposed framework hinder its application to high-dimensional data (if there is indeed any such limitation).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "\n\nThe authors introduce a new approach for implicit generative modeling (or for transforming/bridging samples from a source to a target distribution), called the **score difference flow**. The proposed framework transformation employs deterministic dynamics involving the difference of the logarithmic gradients of the two distributions. \n\nThe authors demonstrate that the proposed particle formulation reduces the KL divergence between target and transient density.\nTo ensure that the proposed score difference is computable they employ proxy distributions resulting from smoothing with multidimensional Gaussians  [9], and define sample dynamics for the noise corrupted instances of the involved distributions.\n\n The authors propose an alternative formulation for the score difference flow in terms of optimal denoising models of Gaussian corrupted distributions. A substantial part of the paper (and probably the most valuable contribution) is devoted to provide theoretical connections to generative adversarial networks (GANs) [Section 4], denoising diffusion models (DDMs) [Section 5], maximum mean discrepancy gradient flow (MMD) [Appendix D], which by itself is a valuable contribution. \n\nThe authors demonstrate the effectiveness of their proposed framework in the appendix through numerical experiments on toy models, which is sufficient given the other theoretical contributions of the paper.\n\nOverall, the paper is technically sound, well-written, and provides valuable theoretical connections that are of interest to theTMLR audience.\n\n\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Suggestions for updates:**\n\n**A]** I think the paper would become more complete if the authors make the connection of the score difference with the Langevin dynamics. In [1,5] the gradient flow structure of the Fokker-Planck was introduced with the KL divergence acting as a cost functional under the Wasserstein metric. \n\nThus for a (_time homogeneous_) diffusion process\n$$ dX_{t} = - \\\\nabla U(X_{t}) dt + \\\\sqrt{2} dW_{t},$$\nwhere $U(x) = - \\\\log p_{ss}(x)$ denotes a 'potential', and $p_{ss}$ is the target (stationary) density,\nthe score difference between a target (stationary) distribution and a instantaneous distribution\nappears by rewriting the Fokker-Planck equation that describes the evolution of Langevin dynamics in terms of the difference of logarithmic gradients of the instantaneous and stationary probability densities\n\n\n$$\\partial q_t = \\\\nabla \\\\cdot \\\\left( q_t \\\\nabla U + \\\\nabla q_t \\right) = -\\\\nabla \\\\cdot \\\\Big(  q_t \\\\left( \\\\nabla \\\\log p_{ss} -\\\\nabla \\\\log q_t  \\\\right)\\\\Big). $$\n\n\nThis can be seen as a Liouville equation with time-dependent velocity field the score difference $\\\\left( \\\\nabla \\\\log p_{ss} -\\\\nabla \\\\log q_t  \\\\right)$. The above equation corresponds to particle dynamics $dx = \\\\left( \\\\nabla \\\\log p_{ss} -\\\\nabla \\\\log q_t  \\\\right) dt$ in a similar vein as in the particle ODE framework (probability flow ODE) of [3, 2]. A recent related approach that claims to scale favorably with system dimension [10] considers a projected kernel density estimation for the score difference. \n\n\nHere the authors consider a _time-inhomogeneous dynamics_ ($\\mu(x,t)$, $\\sigma(t)$) where in principle there might be no stationary probability density associated with the SDE, and thus one may not be able to define the sample trajectories using the gradient flow structure of the Fokker-Planck equation. \n\n\nHowever I consider a mention of the equivalence of the two approaches in this time homogeneous case (when the target density is well captured by a potential) will provide a more clear picture of the contributions of the paper and the connections to the existing literature.\n\n\n**B]** For the aforementioned reasons, as I understand, I would assume that the convergence rates of the score difference flow should be the same to those of the probability flow ODE/Fokker-Planck approach. However in the discussion the authors mention *\"If our goal is to get from point A to point B as quickly as possible, ..\"*. I would consider that alternative transport based approaches may show faster convergence properties to the target distribution. For example the very recent framework of [12] claims to provide possibly even one step interpolation between the two densities. To be on the safe side, I would reformulate this statement.\n\n\n**C]** Here the authors consider the score difference between a stationary distribution (p) and a transient distribution (q) for the definition that reduces the KL divergence between the two probability densities. This resembles the optimal control formulation for diffusion processes introduced in [6, 7], where the authors consider the score difference between *transient* probability densities in an optimal control setting [8] that optimally minimises the KL divergence between the two transient solutions. It might be interesting to discuss the connections of the two frameworks. \n\n\n**D]** In the last paragraph the authors mention *\"SD flow supplies a link between IGM methods that..[provide] high sample quality, mode coverage, and fast sampling\"*. However I am not entirely convinced about this statement. I think the statement should be either reformulated, or the authors should provide links to numerical experiments/links that validate their claims.\n\n**E]** I wonder whether some variant of the score difference flow that employs some state-of-the-art score matching frameworks may be more suitable for image generation. I am not asking for additional experiments on generating images (or other high dimensional data), however I think some statement in conjunction to the appendix E.4 would be valuable to assess the usefulness of the proposed method for practical applications.\n\n**Minor suggestions:** \n\n- The flow of the text would become more coherent if the Section 5 (Relation to Denoising Diffusion Models) follows Section 3, before starting the extensive section on the relation to GANs.\n\n- I wonder whether it would be useful to extend the length of the article and include the numerical experiments in the main text, and also bring the relation to the MMD flow forward. Then the relations to GANs, DDMs, and MMD will be clustered.\n\n- The deterministic ODE process for sampling transient densities associated with SDEs was first introduced in [3], and was then applied in [2] and named “probability flow ODE”. There are already hints of this expression already in [4].\n\n- The figures would improve if the text in axis labels/ticks had bigger fonts.\n\n\n**References:**\n\n[1] Jordan, R., Kinderlehrer, D., \\& Otto, F. (1998). The variational formulation of the Fokker--Planck equation. *SIAM journal on mathematical analysis*, *29*(1), 1-17.\n\n[2] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., \\&\n Poole, B. (2020). Score-based generative modeling through stochastic \ndifferential equations. *arXiv preprint arXiv:2011.13456*.\n\n[3] Maoutsa, D., Reich, S., \\& Opper, M. (2020). Interacting particle \nsolutions of Fokker–Planck equations through gradient–log–density \nestimation. *Entropy*, *22*(8), 802.\n\n[4] Otto, F., \\& Villani, C. (2000). Generalization of an inequality by \nTalagrand and links with the logarithmic Sobolev inequality. *Journal of Functional Analysis*, *173*(2), 361-400.\n\n[5] Otto, F. (2001). The geometry of dissipative evolution equations: the porous medium equation.\n\n[6] Maoutsa, D., \\& Opper, M. (2021). Deterministic particle flows for constraining SDEs. *Machine Learning and the Physical Sciences, Workshop at the 35th Conference on Neural Information Processing Systems (NeurIPS).*\n\n[7] Maoutsa, D., \\& Opper, M. (2022). Deterministic particle flows for constraining stochastic nonlinear systems. *Physical Review Research*, *4*(4), 043035.\n\n[8] Kappen, H. J., Gómez, V., \\& Opper, M. (2012). Optimal control as a graphical model inference problem. *Machine learning*, *87*, 159-182.\n\n\n[9] Zhang, M., Hayes, P., Bird, T., Habib, R., \\& Barber, D. (2020, November). Spread divergence. In International Conference on Machine Learning (pp. 11106-11116). PMLR.\n\n[10] Wang, Y., Chen, P., \\& Li, W. (2022). Projected Wasserstein gradient descent for high-dimensional Bayesian inference. SIAM/ASA Journal on Uncertainty Quantification, 10(4), 1513-1532.\n\n[11] El Moselhy, T. A., & Marzouk, Y. M. (2012). Bayesian inference with optimal maps. *Journal of Computational Physics*, 231(23), 7815-7850.\n\n[12] Liu, X., Gong, C., & Liu, Q. (2022). Flow straight and fast: Learning to generate and transfer data with rectified flow. *ICLR2023.*", "broader_impact_concerns": "There is no Broader Impact Statement in the paper. \nThe proposed framework inherits the concerns on ethical implications of the works of diffusion-based generative modelling when applied to high dimensional image data.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nu8E2nHHVI", "reviewer": "Reviewer_tqcW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nI found the paper easy to follow. And, from my point of view, it is really important contribution of the work under consideration. In a rather simple and expressive narrative, the paper explains a rather difficult yet interesting topic. The Score-Difference flow that the authors propose is actually the gradient flow with respect to KL divergence functional. The existing literature devoted to this subject is either too theoretical or too specific and, more importantly, does not reveal the importance of gradient flows in machine learning community. Actually, the authors fill in the gap and show that SD flows secretly emerge in several generative modeling methods.\nConcerning practical and technical achievements, the authors propose a novel approach to approximate the score difference, which prescribes to corrupt the original distributions under consideration with Gaussian noise and model the SD flow with respect to these corrupted distributions. Thereby, it becomes possible to estimate the score difference by samples, and, at the same time, approximately simulate the original score difference dynamics.\n\n**Weaknesses**\n\nIn spite of a good overall impression on the paper, I found both theoretical and practical contributions to be limited. As I already note, the SD flows, which are the primal subjects of the paper, are known in literature and known to be the gradient flows with respect to KL divergence functional. There are existing papers which propose to utilize the gradient flows (w.r.t. KL or even f-divergence) for generative modeling [1], [2], [3], [4]. Of the special attention is paper [1]. The authors of [1] propose a framework for modeling gradient flows with respect to f-divergence, which reduces to KL divergence for $f(x) = x \\log (x)$. Taking it into account, the results of Section 2.2 (from the paper under review) are in concordance with Theorem 2.1 from [1]. Besides, the optimal non-saturating GAN discriminator analysis (section 4.3) resembles Lemma 2.3 from [1]. Based on this, I believe, that [1] and, probably, [2], [3], [4] should be properly cited in the paper under consideration and the contributions of the latter should be stated more accurately. The comparison with the aforementioned papers is also highly desirable, because they could model the same gradient flows as considered in the paper under consideration, but utilize other techniques for density ratio estimation. By the way, [1] and [2] also exploit the idea to regress the parametric generator with squared $l_2$ norm loss in order to align it with the subsequent steps of gradient flow (Algorithm 2 from the paper under consideration). \n\nApart from the shortcomings described above, I think that the organization of the paper is another point to concern. It seems strange that all practical and algorithmical sections are located in the Appendix. I believe that the content of the paper should be organized in such a way that both theory and practice are somehow covered in the main part of the manuscript. \n\nLastly, I suppose that the practical validation of the approach is somehow limited. There are only proof-of-concept synthetic experiments in relatively small dimensions (additionally, the only “high”-dimensional experiment considered in the paper is to fit the Gaussian distribution, which doesn’t seem to be convincing). From my point of view, the authors should somehow persuade the readers that their approach indeed worth to be applied in real applications. It doesn't have to be high-dimensional image applications (I understand that the proposed method seems to scale poorly to high dimensions), but still, in my opinion, the machine learning is more about applications, not theory. \n\n[1] Gao et. al., Deep Generative Learning via Variational Gradient Flow, http://proceedings.mlr.press/v97/gao19b/gao19b.pdf,  ICML’2019\n\n[2] Gao et. al., Deep Generative Learning via Euler Particle Transport, https://arxiv.org/pdf/2110.02787.pdf , MSML’2021\n\n[3] Feng et.al., Relative Entropy Gradient Sampler for Unnormalized Distributions, https://arxiv.org/pdf/2303.03714.pdf , arxiv’2021\n\n[4] Heng et. al., Generative Modeling with Flow-Guided Density Ratio Learning, https://arxiv.org/pdf/2303.03714.pdf, arxiv’2023\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper deals with the so-called Score-Difference flow (SD flow) - the gradient flow with respect to KL divergence functional. The authors analyze SD flow from the theoretical point of view, establish the connection between SD flow and non-saturating GAN optimization as well as Denoising Diffusion models, and provide practical algorithms for SD flow modeling, which are based on substitution of original data distributions with noised versions of ones. The algorithms are validated on synthetic low and intermediate dimensions experiments.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The general comments about what should be changed in the paper are in the Weaknesses section. Specific comments are below:\n\nPage 3: The Stein’s identity should be properly introduced or cited.\n\nPage 4: It was difficult for me to understand the last paragraph. What does it mean, that “prescribed dynamics for the clean data are the same as for the corrupted data”? The corrupted distributions are not the same as original distributions, so the dynamics should be slightly different. So, this paragraph should be reformulated more clearly.\n\nPage 7: In the unnumbered equation, which follows equation (31), expectation sign is missing.\n\nPage 8: I don’t understand the alternative GAN loss $\\mathcal{L}^{\\text{alt}}$, introduced in equation (32). Why does the optimal discriminator for the case of   $\\mathcal{L}^{\\text{alt}}$ induce the SD flow, as in the case of non-saturating loss?\n", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CtPtUiH1Po", "reviewer": "Reviewer_SSdk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n* I think the link with diffusion models is interesting. There is potential here to build an interesting connection between continuous normalizing flows, the probability flow interpretation of diffusion models and score difference.\n\nWEAKNESSES:\n\n* The striking weakness for me is the lack of experiments. This paper deals with \"Generative Modeling\" and so should at least showcase how the interpretation of diffusion models and GANs in terms of Score Difference yield algorithms with better properties or at least provide a better understanding of their theoretical properties. I don't really see what new insights or properties of diffusion models/GANs can be inferred from their reformulation as Score Differences. If this paper is a theoretical paper I think that these contributions should be made clearer and that a proper theorem should be written about them. If the paper is a methodological/experimental paper then the method should be evaluated properly and compared with other methods for generative modeling and dataset interpolation. As of now, the only experiments are toy (3d up to 50d) with no baseline comparison.\n\n* The paper is not very well written. It is hard to decipher what are the contributions of the authors (see first point). For example, what is the main goal of Section 4? We start with a general decomposition of a loss in Section 4.1 and then move on to continuous normalizing flows in Section 4.2 to finish with the link with SD Flows in Section 4.3. There is no real connections between these sections. For example, what purpose serves Section 4.2?\n\n* I think the authors should do a better job at discussing the links between their approach and existing approaches. For example,  [1] explores another type of annealing (not the same as the mollified approach considered in this work). Overall, I think that a review of existing SVGD generalizations and how they relate to the current approach could be useful. Also the authors do not discuss the probability flow approach of diffusion models [2]. I think that this is more related to the current approach as there is no noise injected at each step in that case.\n\n* Regarding the data interpolation task, I think the authors should discuss and compare their approach with [3,4,5]. [3,4] are Schrodinger Bridge approaches. The authors comment in the main text that \"SD flow is capable of performing data-set interpolation in a manner similar to Diffusion Schrodinger Bridges while being more straightforward to apply\". However SD Flow does not solve the Schrodinger Bridge problem. In particular, one of the main point of [3,4] is that the obtained flow is an Optimal Transport map. I also want to point out that the idea of \"get from point A to point B as quickly as possible, we want to move not only toward B but also away from A\" is already present in Schrodinger Bridge approaches. Indeed if we consider a first forward process given by a Brownian motion going from distribution A to a noised version of A, $(X_t^0) = (B_t)$, then the backward process starting from distribution B is given $(X_t^1)$ satisfying $\\mathrm{d} X_t^1 = \\nabla \\log p_{1-t}^0(X_t^1) \\mathrm{d} t + \\mathrm{d} B_t$, where $p_t^0$ is the density of $X_t^0$. Then, the backward of this backward dynamics initialized at distribution A is called $(X_t^2)$ and satisfies $\\mathrm{d} X_t^2 = (-\\nabla \\log p_{t}^0(X_t^2) +\\nabla \\log p_{1-t}^1(X_t^2)) \\mathrm{d} t + \\mathrm{d} B_t$. Hence, we recover a score difference there too. Note that Schrodinger Bridge approaches go one step further by accumulating the scores (which are approximated by neural networks). If one is not interested in the Optimal Transport properties of the flow then [5] is a very efficient approach based on Diffusion models for Image to Image translation. \n \n* I found some of the claims to be not backed-up by experiments or theory. In particular, in the discussion the authors claim that their approach \"perform well on all three desiderata of the so-called generative learning trilemma -- high sample quality, mode coverage, fast sampling. High sample quality is not demonstrated in this work since no high dimensional experiment is proposed. Mode coverage is relatively illustrated in low dimensional settings with the coverage of a mixture of Gaussian distributions. Finally it is not clear at all that fast sampling is a property of SD flow. It would seem to suffer from the same drawbacks of diffusion models, i.e. multiple iteratives steps at inference times. While distillation strategies could be used it is not presented in the current paper.\n\n[1] D'Angelo, Fortuin (2021) -- Annealed Stein Variational Gradient Descent\n\n[2] Song, Durkan, Murray, Ermon (2021) -- Maximum Likelihood Training of Score-Based Diffusion Models\n\n[3] De Bortoli, Thornton, Heng, Doucet (2021) -- Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling\n\n[4] Chen, Liu, Theodorou (2021) -- Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory\n\n[5] Su, Song, Meng, Ermon (2022) -- Dual Diffusion Implicit Bridges for Image-to-Image Translation", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors introduce Score Difference, a new method to solve generative modeling problems. The score difference can be found as the direction of maximal descent in a forward KL flow. However, since the target distribution $p$ is not known and only accessible through samples, the authors define a mollified version of the target distribution by adding noise to it. Doing so and leveraging Tweedie's identity, they are able to write the score difference as a difference of ratio of expectation which are tractable. The authors then show the links between their approach and two popular generative models: GANs and diffusion models. Toy experiments are reported in the appendix.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The authors should address the main weaknesses highlighted in the previous section. \n* In particular I think that the paper could benefit from a deeper analysis of the link with diffusion models and SVGD.\n* A better experimental setting (image or other higher dimensional tasks with a comparison with baselines) is necessary to assess the experimental properties of SD Flows.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall I believe this work satisfies the TMLR acceptance criteria and thus should be accepted for publication at TMLR.  While I share the frustration of some of the reviewers (especially Reviewer SSdk) that the paper could be noticeably improved, in particular by providing more serious empirical evaluations, I do not think this undermines its adherence to the TMLR criteria in its current form.  The substantial updates that the author's made in the revision have corrected most of the other issues that I think would have prevented acceptance.

I would though like to make the following requests for minor revision before the final acceptance of the work:
- Provide some discussion on the current limitations of the approach for practical use in its current form and what would be needed to use it for real problems.  I am worried at the moment that it might be difficult for others to follow up from a more practical standpoint, and I think the current algorithmic limitations need to be made clear so that it does not end up being a "blocker" to future work (i.e. if the current paper is being accepted on the basis of being a theoretical contribution without providing proper empirical evaluation, it should not get in the way of future work being published that uses similar ideas from a more practical standpoint).
- Addition of the following citation requested by Review XeUD (who is not an author of the work): Wang, Y., Chen, P., & Li, W. (2022). Projected Wasserstein gradient descent for high-dimensional Bayesian inference. SIAM/ASA Journal on Uncertainty Quantification, 10(4), 1513-1532.
- If possible, change the colour scheme of the plots to avoid red-green clashes for people 
- Try to address the following comments made by Reviewer tqcW in their official recommendation:
> I am not fully satisfied with the clarifications given by authors to some of my questions. In particular, the phrase, that “the prescribed dynamics for the clean data y under SD flow are the same as for the corrupted data” (last paragraph of the Section 3.1) sounds slightly strange for me, because the dynamics for noised data is not the same (but close to) the dynamics of the original data. And the reparametrization trick analogy seems not applicable in this case. Secondly, I still don’t understand the idea with alternative GAN loss 
$\mathcal{L}^{alt}$ (eq. (37)). Why does the optimal discriminator for this particular GAN loss is the log density ratio?


Decision: Accept (poster)