Particle Semi-Implicit Variational Inference
Authors: Jen Ning Lim
Adam Michael Johansen
Venue: NeurIPS 2024 spotlight (2024)
Keywords: variational inference
gradient flow
Primary area: probabilistic_methods

Abstract: Semi-implicit variational inference (SIVI) enriches the expressiveness of variational
families by utilizing a kernel and a mixing distribution to hierarchically define the
variational distribution. Existing SIVI methods parameterize the mixing distribution
using implicit distributions, leading to intractable variational densities. As a result,
directly maximizing the evidence lower bound (ELBO) is not possible, so they
resort to one of the following: optimizing bounds on the ELBO, employing costly
inner-loop Markov chain Monte Carlo runs, or solving minimax objectives. In this
paper, we propose a novel method for SIVI called Particle Variational Inference
(PVI) which employs empirical measures to approximate the optimal mixing
distributions characterized as the minimizer of a free energy functional. PVI arises
naturally as a particle approximation of a Euclidean–Wasserstein gradient flow and,
unlike prior works, it directly optimizes the ELBO whilst making no parametric
assumption about the mixing distribution. Our empirical results demonstrate that
PVI performs favourably compared to other SIVI methods across various tasks.
Moreover, we provide a theoretical analysis of the behaviour of the gradient flow
of a related free energy functional: establishing the existence and uniqueness of
solutions as well as propagation of chaos results.

Particle Semi-Implicit Variational Inference
Jen Ning Lim University of Warwick Coventry, United Kingdom Jen-Ning wwwwwarwick....
Adam M. Johansen University of Warwick Coventry, United Kingdom a a w.m.johanseen@wrriickk ac.uk
Abstract
Semi-implicit variational inference (SIVI) enriches the expressiveness variational families by utilizing kernel and a mixing distribution to hierarchically define the variational distribution. Existing SIVI methods parameterize the mixing distribution using implicit distributions, leading to intractable variational densities. As result, directly maximizing the evidence lower bound (ELBO) not possible, so they resort to one of the following: optimizing bounds on the ELBO, employing costly inner-loop Markov chain Monte Carlo runs, or solving minimax objectives. In this paper, we propose novel method for SIVI called Particle Variational Inference (PVT) which employs empirical measures to approximate the optimal mixing distributions characterized as the minimizer of free energy functional. PVI arises naturally as a particle approximation a meelidean-Wassersteee gradient flow and, unlike prior works, it directly optimizes the ELBO whilst making parametric assumption about the mixing distribution. Our empirical results demonstrate that PVI performs favourably compared to other SIVI methods across various tasks. Moreover, we provide a theoretical analysis of the behaviour of the gradient flow of related free energy functional: establishing the existence and uniqueness of solutions as well as propagation of chaos results.
Introduction
In Bayesian inference, a quantity of vital importance is the posterior p(zly) = p(x, y)/ p(z, y) where p(z, 3) is probabilistic model, y denotes the observed data, and the latent variable. An ever-present issue in Bayesian inference is that the posterior is often intractable. This because the normalizing constant is available only in the form of integral, and approximation methods required. One popular method is variational inference (VI) (Jordan 1999: Wainwright and Jordan 2007 Blei al. 2017). The essence of VI is to approximate the posterior with member from variational family Q where each element of Q is a distribution (called "variational distribution") parameterized by 0. These parameters 0 are obtained via minimizing a distance or discrepancy (or an approximation of it) between the posterior p(-ly) and the variational distribution go.
Here, focus semi-inn ammi-implicit variational inference (SIVI) (Yin and Zhou 2018). It enables a rich variational family by utilizing variational distributions, which we refer to as semi-implicit distributions (SIDs), defined as
Dh.r (xr((x) (x) :- ( f k(x)2))(22)
where k: Rd x R2d R is kernel satisfying k(a]=)dæ = 1; r E P(R4:) is the mixing distribution and P(Idd denotes the space of distributions with support Rdd, with its usual Borel g-field, with finite second moments. Here, and throughout, we assume that the distributions and kernels of interest admit densities. SIDs are very flexible (Yin and Zhou 2018) and express complex properties, such as skewness, multimodality, and kurtosis. These properties might be present
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

in the posterior but typical variational families may fail to capture them. There are various approaches to parameterizing these variational distributions: current techniques utilize neural networks built on top of existing kernels (e.g., Gaussian kernels) to define more complex kernels Tiitsas and Ruiz 2019), and/or utilize pushforward distributions (a.k.a., implicit distributions (Huszár) (017]) (Yin) and Zhou 2018). On choosing parameterization, an approximation to the posterior is obtained by minimizing the exclusive as minimizing the free energy (or the negative evidence lower bound) defined as
(KL) divergence. This optimization has the same solution
q1,((2) p(z,8) v) qd.r(dr).
E(k, r) = log
(2)
However, since the integral in yk.r is typically intractable, directly optimizing & is not feasible. As result, SIVI algorithms focus on designing tractable objectives by using upper bounds of & (Yin and Zhou 2018): expensive Markov Chain Monte Carlo (MCMC) chains to estimate the gradient of & (Titsias and Ruiz 2019); and optimizing different objectives such as score matching which results in min-max objectives (Yu and Zhang 2023).
In our work, we propose an alternative parameterization for SIDs: kernels are constructed as before (with parameter space denoted by +) whereas the mixing distribution is obtained by optimizing over the whole space ((IRds). We motivate the case for minimizing a regularized version of the free energy & denoted by εx (see Eq. (4);; thus, SIVI can be posed as the following optimization problem: arg min ( Ex (0, r). As a means to solving the SIVI problem, we construct a gradient flow that minimizes E where the space A x P(Rd)) is equipped with the Seeliddann-assersteee geometry (Jordan et al. [998] Kuntz et al. 2023). Via discretization, we obtain a practical algorithm for SIVI called Particle Variational Inference (PVI) that does not rely upon upper bounds of E, MCMC chains, or solving minimax objectives.
Our main contributions are as follows: (1) we introduce a Euclidean Buclidean-Wassersteee Wasserstein gradient flow minimizing Ex as means to perform SIVI; (2) we develop a practical algorithm, PVI, which arises as a discretization of the gradient flow that allows for general mixing distributions; (3) we empirically compare PVT compared with other SIVI approaches across toy and real-world experiments and find that it compares favourably; (4) we study the behaviour of the gradient flow of a related free energy functional to establish existence and uniqueness of solutions (Prop. 8) as well as propagation of chaos results (Prop. 9)
The structure of this paper is as follows: in Section 2 begin with discussion of previous approaches to parameterizing SIDs and their relationship with one another. Then, in Section B we show how PVI is developed: beginning with designing well-defined loss functional, the construction of the gradient flow, and how to obtain a practical algorithm. In Section 4 we study properties of a related gradient flow; and, in Section 5 we conclude with experiments to demonstrate the efficacy of our proposal. For sake of brevity, we defer our discussion of related works to App.
2 On implicit mixing distributions in SIDs
This section outlines existing approaches to parameterizing SIDs with implicit distributions and how these choices affect the resulting variational family. Before we begin, we shall summarize the key assumptions of SIVI. The kernel is assumed to be a reparametrized distribution in the sense of Salimans and Knowles 2013 Kingma and Welling (2014): Ruiz et al. (2016). In other words, the kernel k is defined by the pair (ø, pk) where Φ= : R X ddd R and ps E ((R)) such that k(-lz) (zz, :) +>p; Furthermore, to ensure that it admits tractable density, the map + 6(2, e) is assumed to be diffeomorphism for all E Rd with its inverse map written o oosso1166 (z.-). From the change-of-varine variable formula, its density is given as k(-1z)) = pk ( (2, )) det Væ -2,0)(,,)) We sometimes write kppps to denote the underlying & and pk explicitly. Furthermore, the kernel A is assumed to be computable and differentiable with respect to both arguments.
Several approaches to the parameterization of SIDs have been explored in the literature. One can define the variational family by choosing the kernel and mixing distribution from sets K. and R. respectively, i.e., the variational family is Q(K, R) := {gk.r k E K,r E R). YYn and Zhou (2018) focused on a fixed kernel with r being a pushforward (or "implicit") distribution, i.e., E (. 9 E G} =: Rgpp. where G is a subset of measurable mappings from the sample space of pr to dd Thus, the variatanaa family is (((k), R. On the other hand, Titsias and Ruiz

---

(2119) considered a fixed mixing distribution r with & belonging to some parameterized class K. The typical example one in which each kernel is defined by composing an existing kernel ks.p, with a function f E F, the result is kippp.u (-12) == ))))()))).x (-1=) = Φi [=), # which clearly satisfies the reparameterization assumption. We denoted this kernel class as KFF.d.r. == = : F} and its respective TTR -er-iiiiaiioaal family is Q(Kg:o.pu {r}). In Yu and Zhang (2023), they combine both parameterization for IC and R. i.e., the Qraz- -ariiiaaati family is F.K.F.oo.p R. We note that this how the variational family presented Yu and Zhang (2023 see Sec. 2) but the authors used Qra variaatiol family in experiments, i.e., r was fixed. While Qraz might seem like it defines a larger variational family than the other approaches, under these common parameterization practices, we show that they define the same variational family.
Proposition 1 (Qraz = Quiz (:K.i Re.p.), ), then there a Qyiz- -aaiiiaiiaainaal family and Prn-variational family (i.e., Qm := @(KFosro-pu 1 {pr}) and Qviz = Q(tko.p.) J. RFoG.p.) such that Qraz
QTR.. Given Qruz-variational family of the form Qraz ==
Qysz QTa
The proof can be found in App. D This proposition shows that Gruz-paramnn Sou--arameteriztionn defines the "same" variational family as Qriz and QIR when we parametrize R with push-forward forward distributions. In practice, F and G are parameterized by neural networks hence Qyuz can be viewed as Qyiz or TTR with deeper neural networks F G. This simplification is direct result of using forwar distributions. Although this parametrization has shown promise e.g., Goodfellow et al. (2020)) they have issues with expressivity particularly when distributions are disconnected (Salmona et al. 2022). In our work, we follow in Qyaz variataiaai families, but, we avoid the use of push forward distributions. Instead, we propose to directly optimize over P ((TRd.) and so, our variational family does not simply reduce to Qyaz or QTR.
3 Particle Variational Inference
In this section, we present our proposed method for SIVI, called particle variational inference (PVI). Similar to prior SIVI methods, the algorithm utilizes kernels (denoted by ko) with parameters Q which satisfy the assumptions listed in Section 2 One example is hgg E (e:g.. where θ is function space induced by neural network. We slightly abuse the notation θ to also indicate its corresponding weight space Rd The novelty of this algorithm is that, for the mixing distribution, we directly optimize over the space ((R)) which loosens the requirement for the neural network in the kernel to learn complex mappings. The result is a "simpler" optimization procedure and increases expressivity over existing methods. Thus, the variational parameters of PVI are (0, θ x P( =: .M with its corresponding variational distribution defined as qa.r := f kk)-12)r(22da PVI arises naturally as a discretization of gradient flow minimizing a suitably defined free energy on θ x ((Rd.) ) endowed
with the Euclidean- Wasseestein geometry (Jordan et al. 1998, Ambrosio et al.) 2005 Kuntz et al..
2023). ). In Section 3.1, we begin by constructing suitably defined free energy functional; then, in
Section 3.2. we formulate its gradient flow; finally, in Section 3.3 we construct PVI from its gradient flow.
3.1 Free energy functional
As with other VI algorithms, we are interested in finding variational parameters that minimize (0,r) + KL(ga...P( p(-1))) This optimization problem can be cast equivalently as:
(x) dr. p(r,y)
arg min E(0,r), where E M R (0,r) + (0.r)eM
ge.r(()) log
(3)
Before we can solve this problem, we must ensure that it is well- In other words, it must admit minimizers in M. In the following proposition, we outline various properties of E:
Proposition 2. Assume that the evidence is bounded log p(y) co and k is bounded; then we have that & is (i) lower bounded, (ii) lower semi-continuous (.s.c.), and (ii) non-coercive.
The proof can be found in App. E. Prop. 2 tells us that even though E possesses many of the
properties one looks for in a meaningful minimization functional, it lacks coercivity (in the sense of Dal Maso (2012) Definition 1.12)): sufficient property to establish the existence of solutions. The key to showing non-coeryivity is that can construct kernel (o that does not depend on
3

---

At first glance, this issue might seem contrived but we note that this problem is closely related to the problem of posterior collapse (Lucas et al. 2019 Wang et al, 2021). To address non-coereivity, we propose to utilize regularization and define the regularized free energy as:
log (0)r(x)) p ( v)
r0, (x))
(0,,))
where R is a regularizer with parameters X. In Prop. 3 we show that if Rx is sufficiently regular, then the Ex enjoys better properties than its unregularized counterpart E.
Proposition 3. Under the assumptions of Prop. 2 Rx is coercive and ls..., then the regularized free energy Ex (i) lower bounded, (ii) Iss.c. (iii) coercive. Hence it admits at least one minimizer in M.
The proof can be found in App.
E.2 From here forward, we shall focus regularizers of the form R (0, r) + XL, Po) + X (6) where À = {Ap, Ag) are the regularization parameters and Po predefined reference distribution. As long as Rg is I.s.c., coercive and λØ X 0, the resulting regularizer R will also be l.s.c. and coercive. There are many possible choices for Po and Rg- For Po. this regularizes solutions of the gradient flow toward it; as such, in settings where there is some knowledge or preference about at hand, we can set Po to reflect that. In our experiments, we utilize Po (0, .M) where M is positive definite (p.d.) matrix. As for Rg, there are also many choices. In the context of neural networks, one natural choice is Tikhonov regularization 1 -1 || resulting in weight decay (Hanson and Pratt 1988) for gradient descent (Loshchilov and Hutter 2019) which is popular method for regularizing neural networks. In our experiments, we either use Tikhonov
regularization or its simple variant ||]| = (0,
3.2 Gradient flow
To solve the problem in Eq. (3. we construct a gradient flow that minimizes Ex. To this end, we endow the space M with suitable notion of gradient V V_MEx(0,r)
(V Ex V,Ex) where Vo and V, denotes the Euclidean gradient and Wasserstein-2 gradient (Jordan et al.) 1998). respectively. The latter gradient is given by V, (0,r) == V. (rV [x0,r)) where V. denotes the standard divergence operator and 6, denotes the first variation which is characterized in the following proposition.
Proposition 4 (First Variation of Ex and R)). Assume that (x1-) log (x. p(x
< for all (0,r)
M; then the first variation of Ex is 8, 8, &8,R, where 6, (()))))((== (x)x) [log and (, A, logr/Po-
(x,((x)
p(X
The proof can be found in App. E.3 Thus, the (Euclidean-Wasse Wasserstein) gradient flow of Ex is
0. = (0,,))
(be, (x) = (
A: = -V, V. (r
We now establish that the above gradient flow dynamic is contractive and that if log- Soboley inequality Eq. 7 holds, one can also establish exponential convergence. The log-Sobolev inequality is closely related to Polyak-Lojasiewicz inequality (or gradient dominance condition) and is commonly assumed gradient-based systems to obtain convergence (for instance, see Kim et al. (2024)). This is formally stated in the following proposition and proved in App.
E.3
Proposition (Contracting Gradient Dynamics). The free energy Ex along the flow Eq. is non-increasing and satisfies
(1 rt) VV (2,, <0 dt
66)
where | (EEx(0,,))) = | (0, || 8 V 10, 11. Moreover, if log- Soboley Inequality holds for constant T E Ryo, i.e., for all (0,7) M, we have
(0 - & (
where E == inf(er))M Ex(O,r); then we have exponential convergence (0,, Ex < exp((t/77)(,(000r0) -22).

---

Typically direct simulation of the gradient flow Eq. ( is intractable as the derivative of the first variation of R involves V. "=logry rri instead, is useful to identify the gradient flow with a McKean- Vlasov SDE, for which they share the same Fokker-Planck equation. The key distinction is that the SDE can be simulated without access to V= log rr. This SDE, which we term the PVI flow, given by
d02- = - VgEx(0,, dt, dz = b(O,, r2, Zx) dt + 2X, dw,
(8)
where r; = Law (ee), the drift is ((0, - =v0,,
X, V= log po (with the first variation given Prop. 4 and We a d dd-dimensional Wiener process. connection between the Langevin diffusion, i.e., d., V log p( Z, y) dt 22W,, and PVI flow can be observed with the fixed kernel (p(rz(z) = 5g (dx) and X, 0, namely, they both satisfy the same Fokker-Planck equation.
3.3
A practical algorithm
Algorithm 1 Particle Variational Inference (PVI)
Input: initialization (0n, 7Z J
JMM Imm)): regularization parameters {)o, ),); step-sizes hg and hy; number of Monte Carlo samples (for Eqs. 11 and 12)); and preconditioner V (I² W.))
for k = to K do rk 1 37 -M1
87 1,m
0x <- 0x-1 ho VoE (0k-1, M
D. See Eq.
11
by + Z + - o Th M (1) + X v (Z
b> See Eq.
12
for m to M do
Zkm + I + h, v br (Zx (2 + YI
D Mm
NO0 la.)
end for end for return (0k, (Zx, M
To produce a practical algorithm, we are faced with several practical issues. The first issue we tackle the computation of gradients expectations for which using standard automatic differentiation is insufficient. The second problem is that these gradients are often ill-conditioned and have different scales in each dimension. This is tackled using preconditioning resulting in adaptive stepsize. Finally, to produce computationally feasible algorithms, we show how to discretize the PVI flow in both space and time. PVI summarised in Algorithm I
Computing the gradients. In the PVI flow, both the drift of the ODE and SDE include gradient of an expectation with respect to parameters that define the distribution that being integrated. Specifically, the terms that contain these gradients are VVε and &x Fortunately, these gradients can be rewritten expectation (as described in Prop. (6) for which the parameters being differentiated w.r.t. is only found in the integrand (see derivation in App. G.I..
Proposition 6. If ó and i are differentiable, then we have
VoE(0,r) = =Epa(e))(() (Vooo [so.r - (z))(z,)) V V28,E (2) =Epu(c) (V200 [sg.r - s2)),,)))
(9)
(10)
where Ved dox denotes the Jacobian (Voσ) Oy (and similarly for oo)); scores are (0,r(2,c) (z,c) == v log (0o (z, E)) (and similarly sp(2,e)) ) and denotes the usual matrix-vector multiplication the sense of M (2,e) M e))
From Eqs. 9 and 10
produce Monte Carlo estimators for the gradients, i.c..
1 4r0(0,)) = T,
Σ Ezar (V [(Vzho [sa,r - (2,
(11)
1.0,800,1] = T,
1
Σ
(v - sp)((e)))
(12)
1=1
5

---

i.d. Pt. This is an instance of a path-wise Monte-Carlo gradient estimator; performant estimator that has been shown empirically to exhibit lower variance than other standard estimators (Kingma and Welling, 2014 Roeder et al., 2017 Mohamed et al. 2020).
where {ei]=1
Adaptive Stepsizes, One of the complexities of training neural networks is that their gradient is often poorly conditioned. As result, for certain problems, the gradients computed from Eq. ( and Eq. (T) can often produce unstable algorithms without careful tuning of the step sizes. When this occurs, we utilize preconditioners (Staib et al. 2019) avoid this issue. Let vr O + Rdo xde and 4r" : I2 + Rd= xds be the precondition for components & and r respectively, then the resulting preconditioned gradient flow given by
d0,- = - (Ot,
O,r, = V.
(13)
If Th and V are positive definite, then Ex(O,, re) remains non-increasing, i.e., Eq. 66) holds. As before, this Fokker Plaker-Planck equation is satisfied by the following Mckean- SDE:
d0, = -V (0.) VoE (z),
(14)
d7 = (2,) Tv. %,) + V. V (Z))
(Z)dd,
(15)
where (V.- VV'); = Σラ= 0.,,
12,1(v) [(V)) Dayl and r. = Law( The equivalence between Eq. 13 and Eqs. (14) and [1) is shown in App. G.2 simple example for the preconditioner allows the 1/ and Ze to have different time scales; ultimately, this results in different step sizes. Another more complex example of preconditioner T is the RMSProp (Tieleman and Hinton 2012), and V/" we utilize a preconditioner inspired by RMSProp (see App. G.2). As with other related works (e.g., see Li et al. (2016)), we found that the additional term V. V can be omitted in practice: it has little effect but incurs a large computational cost.
Discretization in both space and time. To obtain an actionable algorithm, we need discretize the PVI flow in both space and time. For the space discretization, we propose to use a particle approximation for rr, i.e., for a set of particles }Zr. Im=1 with each satisfying Law(Z,
rt. we utilize the approximation r == π.. >m ōz which converges almost surely to r2 in the weak topology as M oc by the strong law of large numbers and a countable determining class argument (e.g., see Schmon et al. (2020 Theorem 1.1)). This approximation is key to making the intractable tractable, e.g., Eq. is approximated by 00.r MI > m= ko(a m.,)
One obtains a particle
approximation to the PVI flow from the following ODE-SDE:
dl = - v (0M r2 dt, Vm E [M] : dz = b(O M 2,, dt + /2A,, dW c.m:
where [M] := (1.... M).. As for the time discretization, we employ -uler-Maruyama discretization with step-size which (using an appropriately defined preconditioner) can be decoupled into different stepsizes for O, and Z, denoted by di and h, respectively.
4 Theoretical analysis
We are interested in the behaviour of the PVI flow (8. However, a key issue in its study is that the drift in PVI flow might lack the necessary continuity properties to analyze using the existing theory. In this section, we instead analyze the related gradient flow of the more regular functional
E Egg. (x) llog (x)r +
(0.1))== + r), where (0, =
(16)
p(a, y)
for 7 > O. A similar modified flow was also explored in Crucinio et al. (2024) for similar reasons: they found empirically that, at least when using tamed Fuler scheme, setting problems in practice. Similarly, our experimental results for PVI found 7 = 0 did not have issues. To provide an additional measure of confidence in the reasonableness of this regularization and of the use of this functional as a proxy for Ex, we establish that the minima of Ex converge to those of Ex in the 7 0 limit.
0 did not cause
Proposition (o-convergence and convergence of minima). Under the same assumptions as Prop. 3 have that E7 I'converges to Ex as 7 + 0 (in the sense of Def. B.1.. Moreover, we have as immediate corollary that
inf (0, = lim inf (0, (0
7-0 (0

---

The proof uses techniques from I convergencce theory (introduced by De Giorgi, see e.g. De Giorgi and Franzoni (1975); see Dal Maso (2012) for a good modern introduction) and can be found in App. F.1 The gradient flow of Ex which we term 7-PVI, is given by
d0p = -
dt, d2, = (0 22) dt + 2X, dW,,
(17)
V= logPo. The derivation follows where r? Law ( and (7 r) = -V,
similarly to that in Section 3.2 and is omitted for brevity. The use of Y > 0 is crucial for establishing key regularity conditions in our analysis. We proceed by stating our assumptions.
Assumption 1 (Regularity of the target p. reference distribution Po. and Ra). We assume that log p(v) bounded; and p. Po and Re have Lipschitz gradients with constants Kp. Kpo KR, respectively: there exists some Bp R such that log p(y) I Bp: and for any given y there cxists a Kp R30 such that V/ por, y) V log'(x', v)ll I Ky x'|l for all x. x' E R& (similarly for po and R₂).
Assumption (Regularity of k). We assume that the kernel h and its gradient is bounded and has Ky-Lipschitz gradient; i.e., there exist constants BR, K& E Rso such that [ko(x]=)] + Bx. and (v=k(((22)
[[V (x) for all 0, 8 O 2, e' E Rdd and z, x' R
Vako (2'12')) Kx(11(0, 22 (6', x' ,²,2") 22)) hold
Assumption 3 (Regularity of e and Pt.). We assume that ø has Lipschitz gradient and bounded gradient. In other words, there is ag E Rzo,by E Rz0 such that 6 satisfies (V(o.))((z; c) ((0 (z',e)ll e)llF S (aøllell + ([I( all (0, =), (0', z') E O Rd= E Rd,, where || - || denotes the Frobenius norm. We also assume that Pt has finite second moments.
(0', 2')l) and (V e)ll S (aolle + bτ) for
Assumptions and are intimately connected; under some regularity conditions, one may imply the other but we shall abstain from this digression for the sake of clarity, Assumptions 2 and 3 are quite mild and hold for popular kemelss such as kg(x)2) 22) N2; pz()); Σ) under some regularity assumptions on Hig and Σ (which we show in App. C. These assumptions are key to establishing that
the drift in Eq. 17 is Lipschitz continuous (see Prop. 11 the App.), from which, we establish the
existence and uniqueness of the solutions of Eq. 17
Proposition (Existence and Uniqueness). Under Assumptions 7
to 3
if
> and
Ep (2, - 8p(2,,)]| is bounded (with *6 (2,6) - V logige. (0(2, + 7)); then,
given (fo,ro) E M, the solutions to Eq. 17 exists and is unique.
The proof can be found in App. F2 Under the same assumptions, we can establish an asymptotic
propagation of chaos result that justifies the usage of particle approximation in place of r? in Eq. (17)
Proposition 9 (Propagation of chaos). Under the same assumptions as Prop. we have for any fixed T:
lim E sup |00 67.-0 + w2 ((r))aM, )
=0
M-00 rE(o.T]
where (17) -.. (77): A.M Law (777 J.M ...s and iinh are solutions
dop.M (0) ,rt.M) dt,
dt, where r.. 1
5
M m==
Vm E [M] : d2 = 67 (07.M
MM v ZZ,M)a + /2A, dW.,
The proof can be found in App. F3 Having established the existence and uniqueness of the P-PVI flow, well as an asymptotic justification for using particles, now provide numerical evaluation to demonstrate the efficacy of our proposal.
Experiments
In this section, we compare PVI against other semi-implicit VI methods. As described in the App. these include unbiased semi-implicit variational inference (UVI) of Titsias and Ruiz (2019). semi- implicit variational inference (SVD Yin and Zhou (2018), and the score matching approach (SM)
7

---

of Yu and Zhang 2023). Through experiments, we show the benefits of optimizing the mixing distribution: compare the effectiveness of PVI against other SIVI methods on a density estimation problem on toy examples; and, we compare against other SIVI methods on posterior estimation tasks for (Bayesian) logistic regression and (Bayesian) neural network. The details for reproducing experiments as well as computation information can be found in App. The code is available at https://githuu: http::///gttbbb.ccm//mmgggg/g/
5.1 Impact of the mixing distribution
True
PV/
PViZero
Densities
Particles
Constant
Push
Skip
LSkip
Comparison PVI and PVIZero on a bimodal mixture of Gaussians for various kernels. The plot shows the density q8,r from PVI and PVIZero as well the KDE plot of from PVI described by 100 particles.
Figure 1:
From Prop. [ can be said that ultimately current SIVI methods utilize (directly or indirectly) a fixed mixing distribution whilst PVI does not. We are interested in establishing whether there is any benefit to optimizing the mixing distribution. Intuitively, the mixing distribution can utilized to express complex properties, such as multimodality, which the neural network kernel ko then exploit. the mixing distribution is fixed, this means that the neural network must learn to express these complex properties
which can be difficult (Salmona al. 2022). This intuition turns out to hold, but for the kernel to exploit an expressive mixing distribution, it must be designed well. We illustrate this using the distributions (M,
(- I) for == 21, 2,4) 4} and following kernels: the "Constant" kernel ((2, 12); "Push" kernel A (fe(2),, o2 "Skip" kernel M fe(2, o242) and "LSkipp" kernel N(W fo(=), T2) where W E R222. We compare the results from PVI and PVIZero (PVI with h, = to result in a fixed r = >/ I2) to emulate PVI with a fixed mixing distribution. As μ gets larger, the complexity of the kernel (or the mixing distribution) must grow to express this (e.g., see (Salmona et al. 2022 Corollary 2)).
Fig. shows the resulting densities and the learnt mixing distribution of PVI and PVIZero for different kernels and various μr. For the constant kernel, PVI can solve this problem by learning a complex mixing distribution to express the multimodality. However, for the push kernel, it can be seen that as μ gets larger PVI and PVIZero suffer from mode collapse which we suspect is due to the mode-seeking behaviour of using reverse KL and why prior SIVI methods utilized annealing methods (see Yu and Zhang (2023 Section 4.1)). As remedy, we utilize a Skip kernel which can be seen to improve both PVI I and PVIZero. In particular, both PVI and PVIZero were able successfully express the bimodality in f 2; however, PVIZero falls short when = while PVI can express the multimodality by learning a bimodal mixing distribution. Since Skip requires d, dz, show that LSkip (which removes the requirement) exhibits a similar behaviour to Skip.
5.2 Density estimation
We follow prior works (e.g.,
. Yin and Zhou (2018) and consider three toy examples whose densities are shown in Fig. 2 (they are given explicitly in App. H.2 In this setting, we use the kernel neural network whose
<x(=)>) N(r; 2 fo(z), 1) with d, dz architecture can be found in App. H.2 As a qualitative measure of performance, Fig. 2 shows the resulting approximating distribution of PVT which can seen to be close match to desired distribution. To compare methods quantitatively, report the (sliced) Wasserstein distance
where fa(z)

---

True PV1
0.03
63010
0.02
0000
X-Shape
Multimodal
Banana
Figure 2: Contour plots of the densities ge.r (in blue) against the true densities (in black) for various toy density estimation problems. We also plot the absolute difference the density of qa.r and the true density, i.c., [ge, pl.
(computed by POT (Flamary et al. 2)21)) and the rejection power of state-of-the ofttte--rttt-tttt-- two-sample kernel test (Biggs et al. (023) between the approximating and true distribution [ The results reported are the average and standard deviation (from ten independent trials of the respective SIVI algorithms). In each trial, the rejection rate is computed from 100 tests and the sliced Wasserstein distance is computed from 10000 samples with 100 projections. matches the distribution, the rejection rate will be at the nominal level of 0.05. It can be seen that PVI consistently performs better than SIVI across all problems. PVI can achieve rejection rate near nominal levels across all problems whilst other algorithms can achieve good performances on one but not the other. The details regarding how the Wasserstein distance calculated and the hyperparameters used can be found in App. H.2
the variational approximation
Problem
PVI
UVI
SVI
SM
Banana 0000..o//0.77...
0.07o.02/ 0.110..0.
1..3o.o/0..14p
0.390.24/0.22 /0..244
Multimodal 1..5..o/0.055.
0.650.23/O.. 20..223/0.166 10.0....0000..
0.....o/0..0o..o
0.140.05/O.
X-Shape
0.06g.03/00 0..0
0.23o.16/ 1.230.1e/01100 00..0.0
0.1lo.04/0.120.00
0.150.11/0.11o.0
Table 1: This table shows the rejection rate p and average (sliced) Wasserstein distance for toy density estimation problems. It is written in the format p/w (lower is better) with the subscripts showing the standard deviation estimated from 10 independent runs. We indicate in bold when the rejection rate minus the standard deviation is lower than the nominal level 0.05, and the algorithm that achieves the lowest Wasserstein score.
Bayesian logistic regression
As with others (Yin and Zhou 2018). we consider Bayesian logistic regression problem on the waveform dataset (Breiman and Stone 1984). The model is expressed as y w ~ (ernouli(Sigmooid(( (z., w)) with prior
R21 (0, 0.01-1 x I22) where (y. w) f0, the response and covariates, and w == [1, w is the covariates with appended one for the in- tercept. The "ground truth" is composed of posterior samples generated from rumning Markov chain Monte Carlo (MCMC) samples in YYn and Zhou (018). We use the kernel kg(0(z) = N(W= fe(z) exp( 1 MMs + M, where exp denotes the matrix exponential which ensures posi- tive definiteness. In Fig. 3 we visually compare certain statistics of the distribution obtained from MCMC and distribution obtained from SIVI methods. Fig. 3a shows the pair-wise marginal posterior distributions for three weights æ1, X2, I' chosen random from MCMC and SIVI approximations; and in Fig. (3.be compare the correlation coefficients obtained from MCMC against SIVI methods. can be seen that PVI obtains approximation close to MCMC with most other SIVI methods obtain- ing similar performance levels (with the exception of SM). See App. H.3 for all the implementation details.
5.4 Bayesian neural networks
compare our methods with other baselines on sampling the posterior of the Bayesian neural network for regression problems on range of real-world datasets. We utilize the LSkip kernel Kg (z|22 Nz: Wz fo(z), (z) In Table 2
Following prior works (e.g., Yu and Zhang 2023

---

MCMC
PVI
UVI
SVI
SM
22
11
2
33
21
T2
23
.1
22
23
T1
22
33
(a)
1.0
1.0
1.0
0.5
NCMC
0.0
0.0
0.0
0.0
-1.0
1.0
-0.5
0.0
0.5
-10
-0.5
0.0
0.5
1.0
-10
-0.5
0.0
0.5
3.0
-1.01
-1.0
-0.5
0.5
0.5
1.0
SVI
(b)
Figure 3: Comparison between SIVT methods and MCMC on Bayesian logistic regression problem. (a) shows the marginal and pairwise approximations of posterior of the weights 21; 22 F3, and (b) shows the scatter plot of the correlation coefficient MCMC (y-axis) vs PVI (x-axis).
show the root mean squared error the test set. It can be that PVI performs well, least comparable, with other SIVI methods across all datasets. The details regarding the model and other parameters be found App. H.4
Dataset
PVI
UVI
SVI
SM
Concrete (Yeh. 2007)
0.43p.03
0.50g.03
0.50n.04
0.92o.o6
Protein (Rana 2013)
0.87 0.05
0.92g.04
0.92g.04
1.02g.03
Yacht (Gerritsma et al. 2013)
0.13p.02
0. 0.18
0.170.02
0.98
Table 2: Root mean square error (lower better) for Bayesian neural networks on the test set for various datasets. Here, we write the results in the form Ho where 4 is the average RMS and is its standard error computed over 10 independent trials. We indicate in bold the lowest score.
6 Conclusion, Limitations, and Future Work
In this work, we frame SIVI as minimization problem of Ex, and then, as solution, we study its gradient flow. Through discretization, we propose novel algorithm called Particle Variational Inference (PVI). Our experiments found that PVI can outperform current SIVI methods. At a marginal increase computation cost (see App. HD compared with prior methods, PVI can consistently perform better (or at least comparably in the worst cases considered) which we attribute to not imposing a particular form on mixing distribution. This key advantage of PVI compared to prior methods: by not relying upon push-forward mixing distributions and instead using particles, the mixing distribution can express arbitrary distributions when the number of particles is sufficiently large. Furthermore, it is not necessary to tune the family of mixing distributions obtain good results particular problems. Theoretically, we study related gradient flow of E and establish desirable properties such as the existence and uniqueness of solutions and propagation of chaos results.
The main limitation of our work is that the theoretical results only apply the case where y > 0: yet, our experiments were performed with y 0 as this is when Σx corresponds to the (regularized) evidence lower bound. In future work, one can address these limitations by reducing this gap. Furthermore, we found that certain kernels were more amenable than others when exploiting an expressive mixing distribution (e.g., the skip kernel). The question of designing these kernels for PVI (or SIVI more generally) is important for future work.
10

---

Acknowledgments
JNL. gratefully acknowledges the funding of the Feuer International Scholarship Artificial Intelli- gence. AMJ acknowledges financial support from the United Kingdom Engineering and Physical Sciences Research Council (EPSRC; grants EP/R03471ON 10/1 and RPPT004134/1 and by United King- dom Research and Innovation (UKRI) via grant EPP/0014500/ as part of the ERC Synergy project OCEAN.
References
Ambrosio, L., Gigli, N., and Savaré, G. (2005). Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media.
Arbel, M., Korba, A., Salim, A., and Gretton, A. (2019). Maximum Mean Discrepancy gradient flow. In Advances in Neural Information Processing Systems, volume 32.
Biggs, F., Schrab, A., and Gretton, A. (2023). MMD-Fuse: Learning and combining kernels for two-sample testing without data splitting, In Advances in Neural Information Processing Systems, volume 37.
Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. (2017). Variational inference: A review for statisticians. Journal of the American Statistical Association, 111(518):859-8777 877.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke,
A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q (2018). JAX: composable transformations of Python+NumPy programs.
Braides, A. (2002). Gamma-Cone Convergence for Beginners. Oxford University Press.
Breiman, L. and Stone, C. (1984). Waveform Database Generator (Version 1). UCI Machine Learning Repository. DOI: http://ko..og//l.443333CCCC....
Caprio, R., Kuntz, J., Power, S., and Johansen, A. M. (2024). Error bounds for particle gradient descent, and extensions of the log-Sobolev and Talagrand inequalities. arXiv.
Carmona, R. (2016). Lectures BSDEs, stochastic control, and stochastic differential games with financial applications. SIAM.
Cheng, Z. Yu, L., Xie, T., Zhang, S., and Zhang, C. (2024). Kernel Semi-Implicit Variational Inference. In International Conference on Machine Learning, volume abs/2405.1899)
Chizat, I.., Peyré, G., Schmitzer, B., and Vialard, F--X. (2018). An Interpolating Distance Between Optimal Transport and Fisher-Rao Metrics. Foundations of Computational Mathematics, 18(1):1- 44.
Crucinio, F. R., De Bortoli, V., Doucet, A., and Johansen, M. (2024). Solving a class of Fredholm integral equations of the first kind via Wasserstein gradient flows. Stochastic Processes and their Applications, 173:104374.
Dal Maso, G. (2012), An Introduction to I-convergence, volume 8. Springer Science & Business Media.
De Giorgi, H. and Franzoni, T. (1975). Su un tipo di convergenza variazionale. Ati Accad. Naz. Lincei Rend. Cl. Sci. Fis. Mat. Nat. (8), 58()))842-850)
Flamary, R., Courty, N., Gramfort, A., Alaya, M. Z., Boisbunon, A., Chambon. S., Chapel, L., Corenflos, A., Fatras, K., Fournier, N., Gautheron, L., Gayraud, N. T., Janati, H., Rakotomamonjy. A., Redko, I., Rolet, A., Schutz, A., Seguy, V., Sutherland, J., Tavenard, R., Tong, A., and Vayer, T. (2021). POT: Python Optimal Transport. Journal of Machine Learning Research, 22(78):1-8.
Fournier, N. and Guillin, (2015). On the rate of convergence Wasserstein distance of the empirical measure. Probability Theory and Related Fields. 162(3):707-738
11

---

Gallouët, T. O. and Monsaingeon, L. (2017). A JKO Splitting Scheme for Kantorovich- Fisher-Rao Gradient Flows. SIAM Journal on Mathematical Analysis, 49(2): 100-1130.
Gerritsma, J., Onnink, R., and Versluis, A. (2013). Yacht Hydrodynamics. UCI Machine Learning Repository. DOI: https//cbo.or//0022333//XX77
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A.,, and
Bengio, Y. (2020). Generative adversarial networks. Communications of the ACM, 63(11))139-114
Graves, A. (2016). Stochastic Backpropagation through Mixture Density Distributions. arXiv.
Hanson, S. and Pratt, L. (1988). Comparing biases for minimal network construction with back- propagation. In Advances in Neural Information Processing Systems, volume 1.
Horn, R. A. and Johnson, C. R. (2012). Matrix Analysis. Cambridge University Press.
Huszár, F. (2017). Variational Inference using Implicit Distributions. arXiv.
Jordan, M. I. (1999). Learning in Graphical Models. MIT press.
Jordan, R., Kinderlehrer, D., and Outo, F. (1998). The variational formulation of the Fokker-Planck equation. SIAM Journal on Mathematical Analysis, 29(1):1-17
Kim, J., Yamamoto, K., Oko, K., Yang, Z., and Suzuki, T. (2024). Symmetric Mean-field Langevin Dynamics for Distributional Minimax Problems. In Proceedings of The Tweifth International Conference Learning Representations.
Kingma, D. P. and Welling, M. (2014). Auto-Encoding Variational Bayes. In Bengio, Y. and LeCun, Y., editors, 2nd International Conference on Learning Representations, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Truck Proceedings.
Kondratyev, S., Monsaingeon, L., and Vorotnikov, D. (2016). A new optimal transport distance on the space of finite Radon measures. Advances in Differential Equations, 21(11/12).
Korba, A., Arbin-Frankowski, P.-C., Majewski, S., and Ablin, P. (2021). Kernel Stein discrepancy descent. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5719-5730. PMLR.
Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., and Blei, D. M. (2017). Automatic differentia- tion variational inference. Journal of Machine Learning Research, 18(14):1-45.
Kuntz, J., Lim, J. N., and Johansen, A. M. (2023). Particle algorithms for maximum likelihood training latent variable models. Ruiz, F., Dy, J., and van de Meent, J.-W., editors, Proceedings of The 26th International Conference Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 5134-5180. PMLR.
Lambert, M., Chewi, S., Bach, F., Bonnabel, S., and Rigollet, P. (2022). Variational inference via Wasserstein gradient flows. In Advances in Neural Information Processing Systems, volume 35, pages 14447.
Li, C., Chen, C., Carlson, D., and Carin, L. (2016). Preconditioned stochastic gradient Langevin dynamics for deep neural networks. In Proceedings the AAAI conference artificial intelligence, volume 30.
L., Liu, Q., Korba, A., Yurochkin, M., and Solomon, J. (2023). Sampling with Mollified Interaction Energy Descent. In The Eleventh International Conference on Learning Representations.
Liero, M., Mielke, A., and Savaré, G. (2018). Optimal Mntropy-Trrnspor problems and a new Gellinger-Kantororn distance between positive measures. Inventiones mathematicae, 211(3))969- 1117.
Lim, J. N., Kuntz, J., Power, S., and Johansen, A. M. (2024). Momentum particle maximum likelihood. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp. F. editors, Proceedings of the 4/st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages
PMLR
12

---

Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In International Conference on Learning Representations.
Lucas, J., Tucker, G., Grosse, R., and Norouzi, M. (2019). Understanding posterior collapse in generative latent variable models.
Mohamed, S., Rosca, M., Figurnov, M., and Mnih, A. (2020). Monte Carlo Gradient Estimation in Machine Learning. Journal of Machine Learning Research, 21(132):1-6 62.
Morningstar, W., Vikram, S., Ham, C., Gallagher, A., and Dillon, J. (2021), Automatic differentiation variational inference with mixtures. In International Conference on Artificial Intelligence and Statistics, pages 3250-3258. PMLR.
Rana, P. (2013). Physicochemical Properties of Protein Tertiary Structure. UCI Machine Learning Repository. DOI:
Roeder, G., Wu, Y., and Duvenaud, D. K. (2017). Sticking the landing: Simple, lower-variance gradient estimators for variational inference. In Advances Neural Information Processing Systems, volume 30.
Ruiz, F. J. R., Titsias, M. K., and Blei, D. M. (2016). The Generalized Reparameterization Gradient. In Advances in Neural Information Processing Systems, volume 29.
Salimans, T. and Knowles, D. A. (2013). Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression. Bayesian Analysis, 8(4):837 -82.
Salmona, A., De Bortoli, V., Delon, J., and Desolneux, A. (2022). Can push-forward generative models multimodal distributions? In Advances in Neural Information Processing Systems, volume 35, pages 10766-10779.
Santambrogio, F. (2015). Optimal transport for applied mathematicians. Birküuser, NY, 55(58-63):94.
Schmon, S. M., Deligiannidis, G., Doucet, A., and Pitt. M. K. (2020). Large-sample asymptotics of the pseudo-marginal method. Biometrika, 108(1) 11):3511..1
Shiryaev, A. N. (1996). Probability. Number 95 Graduate Texts in Mathematics. Springer, New York, second edition.
Staib, M., Reddi, S., Kale. S., Kumar, S., and Sra. S. (2019). Escaping saddle points with adaptive gradient methods. In Chaudhuri, K. and Salakhutdinov. R., editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5956-5965. PMLR.
Tieleman, and Hinton, (2012). Lecture 6.5-rmsprop. coursera: Neural networks for machine learning. University of Toronto, Technical Report, 6.
Titsias, M. K. and Ruiz, F. (2019). Unbiased implicit variational inference. In Chaudhuri, K. and Sugiyama, M., editors, Proceedings of the Twenty-Second International Conference Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 167-176. PMLR.
Wainwright, M. J. and Jordan, M. (2007). Graphical Models, Exponential Families, and Variational Inference. Foundations and Trends@ in Machine Learning, 1(1-2)11-305.
Wang, Y., Blei, D., and Cunningham, J. P. (2021). Posterior collapse and latent variable non- identifiability. Advances in Neural Information Processing Systems, volume 34, pages 5443- 5455.
Yan, Y., Wang, K., and Rigollet, (2024). Learning Gaussian mixtures using the Wasserstein-Fishee Rao gradient flow. The Annals of Statistics, 52(4).
Yeh, I.-C. (2007). Concrete Compressive Strength. UCI Machine Learning Repository. DOI: http://tb..ooy//.44442cCPC...
13

---

Yin, M. and Zhou, M. (2018). Semi-implice variational inference. In Dy. J. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5660-5669. PMLR.
Yu, L. and Zhang, C. (2023). Semi-implicit variational inference via score matching. In The Eleventh International Conference on Learning Representations.
14

---

A Related work
In this section, we outline four areas of related work: semi-implicit variational inference; Euclidean- Wasserstein gradient flows and Wasserstein- graddient flows in VI; mixture models in VI; and finally, the link between SIVI and solving Fredholm equations of the first kind.
At the time of writing, there are three algorithms for SIVI proposed: SVI and Zhou (2018), UVI Titsias and Ruiz 2019). SM Yu and Zhang (023)) Concurrently, Cheng al. (2024) extended the SM variant by solving the inner minimax objective and simplified the optimization problem. Each had their parameterization of SID (as discussed in Section 2 and their proposed optimization method. SVI relies on optimizing a bound of the ELBO which is asymptotically tight. UVI, like our approach, optimizes the ELBO by using gradients based approaches. However, one of its terms is the score Vr log (2)) which is intractable. The authors proposed using expensive MCMC chains to estimate it; in contrast to PVI, this term is readily available to For SM, they propose to optimize the Fisher divergence, however, to deal with the intractabilities the resulting objective a minimax optimization problem which is difficult to optimize compared to standard minimization problems.
PVI utilizes the Maelidean-Wassersterr initially explored in the context of (marginal) maximum likelihood estimation by Kuntz et al. (2023) and their convergence properties are investigated by Caprio et al. (024). In Lim et al. (2024), the authors investigated accelerated gradient variants of the aforementioned gradient flow in Euclidean- Wasserstein geometry. The Wasserstein geometry for gradient flows on probability space has received much attention with many works exploring different functionals (for examples, see Arbel et al. (2019);
geometry. This geometry and associated gradient flows are
Korba et al. (202T); [i et al. (2023)). In the context of variational inference Lambert et al. (2022) analyzed VI as a Bures Wasserstein Gradient flow on the space of Gaussian measures.
PVI is reminiscent of mixture distributions which is a consequence of the particle discretization. Mix- ture models have been studied in prior works as variational distributions (Graves 2016 Morningstar jet al. (02T). In Graves (2016), the authors extended the parameterization trick to mixture distribu- tions; and Morningstar et al. (2021) proposed to utilize mixture models variational distributions in the framework of Kucukelbir et al. (2017). Although similar, the mixing distribution assists the kemel in expressing complex properties of the true distribution at hand (sec Section 5.1 which is an interpretation that mixture distribution lacks.
There is an obvious similarity between SIVI and solving Fredholm equations of the first kind. There is considerable literature on solving such problems; see Crucimio et al. (2024). which is closest in spirit to the approach of the present paper, and references therein. In fact, writing p(- = J k(-12, ())())d.
with k(- 0) = k=l=)z) makes the connection more explicit: essentially, one seeks to solve a nonstandard Fredholm equation, with the LHS known only up normalizing constant, constraining the solution to be in P(2) x {6g 0 E ). While Crucinio et al. (2024) develop and analyse simple Wasserstein gradient flow to address regularised Fredholm equation. neither the method nor analysis can be applied to the SIVI problem because of this non- con-iiiiial constraint. In Yan et al. (2024). the authors also solve a Fredholm-type equation but instead using the Wasserstein- Seteesteii-Fihhe---e Pisher-Rao geometry (Kondratyev et al. 2016 Gallouêt and Monsaingeon 2017 Chizat et al. 2018 Liero et al. 2018).
B I T-convergence
The following is one of many essentially equivalent definitions of Tconvergeecc (see Dal Maso
(2012); Braides (2002) for comprehensive summaries of I c-convergencc). We take as definition the tollowing (see Dal Maso (2012, Proposition 8), Braides (2002) Definition 1.5)):
Definition B.1 (I ((-convergence). Assume that A is a topological space that satisfies the first axiom of countability. Then a sequence F,, : + R is said to I--onverge to F if:
(lim-inf inequality) for every sequence (g).r)) E .M converging to (6,r) E .,, we have lim inf (0,,, r.)) 2 F(O,r).
7-+0
(lim-sup inequality) for any (0,r) E M, there exists sequence (0y. E M. known as recovery sequence, converging to (0,r) which satisfies
lim F,, (0,, F(0,r)
7-0
15

---

T- T-covvergence corresponds. roughly speaking, to the convergence of the lower semicontinuous enve- lope of sequence of functionals and, under mild further regularity conditions such as equicoercivity, is sufficient to ensure the convergence of the sets of minimisers of those functionals to the set of minimisers of the limit functional.
C On Assumptions 2 and 3
We consider the Gaussian kernel kin (3rl2) = (xx ())(z)()))) E), i.e.
0.5 exp (- (- (x- no (=)) (o(=)((()))
kg (x]z) = (2x)* d,/2 dett)) -0.5
where /ig R2 + Rd=; and E E IR xdx and is positive definite. this section, we show that Assumptions and are implied by Assumptions and
Assumption 4. ld is bounded and Σ is positive definite: there exists By E Rso such that the following holds for all (0, 2) E x Rd:
II (o))))) Bis
and for any x Rd \0, o. Ex >
Assumption 5. M4 is Lipschitz and has Lipschitz gradient, i.e., there exist constants K₁ E IRz0 such that for all (0, 2), (0', =²) E x Rdd the following hold:
[| per (z')ll S K,,l((0, 2) (0',2))ll,
(z) (o Kn)l(0,=) (0',
C.1 kg satisfies Assumption 2
In this section, we show that ky satisfies Assumption 2 We first show the boundedness property then the Lipschitz property.
Boundedness. First, we shall show that Ko is bounded. Clearly, we have (x(l)) [0, (2nr -d= (E) hence koo is bounded as consequence of Assumption 4 Now show that (o (x(22)] is bounded, have the following
[=))( -k(()l))-_( (=)))),
V (=/z())z)= = V u1 o? (ao(=))
[oko(r]z) = ((1
Hence, we have gradient of Gaussian density of given covariance w.r.t. is uniformly bounded. Thus, we have shown that ko satisfies the boundedness property in Assumption
(e:
< xo, from Assumption and using the fact the
Lipschitz. For ki, one choice coupling function and noise distribution is cbo(2, e) Eie n (=(=) and pk = (0, Ia_) where Σ be the unique symmetric and positive definite matrix with (Σ E (Horn and Johnson 2012 Theorem 7.2.6); and the inverse map is o, (2, = Σ- {x (2))) Thus, from the change-of-varialee formula, we have
(z V (P (z, (2, V 2 2z,
det 20 (2. (0, (2, x))]
dct(EE1/33) V (2,
(2,
- (0 (z,
where Σ-1 = det(E-1/2) Thus, we have
| (x|z) (x'
Σ 1 (
- - (2(,
< 2 ||
(z,x)) - -aPa (2',
(2,
(2,x) oØ (2', (2'x'')] x')ll
16

---

where C is constant and we use the following facts: || I
co
following from the fact Σ 1/2 is positive definite; Pa is a standard Gaussian density function with Lipschitz gradients; and that the inverse map Φ is Lipschitz from Assumptions and
(1,7(2,0) (=',-')) < |Σ || (2,, po(=)) - (a', po (o) (2')) & (,, 0'',)==''''
Hence, we have shown that kg satisfies the Lipschitz property of Assumption and Assumption holds for kg.
C.2 kp satisfies Assumption
One can compute the gradient
(0.2))00(2,)) = ((o.:)uO(=),
and hence | (V.e)) +) is bounded from Assumption 4
The Lipschitz gradient property is
immediate from Assumption [
Pk has finite second moments since a Gaussian with positive definite covariance matrix.
D Proofs in Section 2
Proof of Prop. We start by showing Qruz Qta. To this end, begin by showing the inclusion Qraz S QTa. lee, Q(K.reepe Rg.p.) S Q(KroG; {pr}). Let q E PoKr:: then there is some f E F and g such that B.p.......... From straight-forward computation, have
EENgeP py (-12) (a) Ezep, po (+1((2)) E Q(KFog:d.v (prh).
where (a) follows the law of the unconscious statistician, and the last element-of follows from the fact that &f (+I(())) off g(c). &Cooss..p- We follow the argument above in reverse to obtain the reverse inclusion. Hence, we have obtained as desired.
That Qyoz Qriz, follows in similar manner, which we shall outline for completeness: let q E @(K R. then
g qlyro 98P: Ezrger pp (+1=)] Be-sogar Ik (+22] Qriz-
One conclude by applying the same logic in the reverse direction.
E Proofs in Section 3
E.1 Proof of Prop.
2
Proof of Prop.
(E is lower bounded). Clearly, have
E00,r) r) = KL - logr(y) 2 - logp(y),
Hence, we have E(0,r) [- log p(v), oo) which lower bounded by our assumption.
(E is lower semi-continuous). Let (0.., rn)NEN be such that lim, Fn = r and lim 0₂ = 0.
We can split the domain of integration, and write E equivalently
p(x,y) y) 8(0rr) = / (1.00) (x.)(x)
qa.r p(z, v)
log
f0 (x) dx
(18)
5o
+ f 10.1
(par,y)
(40,((x) (((x,)) y)
log
g (2) dx
(19)
40,r (x)
20
We shall focus the RHS of (18

---

Note that we have the following bound
( p(z,3) 3)
log gon.tn (2)
(1,00)
2 on (z)
10n.rn (x))
p(uz, y)
p(x, y) )
$ max
0, log
9onrra (x)
S max C},
gen.r (x)
where C' is some constant. The last inequality follows from the fact that the evidence is bounded from above and the kernel is bounded. We can apply Reverse Fatou's Lemma to obtain
-f / [11000)
p(z,
(0 (x) p(x,
lim sup
log
gonirn (x) dæ
(0n.r. (x)
11-000
f lim (-1(1,oo)
p(r, y)
gowrr (x)
log
1onin (x)) dr.
(x)
((
Since we have the following relationships
p(a, v)
p(x,,)) y)
lim sup 11,,0)
S 11,
nn (2)
(0)
22+20
p 1)
plx, v)
lim - -log
log
n-000
40n.rn (x)
(0)
lim 90..7. qa pointwise, n n=000
where the first line from u.s.c. of [1)) i the second line from the continuity of log; the final line follows from the bounded kernel k assumption and dominated convergence theorem.
Thus, we have that
/ 11 [1,,00
p(a,,)) v)
log 90..., (xx) p(ar, y)
lim sup
Q...rn (x) dæ
10,.r. (x)
n-000
s / 111,00
p(2, y)
90.
log
98, (2) dr,
90.r (x)
p(x,
Using the fact that lim supn-soc
Inn
lim inc Ini we have shown that
/
p( y)
(x))
liminf inf n-+00
[(1,00))
log
p(z, v) qo (x) dx
@on.ra (x)
S s- (2,00)
(p(x, 1)
(0,r(x)) (x) dx.
log
(20)
g0,-
p( v)
Similarly, for the RHS of (9).. using Fatou's Lemma (with varying measure and the set-wise convergence of r.. and using the l.s.c. of [(o.1), we obtain that
lim inf / 110.1
p(r,,)
log (0n..r((())
90a.ra (x) dr
2-+00
1on.ra (x)
p(x,
f (0,1)
"p(x,3) y)
log (x)r
(x) dr.
(21)
g0
p(r,y) v)
Hence, combining the bounds
200 and
21I. we have that shown that
lim inf E(9n, ((0n,Tw) (w) n-020
lim inf log n-000
f1o5
goa.rr (Ir)
90 (z) dr
p(x,z)
f log
(x:)
p(x,8) v) 90, (z) dx E(0,r).
In other words, lower semi-continuous.
(Non-Coercivity) To show non-coercivity, we will show that there exists some level set {(0, r) (0 & 8) that not compact. We do this by finding sequence contained in the level set that does not contain a (weakly) converging subsequence.
18

---

Consider the sequence II = (0n, TnlN where 0n 8o: |10ull < oc; nn ōn: kz(x(2) = N(;; 8, Iaz); and p(a 8) Nax; La,). Clearly, we have ga,((x) Nx la.) and so (. = 2 @0|2. Hence, there a ao such that
E(O., ra) KL(g0..... p(-l)) logp(u) - 2 110611 p(v) 3.
Thus, we have shown that IT ((0,r) E(0,r) - 3). However, since the support of the elements of and hence of any compact set, 1I is not tight. Hence, Prokhorov's 's theorem (Shiryaev 1996, p. 318) tells us that, as II is not tight, it is not relatively compact. We conclude that, the level set is not relatively compact, the functional is not-coercive.
{Tn E P(R4))), eventually lies outside a ball of radius R for any R
E.2 Proof of Prop.
Proof of Prop. 3 (Coercivity) Consider the level set (0.r)) [0( $ 5),,, which is contained in relatively compact set. To see this, first note that
[(0,r) (0, < 8) [(0, logp(y) + (x S (3} ((0,r) (x S B + log p(y) By coercivity of Rx, i.e., the above level set is relatively compact hence εx is coercive. (Lower semi-continuity) Lower semi-continuity (l.s.c.) follows immediately from the l.s.c. of & and Rx.
(Existence of minimizer) The existence of minimizer follows from Dal Maso 2012 Theorem 1.15) utilizing coercivity and l.s.c. of Ex.
E.3 Proof of Prop.
Recall from Santambrogio 2015
Definition 7.12),
Definition E.1 (First Variation). If p is regular for F, the first variation of F P(TRd)) - R. if exists, the element that satisfies
F(p + ex) -(p)
f 8,F()(())x(dd))
lim c-50
for any perturbation X p- with P(IRd)) n 20 (Rd:) (see Santambrogio 2015 Notation)).
One can decompose the first variation of Ex
8, 8,E7 (0, + 8, (00
where 87 (0,r) + log (=..((=)+t 4 (dr). Since 8, 5, = (L its first varia-
tion follows immediately from standard calculations (Ambrosio al. 2005 Santambrogio 2015). As for E, we have the following proposition:
Proposition 10 (First Variation of E7), Assume that for all (0, r, E M x IR,,
[x0,(()+ +7 (1)m1(())))
E E&s (X(x)
log
p(X,v) < <00,
then we obtain
log (x)+ +
(X)
(0)((),)((=)) = Exa(X(
p(X,y)
(X) +
Proof. Since qa, = ko(-l=)(r + ex)(z) = 90,r + equ,x1 we have
(0.r+ex(x)) + dx
87 1 (x)
90,r+ox (2)rxx())oogg) log
p(U. x)
[ 14,.r + (x) log((4o.r aqo + dr
- Ige.r log
19

---

Applying Taylor's expansion, we obtain (z + log(2 + + 7) = zlog(0 + 7) + ey (log(x+ +7) + z47) + o(e), we obtain
3 200 = / (0. log (0,((x) + dx p(U,
+ e 1 (0x(20)
log 90. (0, +2 10-((3)
(0,r
+
dr + o(c).
p(y. x)
(0,( +
Hence, we obtain (0, + (x) -x) -0)
f (0x(()) log (0.(x))+
q0,r(a))
dx
p(u,2)
(0.)(x)) +
S 1/ ko (a(ol))x(de))
log (0).((x) +
(u.((x) ] dr
+
p(y,20 x)
qe (2) +7
/ (Jx (0(())))
log
(0.r(a)) +
(0))
dx x(2)
p(U, x)
(z)
+
One can then identify the desired result. In (a), we appeal to Fubini's theorem for the interchange of integrals whose conditions
S2/e ko (o((l)))
[og (0.r + p(y.z)
(x)
x(=) drdz < 00,
(22)
(0))(x) +
satisfied by our assumptions. This can be seen from
(xI=) log 20 +
(x,r(X)
LHS Eq. 22
+
(X) (x)++ + Ix(2)ld 50,
p(X,y) v)
where assumption and the fact that X absolutely integrable.
Proof of Prop.
The result can be obtained from direct computation. We begin
(.) + / 6,Ex10., (. dz
dt (0,. =
The second term be simplified
f 8, re) O. dz / ( re(())))
=- r2 rel| r1]= de
where the last inequality follows from integration by parts. Hence, the claim holds. If the log-Soboley inequality holds, then we have
d
d (0,,)r) = - - rell S - (x), -x
From Grönwalls inequality, we obtain the desired result.
F Proofs in Section
F1 Proof of Prop.
Proof. We first begin by proving I' I'coonvegeence directly via its definition, i.e., demonstrating that the liminf inequality holds and establishing the existence of recovery sequence. The latter follows from pointwise convergence:
lim (0. r0, 7-+0
upon taking (0y, = (0,r) r) for all
The liminf inequality be
follow similarly from the l.s.c. argument in App. E.1
To arrive the convergence of minima, we invoke Dal Maso
(012 Theorem 7.8) by using the
fact that E7 is equi-coercive in the sense of Dal Maso (2012 Definition 7.6). To see that E is equi-coercive, note that we have E Ex and & is I.s.c. (from Prop. 3 then applying Dal Maso (2012) Proposition 7.7).
Dal
20

---

F.2
Proof of Prop.
Proof of Prop.
We can equivalently write the 7-PVI flow in Eq. as follows
d(Or, Z.) = (0,, Law(Z,), 2,) dt dW
(23)
where a
and
0
v 22, Id.
- (0,E7 60,0r,
b : R P(Z) Rd + Rdo+d. (0,r, +
In App. F.4 we show that under our assumptions the drift 8a is Lipschitz. And under Lipschitz regularity conditions, the proof follows similarly to Lim al. (2024) which shall outline for completeness.
We begin endowing the space @ x P((T)=)) with the metric
d((0,r), r/)) - / 110 011 (2(g.g").
Let Y E C(O,, P()))) and denote T, (0) 2²²) for it's respective components. Consider the process that substitutes Y into (23). in place of the Law( Ze) and 0,.
a 2,-) (0, vf 2,7 dt + a dW.
whose existence and uniqueness of strong solutions are given by Carmona Define the operator
1.2].
Fr C( TJ, P(R())) C((O, P(R.)) Y (1+ (0) Law )).)).
Let (0,, Z.) denote process that is solution to (23) then the function & + (0,, Law Law(Zi)) is fixed point of the operator Fr. The converse also holds. Thus, is sufficient to establish the existence and uniqueness of the fixed point of the operator Pr. For Y = (0,v) and T' = (0',
=1-11==B// =11 (1,,, Ms. 2, (0',,
110 0, 112 ElI 112
1/g 2,- ds
S S10 S/" [111Z] -2-" 112 ||0, 82112 Wi(,s, [2) ds
$ C((1) / (W3(us, v/) 110 - 11]1'
where we apply Jensen's inequality; C, C,-inequality; Lipschitz drift of 8n; and Grönwall's inequality. The constant C. == 3バ? and C(t)) == 1C exp C). Thus, we have
d3 (FF( FT s C(t) L d' d.)
Then, for F/ denoting & successive composition of Fr, one can inductively show that it satisfies
(+C(t))
da (F
d2(M,, k! eE[0,]]
sup
Taking the supremum, we have
sup (T)) < (TC(T))* sE[o.]]
sup dT T.). sE(0,2]
k!
Thus, for a large enough k we have shown that E* is contraction and from Banach Fixed Point Theorem and the completeness of the space (C((0, T), θ x P(RR==), sup we have existence and uniqueness.
21

---

F.3
Proof of Prop. 9
Recall, the process defined in Prop.
do?. = - (0 :.t (f.MMddl dl, where r7
1 M
27
m==
Vm E [M] m2 = 6" (07 .7 7. 27, dt + 222, dW,,
M
and 7-PVI (defined in Eq. (77) augmented with extra particles (in the sense that there are M independent copies the Z-process) facilitate synchronous coupling argument
d0? (07 Law (2,1)) dt,
Vm E [M] dZ, = 6" (07, Law (221), Z7,m dt + v 2d, d.
Proof of Prop.
This is equivalent proving that
M
E sup EE TE
|00 07 11 1 + E
sup EE (0,77
Zi.
Zim 1|
o(1).
(24)
M
m==
We shall treat the two terms individually. We begin with (a) where Jensen's 's inequality gives:
1./0 (02 .ng VoE ds
(a) in
24 E sup tEfo,T
V0E ,. - VO (0)) r2) di
TE
< Ce
B1102 0g.h M 2 rr) dt.
(25)
where Co := 2TK?,
g7, we apply Gauchy-Schward and the C, inequality with the Lipschitz continuity Using the C, inequality again, together with the triangle inequality:
of VNEz from Prop. 12
EW rr) s 2EW? (r7. f2) + 2EW2
MM .2)
M
o(1) + M
Σ BIZ
(26)
a.m
m==1
where #3
1 ΣM
82zm.. and Fournier and Guillin (2015).
Note that also have
1107 - M1 <
sup "'E(0,
|08
0 M12
(27)
M
M
M
a.m
1|2
sup E[UTTT
M m=1
12 12 22,
(28)
m=l
Applying Eq. 26
in Eq.
25 then Eqs.
and
28
obtain
M
{- E sup
(a) < 2C,
103
sup O..
12 - s.m 1 2 ds + o(1).
(29)
M
se [0,77
Se (0,7]
m=1
Similarly, for (b) in 24
we have
M Σ / (7 1..M 2y.M 607 Law (22,), 27 a.m
(b) = sup
18
E (0.T)
M
m=
M
< C.E.
||0..M - 1 + W Law 2221)
Z 1|2 ds.
M
mm
22

---

where C2 == 3Kg and, as before, we apply Schwarz, Lipschitz and C, inequalities. Then
from Eqs. (28) we obtain
M Z7.m 112 + o(1)ds.
(b) = CE
sup sE[U,2]
107.Mr 071
sup aE|O,T]
(30)
M m==1
Combining Eqs.
29 and (30 and applying Grönwall's inequality, we obtain
M Z,.m 112 {- o(1)
E sup (E(T.
|00
02.M
E sup [c
1
1
M m=1
Taking the limit, we have the desired result.
F4 The drift in Eq. 17 is Lipschitz
In this section, we show that the drift in the y-PVI flow Eq. (17p is Lipschitz.
Proposition 11. Under the same assumptions as Prop. 8 the drift b(A,r) r) is Lipschitz, i.e., there exists a constant Kg Rzo such that:
| (0, 2 r 2 & (0, (6',, (f,,2))l+aa(r,r)) 40,0' e 0,2,2' E 2,r, E P(Z).
Proof. From the definition and using the concavity of V (which ensures that for any a, b Va+bb + b vā + vb), we obtain
| (0, ("'," r. = (VEE((0))) VOE
(1,30,,,; (00,
It is established below in Prop. 12 that VoET satisfies Lipschitz inequality, i.e., there is some Kez E 1R50 such that
(V, V V (0', r') Kez -110- - 0'| + Wa(r,r'))
is established below in Prop. 13 that b7 satisfies a Lipschitz inequality, i.e. there is some Kyt E Ryo such that
(b; - (" ('00','' 2')l < Kor (1((0, - (0', We(r,r))..
Hence, we have obtained as desired with K₃ = Kgg + K6o.
Proposition 12. Under the same assumptions as Prop. g the function (0,r) + (O is Lipschitz, ie., there exist some constant Kgz E RR such that
(V, (O', r')l & Ksz (110 0'll Wa(r,r)), ((0,r), E EM.
Proof. From the definition, have
(e,) VOE (o,)
(
Thus, if both V and V&Ra are Lipschitz, then so is VoET. Since R, has Lipschitz gradient (by Assumption [}. it remains to be shown that VOET is Lipschitz. From Prop. 6 we have
(0,) Bp((c)(())
[Vodo [s] sp]M2,e)] = Vods dz. (z, e) (dde))
where d (2, = (2) sp) Then, applying Jensen's inequality, we obtain
(V,)(00,)) - -VEE7(B', r'))
= // ple) / [Vorho (2,e)()) Vadhor (2, ()r(())
(z) dzde
p()) ///l///ol [Vooo 10, (2,((())
Vodo d (2, dz de.
(31)
23

---

Focusing on the integrand, we can upper- bound it with
Vodo (2,))))) - Vodo dy ()), (2,
dz
/ Vodo de (2, (r(=) ['s)]dz
+ +11/ V 0000 -0, (2, - Vodo (2,)
(2, c) r(())s
(6)
/ Vorbo dzf b) (m(z) - r''(=
/ Vosso (2, - Vogw dy (())dz.
where in (a) we add and subtract the relevant terms and invoke the triangle inequality, and in (b) we apply Jensen's 's inequality. Plugging this back into Eq. (31,, we obtain
(0,0 VOET(0',r"))
(pecc)) /| Vordo . (2, t) |r(2) - (=)=
(32)
/ Bpu(t) Vodo df. (2, Vodo dgi (z, (2. ((()dz,
(33)
where the interchange of integrals is justified from Fubini's theorem for non-negative functions (also known as Tonelli's 's theorem).
As we shall later show, the two terms have the following upper bounds:
KW1 r'), and K(10 0'll + Wi(r,')),
(34)
(35)
where K denotes generic constant; and, upon noting that W1 S W2, we obtained the desired result. Now, we shall verify Eqs. (34) and (35). For the Eq. (34). we use the fact that the map = +
Epe(e) Vadao (z,) is Lipschitz then from the dual representation of W1, we obtain the
desired result. To see that the aforementioned map is Lipschitz,
Vodo (2, <) Epr(c) Vogo (z
(a)
(e)
Vocio (2, Voca d2')
(6)
Ep((O)
Vo00(2,e) (2, ((8))
(c),(())))
+ (p)(e)
(Vobo(z,c)) Vodo -o)o0(")))))-(711
(0)
(p)(e)
Vode(2, 12) (6)
+ Ep)(e)
[Vooo(2,,) V0 e) |
(s', (s',
(8)
$ Epa(e)
[< + b2) ba)l || z'1l
[(aolle]] ||||||| + bb) (2', e) ||] ||> 2'11.
+ Ep((c)
(2)
(p)(e) [(axleel| + ||||| - ba)] ||= --=-1 z'1l
+ 2 (ette) (aollell + 2z)² + dg (2',22)))') E)
||]
|| 2'11,
where (a) we use the reverse triangle inequality: (b) we add and subtract relevant terms and apply the triangle inequality; (c) we use a property of the matrix norm with || || denoting the Frobenius
norm; (d) utilize Assumption 3 and the Lipschitz property from Prop. 15 (c) we apply Young's
24

---

inequality. Then, from the fact that
| < oc, (36)
[2]
Eppae) [[as|el]| + (adlle]] + ba]l < 00, Bpa(e) (aollell + bb)² + dB. (z', k)
which holds from the assumption that Ps: has finite second moments Assumption B and from our assumption that (e) do (=', e) is bounded. Hence, the map is Lipschitz and so Eq. (34) holds.
As for Eq.
B5), focus the integrand in Eq. 33
E pat(s)
V Voda de V Voog dg. (2,
< E (e
Vogo (d)p7 - -dg; (3, <) + (V000 Vodo dy. (2,
Vorbo (z, ))l [10)) (2, 11Vood Voga (z, (2, |1 || (z. )
< pa(e)
[as||e]] + (aul + ba)l (100 0'1 o(1+'((r;)))
+Bpole (apllell + b6)
db. (z, e) 110 0'll
where, for the last line. we apply Prop. 15 and Assumption have the desired result.
3
Applying Young's 's inequality and (36
Proposition 13 (b is Lipschitz). Under the same assumptions as Prop. g the map by is Kb-- Lipschitz, i.e., there exists a constant Kr E Ryo such that the following inequality holds for all (0, r0, (0', z'rr) E θ x Rd= x P(Rd))): (0, (8', r 222' S K& (Il(0,, - (6', (r(r,r').
Proof. One can write the drift by as follows (can be found in Eq. (43) similarly to Prop. o we have
(0, = (c) [V [s sp + (Z,,)ee,e) + Va logpu(z),
where (z, 2V..0.. (s.))(()(
Hence.
[Bb(0, - 63 (0',,',,=)) s [Bpate) (V :000 [d8, + (2, (V=00 Idb + (0,,)))( o]ll
| log Po(=) (ologo((2')
pp (e) 20 dø. + (2, (V (2=00 (d + DE.D)(E
K z'11.
where for the last inequality we use Jensen's 's inequality and Assumption] |
Since we have
20 [d37 + (7,,)) (V.d0 (d + +1 (2(,))(
(0)
3
(V.
(6 + + (z, V= (d) + -( VV
+E
+Ep.cell
(d0 + r3 1[ (V (d + ()rrr)
E pL V
+ (2, (d + Mr + (E),
+Ep. V V
de (2' +) [( +17 br (2',,)))
< E pa ||||| + (1)
d (z', a] + (2, - 17 - (2),
(37)
+ +Ep (as |||| + ba) (db; + + (= e) Ill (1) (0', 2')ll
(38)
where, for (a), we add and subtract the relevant terms and invoke the triangle inequality, in (b) use properties of the matrix norm, and in (c) we use the bounded gradient and Lipschitz gradient in Assumption 3 For Eq. (37): upon using Props. 14 and 15 which established below, obtain
E (asllll + by) (1d) (z, de (z' e)ll [1,2, e) - 1% (x',
SE Ee + baa + arl] + (ba + by (()) 2) - (6', (6',')1l + (')rr'')).
(39)
25

---

As for the second term, Eq. 38)
E px ()) (aslle]l + ba) + + (r
(n)
p. (o [|loolllll+++]@|l [op. (z', e) + 11 (2',
E Pust
(z', +) + Brl
- B (aجl|c] + )b)² + 2 l ( (2', O) + Br(asllell + bo)
(40)
where (a) follows from the triangle inequality, (b) we use Prop. bouuddedess of T, (c) we apply
Young's 's incquality to the first term. Similarly to Eq. (36). from our Assumption Jand our boundness
assumption of the score, we have as desired. Combining Eq. 39 with the result of plugging Eq. (40)
into Eq. 38 we obtain the result.
Proposition 14 (T' is Lipschitz and bounded). Under Assumption 2 the map (0, r,) + 1 (z, e) is Lipschitz and bounded. (Lipschitz) there are constants ap, br E 1k=00 such that following hold: (2, e) - To (z, (2,e S (ar|lell + (Il - (0', = ll + (r,)))))
(2,
Furthermore, it is bounded
(2,))l s Br.
Proof. Since 18 V.. (ogigu.(()++)) where x co (0 (2,,)),, and x' = de (2'220/000 (2), we have: (0,
[[T2 8,, (z,e) - ro (00,,((=''' (2',
(gorr (xx') + V. log((0,r (3)) + 7) (g0,- V log bog(gor,- (=0') ?) (40, + 7) (x') + +7)
$> Y
1
I00 (1') gu. (x) (x)] log(go, 7]
7
+ (1) + v) log(90.- 7) V log(gor, (x') + 7)
Bk (x')) (x) + (Bs + 7) (2,)) ,r (r( (z', c)]
72
7
Bx (1 + aolle||+ + (1) 2) - (8', +
72
+ B&
(0', 2')l + Wi(rr'))
where the last inequality follows from applying Prop. 18 and Assumption 3 to the first term and Prop. to the last term Hence, we have desired.
Boundedness follows from the fact that
(x.. 1yo (del.
S [[V 4.
Proposition 15. Under Assumptions to 3 the map (0, r0) + s (2, sp(2,t) 8 (2) satisfies the following: there exist aa, ba Rzo such that for all (0'',, ,4) E _,, 2 E IRd, we have
[0 - d (2',,)))) e)ll S (aallell + -)) (0')=')1 (r,),
Proof. Let x = 60(2, e), and x'= == oe (z', e). Then, we have
[1,, (2,e) dp:7 ,0)l W V log v) V, logp(=', v)ll
+ || log((a, (z) + 7) - v. log yd (x') + +)]
< Kyllx - x'll + V. log(g0,- (x) + 7) V. 1og(10.,--1 + +
W Kplas||e|| + (10 2) (0',2'11
+ (a.llell + (I) (0', 21)| + Wi(rrr'))
where Prop. and Assumptions and are used.
26

---

Proposition 16 (s is Lipschitz). Under Assumptions Bandy > 0, the map (6,r, + $0 (z, e) satisfies the following: there exist constants as ba E R20 such that the following inequality holds for all r),r). (6', r') E M, and 2, z' E Rd:e: (2,c) - (2' .)) $ (agllell + b,) 2) (0', 20)2')| W1(1, r')),
Proof. For brevity, we write I = (2( and x' = 000 , e); from the definition, we have
V Vo (I)
Vogorr' (a')
|| E) - (z',
98.r (x) +
q0 (x') +7
V Vx90. (x)
Vagorr' (x')
V Vo pr' (x') V Vzgor .r' (x')
10,r (4) +7
40,r (1) +7
Y0.r (1)) +7
qu,r" (x)) +
|1 V (x: Vagorr ir' (z')ll
98 (x) + 7
+ V V' (x')
90,r (x) + q0'r" (x') +
IA
1
Vaterr (x) - Vo (x')ll + 72 (x) - rr (z')l.
Bx
(41)
where (a) we add and subtract the relevant terms and the triangle inequality: (b) we use the fact that
V[ r' (x') || [ (="))("))))) (d)=))))) I Bg (from Cauchy-Sehward and the boundedness part of Assumption 2). Now.
deal with the terms individually. For the first term in Eq. (41), we
use the fact that the map (0, r, 2) + V (0ve (z,(z,)) e)) is Kiy-iischitz from Prop. 17 As for the
second term in Eq. (41). we apply Prop.
Hence, we obtain
Kgg +
Bg
(2,e) - (t',
(1((, - (0', 2'') + W2()rr))
72
gg + 722
Bg
(1 + asllell + bo) 2) - (0', + Wi(rr))),
where we use Assumption 3 for the last line.
Proposition 17. Under Assumption 2 the map (0,r, + (x) is Lipschitz. ie. for all e
there exists Kgg E RR such that the following inequality holds for all (0, r), (6'r') M and 2,2' E Rdd,
(x) Vagor, S Ka (1((,, x)) (8' x0',x')1l (r
Proof. From direct computation,
/ (o (2) Vako (x'le)) (z)]
V 10, (x) Va9o o (x')ll
(4)
1
V (ko - ko ko (2'1=) (2) dz
+
V ko (1' [2) |r- r'l (2)
(8)
(lg() (0', 2')l W2(r,r)).
where (a) we add and subtract the appropriate terms, apply the triangle inequality and the Cauchy- Schwarz inequality; (b) for the first term, we use the Lipschitz gradient Assumption 2 and for the second term, we use the dual representation of W1 with the fact map 2 + [2 log (x(=)))] is Ky-Lipschitz from the reverse triangle inequality and the Lipschitz Assumption 2
Proposition 18. Under Assumption 2 the map (,,, + (x) is Lipschitz, i.e., there exists some K2 E Ry such that for all (0, r, x), (0', ,','''')))))) x') E x P(Rd:) x Rds, we have
Ige. qo .r (x') < K2 (1) x) (0', x' || + Wi(r,r")).
27

---

Proof. From direct computation, we have
(g))(()) - -0', (x))) - (20,((z) - (0. + (x)) - (00))(=')) + [Ko[2|2) ko (ool"))))-k((a
1 (r,Wi( +1 (0,x
(42)
where (a) for the first term, we use the fact that the map 2 + [ko(=|2) is Bx-Lipschitz (from the bounded gradient of Assumption 2). and again the Lipschitz property of from the same assumption.
G Algorithmic details
G.1
Gradient estimator
Proof of Prop. 0 We show the derivation of the estimators in Eq.
10 Eq. 9 will follow similarly.
We have
log L (x)
V v=0,E10rr((2) = (.EEA(
ply, x))
log 2o (00
V,Eerpp
p(y. (z(
Assuming that 09 and pk are sufficiently regular to justify the interchange of differentiation and integration, we obtain
V v= = Eerup V= log (0((()(2))))
((2,0(2.c))
To obtain as desired, one can apply the chain rule.
Similarly, one can derive a Monte Carlo gradient estimator for V V as follows:
V V=0,E7(0,r1(3 V BRo(ele) log (0,-(x) +7 10.- (x)
p(y.z)
(0)(()) +
log (00 E)) +7
(0,(0((2,e))
V.Eeope
ply. (0(2,)))
(00(( +
As before, if de and pk sufficiently regular, obtain
(0.,(((2,)))
(= = Ee-pp
V log p(y, o(2,E))
(43)
(0s (2) 7)²
To obtain desired, one can apply chain rule.
G.2 Preconditioners
Recall that the preconditioned gradient flow is given by
d0; - V (e), Or,, V.
where 6,E r| 5,E[0,r] + logo We rewrite the dynamics of
0rr: Vz (5,E log + rggr;
V. (r. 5,, 0, log V (r, VV logrt).
The second term be written as
V V. logr,) V.
(V.V 2r)
(V [Trr V (r. w
28

---

Layers
Size
Input
din
Linear(din den da). LReLU Linear(dy, dl.). LReLU
dn
dn
Linear(dy, dout).
dout
Table 3: Neural network architecture defined by NN(din; duy dout).
where (V w.)
dz, [VV))
and last equality holds since
d:
d2
d2 (d.,
Σ
V V.)r) rt Σ Oz.
Hence, we have the following dynamies of rx:
d, = V. "re (V, 16, (0,, ri]
log V VD) V. (V. (Vrr)))
Examples. Following in the essence of RMSProp (Tilleman and Hinton 2012), we utilize the preconditioner defined as follows:
B 3Bx-1 (1- ())iaa(A((V rw] (Z) M 1=)))
V (Z) (Bx)
where B₁ Xd, and AA is aggregation function such the mean max. The idea is to normalize by the aggregated gradient of the first variation across all the particles since this is the dominant component in the drift of PVI. Similarly to RMSProp, it keeps exponential moving average the squared gradient which can then be used the preconditioner.
H Experimental details
In this section, highlight additional details for reproducibility and computation. The code was written JAX (Bradbury et al. 2018) and executed on a NVIDIA GeForce RTX 4090.
H.1 Section 5.1
Hyperparameters. For the neural network, we use fe NN(2, 128, 2) defined in Table the number of particles 100, d, = 2, K = 1000, hg = 10 -4, h2 = 10 A = 10-8, for we
RMSProp and set 1/"" Iz.
Computation Time. Each run took 8 seconds using JIT compilation.
H.2 Section 5.2
In this section, we outline all the experimental details regarding Section 5.2 Densities. Table 4 shows the densities used in the toy experiments.
Name
Density
Banana
N N(
1) 0,2)
1.8
2
-1.8)
)
X-Shape
0,
0,
1.8
2
1.8
2
Multimodal
+ W
!
2
(
(-2)
2 .7)
Table
Densities used toy experiments (see Section 5.2)
Hyperparameters. We set the number of parameter updates and particle steps to be K 15000,
and d, 2.
29

---

fe. We use fo = NN(2,512,
PVI. We use M 100, Ag RMSProp preconditoner, Vp" = - and I. 250.
0. X, 10-8, hr 10-2, ho = 10-1, Tpe we use the
SVI We use K 50 estimate the objective (Yin and Zhou 2018 Eq. (5)) which around the values used in Yin and Zhou (2018). We utilize RMSProp with step size 10-², and (V I. The implicit distribution is set to (0, J.
UVI. For the HMC sampler, we follow in Titssass and Ruiz 2019) and use 50 burn-in steps, with step-size 10-1 and 5 teap-frog steps. We use the RMSProp optimizer with stepsize (a.)
10 for kg. The implicit distribution is set
SM. For the "dual" function written as f in the original paper Yuu and Zhang 2023 see NN 512, 2). We utilize RMSProp with decaying learning rate from
Algorithm 1) 10-4 to 10-5 to optimize the kernel kg, and RMSProp with 10-3 (0-3 to 10-1 for the dual function f. The implicit distribution is set to (0, Ia,)-
Sliced Wasserstein Distance. We report the average sliced Wasserstein distance using 100 projections computed from 10000 samples from the target and the variational distribution.
Two-Sample Test. We use the MMD-Fuse implementation found in https:///ithhnn com/ antoninschhaa/mmdduus gitt
Computation Time. An example run on Banana with JIT compilation, PVI took 42 seconds, UVI took 10 minutes 36 seconds, SM took 45 seconds, and SVI took 38 seconds.
H.3
Section 5.3
In this section, we outline all the hyperparameters for each method used.
Hyperparameters. We use K = 20000 set d₂ 10. For all kernel parameters, we use RMSProp preconditioner with step size he 10-3.
fa. We use fo NN 512, 22).
PVI. We use M 100, A X, 10-8, h 10-2, and for Drr we use the one described in App. G.2 with mean as the aggregate function.
SVI. We use K 50 estimate the objective (Yin and Zhou 2018 see Eq. (5)) which are around the values used Yin and Zhou (0188
UVI For the HMC sampler, we follow in Titsias and Ruiz 2019 and use 50 burn-in steps, with step-size 10-1 and 5 leap-frog steps.
SM. We were unable improve the performance of SM with our chosen kernel and instead used the implementation in https:://// thub. conn//ongia/////SLS///uuu _eurcee catalyzex. com obtain posterior samples with implementation details found in the code repository and in the paper (Yu and Zhang 2023).
H.4
Section 5.4
In this section, we outline all the experimental details regarding Section 5.4
Model. We consider the neural network BNN .dhnn) defined (o) WT ReLU(W b1) + by where O E Badbun = [vcc (w2), b2, veo(W1), b,]T, W2 E RR x1 by E R. Wi E R ban xataa bj Rd Given an oututt pair Y : {(O,,, H) the model can be defined
as P x)) = (((z))p())) where the likelihood is (Y(z) NYY 0.013) and the prior is ((x; 0, 25I).
Datasets. For all the datasets, standardize by removing the mean and dividing by the standard deviation.
Protein. For the model, we use BNN 30) which results in the problem having dimension d 331. The dataset is composed of 1600 train examples, 401 test examples.
30

---

Yacht. For the model, we use (NN(6, 10) which results in the problem having dimension
81. The dataset is composed of 246 train examples and 62 test examples.
Concrete For the model, we (N, 10) which results in the problem having dimension dz 101. The dataset comprises of 824 training examples and 206 test examples.
Hyperparameters. We use 1500 set d, 10. For all kernel parameters, we use RMSProp preconditioner with step size ha 10 that decays to 10 following constant schedule that transitions every 100 parameters steps.
Jo, ag. We use So = NN(d2, 512, dz)) and 0g = SoNdplus(NN(d,,5222 dz)) + 10-8 and they share parameters except for the last layers.
PVI. We use M 100. Xg-
10 3, hg 10-3, and for yr we use the one described in App. G with mcan as the aggregate function.
SVI. We use K = 50 to estimate the objective (Yin and Zhou| 2018 see Eq. (5)) which around the values used in Yin and Zhou (2018). The implicit distribution is set to
r =O, Ta.)
UVI. For the HMC sampler, we follow in Titsias and Ruiz, 2019) and use 50 burn-in steps, with step-size 10-1 and 5 leap-frog steps. The implicit distribution is set to r = N(,0, La )
SM. For the "dual" function written as f in the original paper Yu and Zhang 2023 see Algorithm 1) we use (d 512, dz) and trained with RMSProp with stepsize 10 We tried decaying learning schedule to 10-4 but found that this degraded the performance.
We used ReLU activations instead as we found that using leaky ReLUU harmed performance. The implicit distribution is set to (N Id,).
Computation Time. For each run in the Concrete dataset with JIT compilation, PVI took 37 seconds, UVI took approximately 1 minute 40 seconds, SVI took 30 seconds, and SM took 27 seconds.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly define paper scope and contributions at the end of the introduc- tion with references to why they are accurate.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We highlight the limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

For each result, we outline relevant assumptions which can be found in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
These can be found in App.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer: [Yes]
Justification:
We provide code for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https htt////ooeSumissseeeeee httpl////gueeeeeeeeeebbiiieeses
7/nips.com wttp:ccpppbccccceee wwwpbbl/gguidee.///eee
swiiiiissipppccccccccoooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
These can be found App. H and explanations can be found in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In our results, report the mean and standard deviations from independent trials which can be found
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
34

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We outline the computational resource in App.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We conform.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
35

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release data or models that have a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, reference the dataset used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprswiiiiiie theode com datastts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
36

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects were used.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We did not have human participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "Ln5kefgrHD", "reviewer": "Reviewer_o93p", "summary": "The paper proposes PVI as a new method to conduct variational inference using the semi-implicit distribution. The method is to construct a gradient flow to minimize a regularized ELBO, which is practically implemented as the particle propagations. The empirical studies show the accuracy over density estimation and posterior predictions.", "questions": "Does the regularization in Eq 4 introduce bias in the posterior inference?\n\nDoes the theoretical analysis in Sec. 4 provide guidance in designing the algorithm? \n\nHow to check coercive and l.s.c. in Proposition 3?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea of using Wasserstein gradient flow to optimize the intractable ELBO of SIVI is novel. \n- The method has good accuracy in the provided simulations.\n- The techniques in the method derivation may be useful for other areas such as flows and diffusion models.", "weaknesses": "A major concern is about the simulations. First, it is unclear whether the comparisons between PVI and the baselines SVI, UVI and SM are fair. The accuracy of SIVI methods can often improve with an increase in the computation such as the number of samples in SVI and the MCMC steps in UVI. For a fair comparison, all methods need to be under the same computation budget. However, the current paper does not provide details of how the SVI, UVI, and SM are implemented. It would be better to include figures that show computation/time versus the accuracies for all the methods. \n\nCurrent simulations are relatively too simple and are only conducted on several 2D toy examples and UCI datasets. First, none of these simulations directly show the accuracy of the posterior inference for a complete Bayesian model. Second, all the simulations are in low-dimension. Does the particle method suffer from the curse of dimension problem? How does the number of particles scale with dimensionality to maintain accuracy?  It would be interesting to verify uncertainty estimation in high dimensions.\n\nLast, the paper writing is not coherent in some places. For example\n- The non-coercive is not defined in Proposition 2\n- The paper mentions the non-coercive \"is closely related to the problem of posterior collapse\"; what is the relationship exactly?\n- How to compute the precondition matrices in Eq 13 is not discussed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the effort you spent on our work.\n\n> However, the current paper does not provide details of how the SVI, UVI, and SM are implemented. \n\nWe perhaps didn't make this clear enough (and can address that in subsequent versions), but line 281 of the main text points out that all hyperparameter choices and runtimes for all methods are provided in Appendix H. We usually follow the recommended settings found for each competing method in their respective papers (except for when we did not find explicit recommendations). Source code for our implementations of all methods was provided at the time of submission.\n\nThere is a question of whether runtime is a meaningful measure of fairness given that there can be discrepancies based on how an algorithm is implemented. As the runtimes provided in Appendix H demonstrate, other algorithms were not disadvantaged. For instance, we followed the recommendations of UVI paper which resulted in an algorithm that exceeded our time computation by an order of magnitude. In the supplementary PDF, we include the requested figures for the density estimation task. To achieve high speeds with PVI, one takes advantage of the fact that the particle update is easily parallelizable.\n\n> First, none of these simulations directly show the accuracy of the posterior inference for a complete Bayesian model.  \n\nWe agree. We now provide (preliminary) experiments on the Bayesian Logistic Regressions setup studied in prior works [1, Section 5.4]. In the attached PDF, we provide plots that compare the quality of the posterior against MCMC samples obtained in [1]. It can be seen that PVI obtains a posterior quality that closely matches that of the MCMC samples.\n\n[1] Yin, Mingzhang, and Mingyuan Zhou. \"Semi-implicit variational inference.\" International conference on machine learning. PMLR, 2018.\n\n> Does the particle method suffer from the curse of dimension problem? How does the number of particles scale with dimensionality to maintain accuracy? \n>all the simulations are in low-dimension.\n\nAs noted in Appendix 5.3, the Bayesian neural network example has dimensionality {331, 81, 101} which one can claim is not \"low dimensional\". Like other methods, PVI utilizes neural networks to allow the method to scale to high dimensions while retaining the expressivity of the particle method.\nExperimentally, we found that $100$ particles were sufficient for good performance across the examples considered. We kept this constant throughout our experiments and did not finetune it. The source code for our implementations was provided at the time of submission.\n\n>Last, the paper writing is not coherent in some places \nThe non-coercive is not defined in Proposition 2.\n\nThe coercivity definition is a standard one for which we provided an explicit reference for it in the main text on line 128. We shall add the definition to the Appendix.\n\nWe will reread the manuscript carefully and address any similar issues; we'd be happy to address any specific concerns you may still have.\n\n> The paper mentions the non-coercive \"is closely related to the problem of posterior collapse\"; what is the relationship exactly?\n\nIn the amortized VI, one definition of posterior collapse is when the approximate posterior $q(x|z)$ does not depend on $z$ and collapses to the prior. In the context of PVI, one can prove that the functional $\\cal{E}$ is not coercive by using the kernel $k(x|z)$ that does not depend on $z$. Since the kernel is $k$ and approximate posterior $q$ often share the same functional form, e.g., they ($k$ and $q$) are often parameterized with $\\cal{N}(\\mu(z), \\sigma^2I)$ where $\\mu$ is some neural network, the comment suggests to the reader why it may be unsurprising if they share this issue. \n\n> How to compute the precondition matrices in Eq 13 is not discussed\n\nThe preconditioning matrix was discussed in Appendix G.2 for which we have provided a reference at line 209 which is within the main text.\n\n> Does the regularization in Eq 4 introduce bias in the posterior inference?\n\nYes. As with any non-parametric estimation problem with a finite sample size, regularization is an important algorithmic step. Loosely speaking, one can decrease the weighting of the regularization as the number of samples goes towards infinity to remain consistent.\n\n> Does the theoretical analysis in Sec. 4 provide guidance in designing the algorithm?\n\nNot directly. The primary purpose of the analysis is to provide \"theoretical underpinnings give a rigorous basis for the practical algorithm\" to quote Rev JYYZ.\n\n> How to check coercive and l.s.c. in Proposition 3?\n\nThere are various proof techniques that one can use to show coercivity and lsc. Our proposed regularizer satisfies the assumptions of Proposition 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UxdUmFTdpM", "reviewer": "Reviewer_38kT", "summary": "The authors propose a method for SIVI called Particle Variational Inference (PVI) which employs a particle approximation of an Euclidean–Wasserstein gradient flow.  PVI directly optimizes the ELBO, and it makes no parametric assumption about the mixing distribution. Their empirical results demonstrate that PVI performs favorably against other SIVI methods across various tasks. The authors provide a theoretical analysis of the behavior of the gradient flow of a related free energy functional.", "questions": "Line 329: what's the meaning of 'Here the ± denotes the average ...'?", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors provide extensive theoretical analysis to support the proposed method. The authors provide solid theoretical results along with the proposed method. The paper is well written.", "weaknesses": "Experiments on a real-world application could strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time spent reviewing our work.\n\n> what's the meaning of 'Here the ± denotes the average ...'?\n\nThank you for pointing out this mistake. The comment referred to the notation $\\mu_{\\pm \\sigma}$ where $\\mu$ is the average and $\\sigma$ is the standard deviation computed from $10$ independent trials. This shall be fixed in future iterations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J8yb1aQM4p", "reviewer": "Reviewer_JYYZ", "summary": "The authors introduce a novel algorithmic approach to fitting semi-implicit variational approximations. This method is based on discretizing a suitable gradient flow, and the paper provides a comprehensive theoretical analysis to support it. The approach, called Particle Variational Inference (PVI), directly optimizes over the space of probability distributions for the mixing distribution rather than optimizing a specific parametric form for the mixing distribution, as in previous SIVI approaches.\n\nNumerical examples demonstrate improvements compared to existing semi-implicit variational inference methods. The paper provides a solid theoretical foundation and a practical algorithm with beneficial properties over existing methods in the literature.", "questions": "1. How sensitive is PVI to the choice of kernel and number of particles?\n2. Can the theoretical analysis be extended to cover the γ = 0 case used in practice?\n3. How does PVI scale to higher-dimensional problems or larger datasets?\n4. Are there specific types of problems where PVI is expected to significantly outperform existing methods?\n5. How does the choice of preconditioner affect PVI's performance?", "limitations": "The authors have adequately addressed limitations.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper is a very strong submission, and I believe it is worthy of acceptance at this conference. The strengths that I would like to highlight are the following:\n\n- **High degree of novelty:** Semi-implicit variational inference (SIVI) methods have existed for some time. Existing approaches faced challenges due to the nature of their design, often leading to optimizing bounds on the ELBO or difficult algorithms. This paper's essential contribution is constructing an objective function that can be directly optimized by its gradient flow. This is a significant advancement in the SIVI literature, enabling direct optimization of the ELBO.\n- **Theoretical foundations:** The paper provides a solid theoretical analysis of the proposed method. This includes the study of a related gradient flow, establishing existence and uniqueness of solutions, and providing propagation of chaos results. These theoretical underpinnings give a rigorous basis for the practical algorithm.\n- **Practical algorithm:** The particle variational inference (PVI) algorithm is derived as a practical implementation by discretizing the gradient flow of the proposed objective function. This direct link between theory and practice is a strength of the paper, as it provides a clear path from the mathematical formulation to a computationally feasible method.", "weaknesses": "The following points, particularly the limited experimental validation, prevent me from giving the submission a higher rating (8/9):\n- **Limited scope of numerical experiments:** The paper's major weakness lies in its experimental section, which represents the minimum set of experiments for an acceptable paper. Specifically: a. Lack of empirical demonstration of expressiveness: Despite claiming in Section 2 that PVI can learn potentially more expressive variational approximations than other SIVI methods, no experiments empirically demonstrate this advantage. b. Insufficient exploration of optimization stability: The paper misses an opportunity to address a key challenge in SIVI methods - the difficulty in getting objective functions/algorithms to converge (or meaningfully work). Experiments showing that PVI provides consistently more stable optimization than other SIVI methods or converges on complex models where other SIVI methods struggle would have significantly strengthened the paper. Additionally, comparing performance accuracy on Bayesian Neural Networks (BNNs) for different SIVI methods doesn't show meaningful advantages to this method.\n- **Theoretical-practical gap:** The theoretical analysis is for a modified gradient flow (with γ > 0), but the practical algorithm uses γ = 0. However, this discrepancy is mitigated by the authors' justification that this approach is relatively common in the literature, and empirical results did not show obvious discrepancies between theory and practice.\n- **Clarity in explaining the flexibility of PVI's variational approximation:** The novelty in PVI's approach - not optimizing a specific parametric form of the mixing distribution - was not immediately apparent to me from the explanations in Sections 2 and 3 upon first read. In particular, the last sentence in Section 2 (lines 103-105) is confusing. While it becomes more understandable after reading Section 3, the intuition behind why Q_{YuZ} might not reduce to other parameterizations when fit with PVI is not fully explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your hard work on our submission, and for recognizing the value of our work.\n\n> How sensitive is PVI to the choice of kernel and number of particles?\n\nWe found that PVI the choice of kernel is an important one. In Section 3, we discuss the implications of the kernel choice more explicitly.\n\nAs for the number of particles, after a certain number, we found that there were diminishing/no returns. In all our experiments, we used $100$ particles and found that this was sufficient for good performance. We did not finetune this quantity.\n\n> Can the theoretical analysis be extended to cover the γ = 0 case used in practice?\n\nUnfortunately, we do not currently know how to do this. Taking $\\gamma \\rightarrow 0$ will result in a non-Lipschitz drift which would violate the assumptions of our theoretical analysis. The bounds in Appendix F.4 would diverge.  In order to extend our analysis, one would need to establish existence and uniqueness via other arguments. However, the theory of McKean-Vlasov SDEs is at this stage much less developed than that for simple SDEs and we are not aware of any existing arguments which apply (if any) for the $\\gamma = 0$. Our belief is that this reflects the current state of knowledge about McKean-Vlasov SDEs rather than being a fundamental issue.\n\n>  Despite claiming in Section 2 that PVI can learn potentially more expressive variational approximations than other SIVI methods, no experiments empirically demonstrate this advantage.\n\nIn our experiments, we found that PVI performed well and outperformed all other existing semi-implicit VI methods. The advantage of our expressivity results in obtaining better approximations. In Section 5.2, the quality of the approximation can be seen through lower sliced Wasserstein scores and lower rejection rates; and in Section 5.3, we use MSE as a proxy of posterior quality for which PVI achieves the best overall performance. We also provide additional experiments as part of the rebuttal that compares MCMC samples with PVI samples on a Bayesian Logistic Regression example studied in [1, Section 5.4]. Although this does experiment does not demonstrate an advantage, it is encouraging that our method aligns with MCMC samples.\n\n[1] Yin, Mingzhang, and Mingyuan Zhou. \"Semi-implicit variational inference.\" International conference on machine learning. PMLR, 2018.\n\n>  Are there specific types of problems where PVI is expected to significantly outperform existing methods?\n\nWhenever it's difficult to specify a variational family which is sufficiently expressive to capture the structure of the posterior even in this semi-implicit setting (e.g. in substantially multimodal settings) we would expect this approach to come into its own. In our numerical examples, we have focussed on fairly comparing the method with alternatives in settings in which they (other semi-implicit algorithms) do perform well (rather than contriving settings in which they fail) and have seen that the particle-based approach is extremely competitive even on the examples used to showcase earlier methods: we felt that demonstrating that nothing is lost in using this more general framework when other methods work provided a strong motivation for using it.\n\nHowever, we do not have an example where other methods will fail terribly (we have not actively tried to find one as we had focussed on comparing our method with early approaches using examples chosen by those authors to showcase their work; we can certainly explore this further and would welcome suggestions). Due to the mode-seeking behaviour of reverse KL that other methods minimize (at least approximately), we expect that these methods would (at worst) recover one of the modes for well-defined models. Please do share if you have any suggestions on this and we will include it in any future versions of the manuscript.\n\n> How does the choice of preconditioner affect PVI's performance?\n\nWhen a problem/kernel is ill-conditioned, we found that the preconditioner can be used to stabilize the training procedure. In the Bayesian neural network experiment (Section 5.3), we utilise this trick. In situations where the algorithm is already stable, it may not be required and the algorithm performed well regardless.\n\n> Clarity in explaining the flexibility of PVI's variational approximation.\n\nThank you for pointing this out. We shall amend the final paragraph of section 2 to make this distinction clearer and more explicit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SgsRbl5h9G", "reviewer": "Reviewer_KYHY", "summary": "This paper introduces Particle Variational Inference (PVI), a novel method for semi-implicit variational inference (SIVI) that employs empirical measures to optimize the mixing distribution without making parametric assumptions. Unlike existing SIVI methods that face challenges with intractable variational densities and rely on costly computational techniques, PVI directly optimizes the evidence lower bound (ELBO) using a particle approximation of an Euclidean–Wasserstein gradient flow.", "questions": "1.\tFigure 1: The current caption makes it difficult to interpret the figure. It would be helpful to explicitly mention that lighter shades represent smaller μ\\muμ values, while darker shades represent larger μ\\muμ values.\n2.\tFigure 2: The second sentence of the caption is somewhat confusing. Referring to Table 1 for additional clarification might improve the reader's understanding.\n3.\tMinor Typo: In line 303, \"mu=2\" should be corrected to \"$\\mu=2$\"", "limitations": "The authors have adequately discussed the limitations of their approach.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized and presents the complex concepts of the proposed method with clarity.\n\n2. The Particle Variational Inference (PVI) algorithm, which discretizes the Euclidean–Wasserstein gradient flow to accommodate general mixing distributions, is both practical and innovative. This algorithm is not only grounded in solid theoretical foundations but also implemented effectively, showcasing its real-world applicability.\n\n3. Empirical comparisons of PVI with other SIVI approaches across a variety of toy and real-world experiments demonstrate their good performance.", "weaknesses": "I do not identify any significant weaknesses in the paper. However, there are a few points that require clarification, which I have addressed in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts in reviewing our work and for the helpful feedback and suggestions. All typographic errors will be fixed.\n\n> Figure 1: The current caption makes it difficult to interpret the figure. It would be helpful to explicitly mention that lighter shades represent smaller μ\\muμ values, while darker shades represent larger μ\\muμ values.\n\nThis is a good suggestion. We shall make the necessary amendments.\n\n> Figure 2: The second sentence of the caption is somewhat confusing. Referring to Table 1 for additional clarification might improve the reader's understanding.\n\nThank you for pointing out this mistake. The comment referred to the notation $\\mu_{\\pm \\sigma}$ where $\\mu$ is the average and $\\sigma$ is the standard deviation computed from $10$ independent trials. This shall be fixed in future iterations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all of the referees for their helpful and thoughtful comments; we were pleased that three of the four reviewers reacted positively to the initial submission and hope that we can address the points that were raised during the reviewing process here.

The main area of concern overall appears to have been the extent of the numerical evaluation of the algorithm, although referees differed slightly on the things that they would have liked to see beyond those experiments already provided. We have now added an additional example, a Bayesian Logistic Regression, thereby including most of the examples considered by previous work on semi-implicit variational inference and allowing us to show prior approximation quality on a full Bayesian model and compare with alternative Monte Carlo Markov Chain methods in this challenging setting. In addition, for our prior experiments in Section 5.2, we explore runtime behaviour in further detail.

Aside from numerical evaluation:
* One referee found some lack of clarity in the exposition: we hope this is now addressed.
* All specific points of detail raised have been addressed.
* All other identified weaknesses and questions have been responded to below: we are not able to close the theoretical-practical gap, but this is common to many methods based around the nascent field of mean field approximation of McKean-Vlasov SDEs rather than being specific to our work; we have otherwise been able to at least partially answer all of these.

If you have any further questions or feel that we have not adequately addressed any of the points which were raised, then please let us know in the discussion and we shall endeavour to do so.

Meta-review: This paper proposes an algorithm called Particle Variational Inference (PVI) to fit semi-implicit variational distributions using gradient flows. Unlike previous SIVI approaches, PVI directly optimizes over the space of probability distributions for the mixing distribution. The paper provides solid theorical contributions, as well as numerous experiments, using multiple metrics and settings.

All four reviewers support acceptance of the paper. Reviewer JYYZ mentions the paper could potentially influence the area of generative modeling, as the proposed approximate inference tools could be used for training generative models.

However, one main weakness, shared by several reviewers, is the dimensionality of the considered problems (which is moderate but not high), which indicates that future research may be needed to target higher-dimensional distributions.

Given the novelty and good execution of the paper, I believe this is not necessarily a limitation, and recommend acceptance.
Decision: Accept (spotlight)