Sequential Controlled Langevin Diffusions
Authors: Junhua Chen
Lorenz Richter
Julius Berner
Denis Blessing
Gerhard Neumann
Anima Anandkumar
Venue: ICLR 2025 Poster (2025)
Keywords: variational inference
sequential importance sampling
monte carlo
SDEs
diffusion models
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We propose a framework for combining ideas from Sequential Monte Carlo and Diffusion samplers to sample unnormalized target densities

Abstract: An effective approach for sampling from unnormalized densities is based on the idea of gradually transporting samples from an easy prior to the complicated target distribution. Two popular methods are (1) Sequential Monte Carlo (SMC), where the transport is performed through successive annealed densities via prescribed Markov chains and resampling steps, and (2) recently developed diffusion-based
sampling methods, where a learned dynamical transport is used. Despite the common goal, both approaches have different, often complementary, advantages and drawbacks. The resampling steps in SMC allow focusing on promising regions of the space, often leading to robust performance. While the algorithm enjoys asymptotic guarantees, the lack of flexible, learnable transitions can lead to slow convergence. On the other hand, diffusion-based samplers are learned and can potentially better adapt themselves to the target at hand, yet often suffer from training instabilities. In this work, we present a principled framework for combining SMC with diffusion-based samplers by viewing both methods in continuous time and considering measures on path space. This culminates in the new Sequential Controlled Langevin Diffusion (SCLD) sampling method, which is able to utilize the benefits of both methods and reaches improved performance on multiple benchmark problems, in many cases using only 10% of the training budget of previous diffusion-based samplers.

Published as a conference paper at ICLR 2025
SEQUENTIAL CONTROLLED LANGEVIN DIFFUSIONS
Junhua Chen f.. Lorenz Richter 2.3, Julius Ber Denis Blessing" 6.
Gerhard Neumann Anima Anandkumar
University of Cambridge, "Zuse Institute Berlin, ddida Datenschmiede GmbH, ANVIDIA, Karlsruhe Institute of Techhology, CCalifornia Institute of Technology
ABSTRACT
An effective approach for sampling from unnormalized densities is based the idea of gradually transporting samples from an easy prior to the complicated target distribution. Two popular methods (1) Sequential Monte Carlo (SMC), where the transport is performed through successive annealed densities via prescribed Markov chains and resampling steps, and (2) recently developed diffusion- based sampling methods, where learned dynamical transport is used. Despite the com- mon goal, both approaches have different, often complementary, advantages and drawbacks. The resampling steps in SMC allow focusing on promising regions the space, often leading to robust performance. While the algorithm enjoys asymptotic guarantees, the lack of flexible, learnable transitions can lead to slow convergence. On the other hand, diffusion-basse samplers are learned and can po- tentially better adapt themselves to the target at hand, yet often suffer from training instabilities. In this work, we present a principled framework for combining SMC with diffusion- based samplers by viewing both methods in continuous time and considering measures on path space. This culminates in the new Sequential Con- trolled Langevin Diffusion (SCLD) sampling method, which is able to utilize the benefits of both methods and reaches improved performance on multiple bench- mark problems, in many cases using only 10% of the training budget of previous diffusion- based samplers.
INTRODUCTION
We consider the task of sampling from densities of the form
2 with Z = Sge Prarges (z))dz,
Prarget
where Ptarget E C(RRd (Roo) can be evaluated pointwise, but the normalizing constant Z is typically intractable. This task is of great practical interest, with numerous applications in the natural sciences (Zhang et al., 2023b), for instance, for Boltzmann distributions in molecular dynamics or lattice field theory in quantum physics, as well as posterior sampling in Bayesian statisties (Gelman al., 2013).
Sampling problems vs. generative modeling. The sampling problem poses unique challenges not found in other areas of probabilistic modeling. For instance, while both generative modeling and sampling involve approximating target distribution Purget, they differ fundamentally in terms of the information available. In generative modeling, one has access to samples X Planget: whereas in sampling, we only have access to pointwise oracle Prargot (and, potentially, its pointwise gradients) and no samples. This distinction introduces obstacles for the sampling problem that do not exist in generative modeling. For example, key challenge in modeling distribution is identifying its regions of high probability, or modes. When samples are available, they can directly reveal the loca- tions of these modes. In their absence, however, the sampling algorithm must include exploration strategy to discover them and identify their shape. This exploration becomes exponentially more dif- ficult the dimensionality of the state space increases, making the sampling problem challenging even in moderate dimensions (c.g., 10 - 50).
Dynamical Measure Transport A general idea to approach the sampling problem is to draw parti- cles from an easy prior distribution and gradually move them toward the complicated target (some- times termed dynamical measure transport). In this work, we focus on two popular paradigms:
Equul contribution.
Work partially done at the California Institute of Technology.

---

Published as a conference paper at ICLR 2025
Priot
Diflision Steps
Resampling MCMC
Difficco Skeps
Tiegst
Figure 1: Illustration of our SCLD algorithm, which combines controlled Langevin diffusions with Sequential Monte Carlo methods. The goal is sample from target distribution by learning a stochastic evolution (dif- fusion steps) that starts from a tractable prior and evolves along a prescribed annealed density to the target. We do not have access to samples from the target distribution but can only evaluate its density up to a normaliz- ing factor. At intermediate timesteps, we resample according to the importance weights of each subtrajectory (black dots) and use MCMC steps for additional refinement (yellow dots).
In Annealed Importance Sampling (AIS) (Neal, 2001) and its extension Sequential Monte Carlo (SMC) (Chopin, 2002; Del Moral et al., 2006) particles are successively updated and reweighted, to approach relevant regions in space, targeting an annealed sequence of intermediate distribu- tions. This procedure is typically formulated in discrete time and does not require learning.
In difsussion-useed sampling (Richter & Berner, 2024; Vargas al., 2024) the idea is to learn drift of a stochastic differential equation (SDE) to transport the samples from the prior to the desired target. typically formulated in continuous time. The absence of samples means that data-driven approaches such as for generative modeling (Song et al., 2021) are not possible, and training is instead done via variational inference, gaining information through evaluations of Puagett
Each paradigm brings its own advantages and drawbacks. Traditional SMC methods rely pre- defined rules for particle updates, such as Markov Chain Monte Carlo (MCMC) and resampling methods, which help to direct computational effort onto promising regions of the space and enjoy asymptotic guarantees. While they do not require learning, the employed MCMC methods can, in many cases, exhibit slow convergence to the target (Del Moral et al., 2006). Diffusion-basee samplers, on the other hand, require a training phase, which enables them to automatically adapt the given target. However, training can take significant time and often suffers from numerical instabilities as well as mode collapse (Richter & Berner, 2024).
Sequential Controlled Langevin Diffusions. In this work, we show that the two methods can complement each other. SMC can benefit from the flexible nature of the learnable transitions, and resampling and MCMC can help diffusion- based samplers converge faster and counteract numerical stability issues arising, for instance, from outlier particles. Motivated by this, we identify prin- cipled and general framework unify the two methods, culminating our Sequential Controlled Langevin Diffusion (SCLD) algorithm, which alternates between SMC and diffusion steps as illus- trated in Figure 1. In addition, we devise a family of loss functions that enables end-to-end training (i.e., for which the algorithm used during inference can be directly optimized). This becomes possi- ble by viewing both methods in continuous time and considering measures of the underlying SDEs on the path space. Overall, Our contributions can be summarized as follows:
Taking the continuous-time perspective, we can rigorously connect and unify SMC and diffusion- based sampling by performing importance sampling in path space.
The principled framework of path space measures allows us to readily propose suitable loss fune- tions, which allow for off-policy training with replay buffers and provably scale better to high dimensions than previously used losses.
propose our sampling method Sequential Controlled Langevin Diffusion (SCLD) as a special case of our framework.
Building those connections,
We show that method achieves competitive performance ples, improving over other baseline methods in almost every task, and in many cases only using 10% of the training budget. In two tasks based on robotics control, method is the only one to approximately recover the true distribution.
11 real-world and synthetic exam-
1.1 RELATED WORK
We present an extensive comparison to related works in Appendix A.I. To summarize, our proposed SCLD sampler relies three crucial building blocks:
2

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of different methods (see Appendix A.1 for details). By diseretizatonnflexibble describe the fact that we can include resampling and MCMC steps at arbitrary times. Finite-time convergence refers to the property that the target distribution can (theoretically, in the optimum) be reached in finite time. We note that stochastic transitions allow omitting or reducing (costly) MCMC steps in learned SMC methods.
Truditional
CMCD
PDDS
(ours)
Learndd Transition
X (MCMC)
/ (Ncural SDE)
(Ncural ODE)
(Ncural SDE)
(Noural SDE)
Stochastic Transition
End-to-end Training
(needs importance weights)
(alternating)
(inel. hyperpatameters)
Particle Method
X
Discretization- flexiblee
/
/
theory)
Finite-time Convergence
X
Sequential Monte Carlo (SMC). SMC methods (Chopin, 2002; Del Moral et al., 2006) describe a general methodology sample sequentially from sequence of annealed distributions, using transi- tion kernels (typically based on MCMC) and resampling steps. To mitigate drawbacks such as long mixing times and tedious tuning. previous works proposed learn the kernels (Phillips al., 2024; Matthews et al., 2022: Arbel et al., 2021). However, prior training objectives suffer from various shortcomings, either requiring importance sampling with potentially high variance, exhibiting bias, or relying alternating methods that preclude end-to-end training (see also Table 1). Further, they need to place restrictions on their parameterizations or suffer from unfavorable computational costs. In particular, approaches with deterministic transitions, such as normalizing flows, require compu- tations of divergences or Jacobian determinants, and MCMC steps to recover sample diversity after resampling. This is not needed for methods based on stochastic transitions like our method SCLD.
Diffusion-based samplers on subtrajectories. To overcome these shortcomings and flexibly pa- rameterize the transition kernels, draw ideas from recent work controlled SDEs for sampling problems (Zhang et al., 2023a; Richter & Berner, 2024). This can be done by partitioning the SDE trajectories in time. However, to compute importance weights (in path space), which are necessary for resampling as well as for MCMC kernels in SMC, the SDE marginals after each subtrajectory need to be known. To cope with that, we identify the recently proposed Controlled Monte Carlo Diffusions (CMCD) (Vargas et al.. 2024) as a suitable framework since it allows us to define a prescribed (and therefore known) target evolution of the SDE marginals. Building upon this, we develop an extension of SMC to continuous time, where resampling (and, optionally, MCMC steps) can be employed at arbitrary times.
Log-variance loss. However, the subtrajectories and discrete resampling steps make optimization challenging. Previous methods either relied on alternating schemes or approaches based on the re- verse KL divergence and importance sampling. known suffer from mode collapse and potentially high variance. We show that the log-variance loss (Nüsken & Richter, 2021) offers a way to obtain principled, efficient, and low-variance objective such that we can optimize sampler and parts of the hyperparameters in an end-to-end fashion using replay buffers.
SEQUENTIAL CONTROLLED LANGEVIN DIFFUSIONS
We start by giving an introduction to Sequential Monte Carlo methods. However, different from previous work, our focus is on a continuous-time perspective that can be readily integrated with diffusion-based samplers.
2.1 A PRIMER ON SEQUENTIAL MONTE CARLO IN CONTINUOUS TIME
Importance sampling (IS). The idea of utilizing samples from a prior distribution in order to com- pute statistics relying on samples from a target can be motivated by importance sampling. In its simplest case, one can compute unbiased estimates w.r.t. the target distribution via
Expoptaront [xx(X)) = Exprrpptor (v((x) w (Xo) = 1 1 (xx) Ju(X
K
(2)
where & E C(Rd, R) a function of interest, the weight is detined' as w Puarge and (x (x( are i..d. samples from Pprior- Since importance sampling becomes highly inefficient the high-
)==
Pocior
If the normalizing constant Z is not available, we can compute unnormalized weights w = Ptarget and Pprtor
normalize them by their sum, leveraging the identity Z Exoe-priion [o)X (self-normalized importance sampling). While this introduces bias, the estimator is still consistent K no (del Moral, 2013).
3

---

Published as a conference paper at ICLR 2025
probability regions of prior and target do not overlap substantially, key idea is to gradually "trans- port" Xo to Xr.
Annealed importance sampling (AIS). In particular, we may sequentially move particles from the prior to the target along a curve (eetor)) chosen such that (:,,0) 0) = Pprior and ... T) = Prarget: e.g., by linear interpolation in log-space (Dai et al., 2022). To this end, consider two (time-dependent, forward and backward) Markov kernels Palr and Prts- Given a time grid 0 = to < t IN T (also referred as annealing steps), we may now sample Xto ~ Pprior and iterate for each - .:
1. Sample Xt,, ~ Pe. - (.
(X do Wr (Xt, (x
2. Compute the weights wt, 1 (Xt, 1 Xx.)) m(Xt,-- fn- n' (Xe, [1
We can then perform importance sampling on an augmented target distribution via the weights
(o XiN) = II. Wt..-,,h (X X,.)
Plo v (in)) P y (X Xiy)
(3)
where Pta.....y and Pio.....y are the joint densities of the "forward" and a corresponding "backward" operation. In particular, in analogy to (2), it holds that
E Xey [pp(X
Xex)] = Er (x(X))
(4)
Resampling, In principle, any forward and backward Markov kernels lead to an unbiased estima- tor of the expectation of interest, as stated in (4). In practice, however, a notorious problem with importance sampling is its potentially high variance. Specifically, the variance might increase ex- ponentially with the dimension, sometimes termed curse of dimensionality, see, e.g., Chatterjee & Diaconis (2018); Hartmann & Richter (2024). To circumvent this issue, one idea is to sequentially "update" samples (also referred to as "particles") during the course of the simulation according to their weights, as refocus computational effort on promising particles- a procedure referred to as resampling. For instance, we can select only certain (relevant) samples x for the estimation of the expectation in (2). To this end, let (()) be random variable with values in (0...., , K) and ([O()) ................................ xx(1 how many times we select the k-th sample. Due to the tower property, we can then also obtain a consistent estimator of the expectation in (2) via
KW(X)( where W(X) w (x((")) E/= wxx(()), defining
ExrePrarred [<((T)] ~ EK (xxx )Ow)
A common choice is to consider O MR (x distribution with K trials, where the normalized weights determine the event probabilities (Gordon al., 1993). We note that with this resampling step, we introduce additional stochasticity. How- ever, at the same time, it can bring statistical advantages by focusing on "relevant" samples, e.g., stabilizing effects and variance reduction (Dai al., 2022).
)(xx drawn from a multinomial
Remark 2.1 (SMC formulation in continuous vs. discrete time). We stress that, even though we evaluate process X on N + discrete time instances, the formalism above includes time- continuous processes (X2)xEjO.TT) While some transition kernels used in SMC, e.g., uncorrected Langevin kernels, can be interpreted in continuous time. SMC is typically stated for a fixed number of discrete steps. We will see in the sequel how the continuous-time formulation offers an ele- gant framework with certain advantages, particular, allowing us to integrate learned SDE-based transition kernels and interleave them with resampling and MCMC steps at arbitrary times.
2.2 CONTROLLED SDES AND IMPORTANCE SAMPLING IN PATH SPACE
A central question in SMC is how to choose the forward and backward transition densities Paje and Pris defined above. Clearly, when the forward and backward joint densities stated in (3) agree, we achieve perfect sampling in the sense that no corrections with importance weights are necessary. However, typically not possible to obtain such transitions, and thus the choice of Paje and pil. to approximate this criterion is of critical importance to the success of SMC. Whereas, tradition- ally, MCMC steps have been employed as the transition kernel (Dai et al., 2022). they are known to require large number of steps to achieve approximate transportation between densities. In recent years, there has been interest in employing learned transition densities to overcome the slow conver- gence times of fixed MCMC kernels (Matthews et al., 2022; Phillips et al., 2024). Advancing those
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Sequential Controlled Langevin Diffusion (SCLD),
D See Algorithm 2 for details.
Require: Annealing path тr learned control u, time grid 0 = to < ... IN = T
1: Initialize: Xo = XD x-1:K) ~ Ppriar and Wo = Wo
(1:K)
2: for = to = N do
3: Transport: X Lita] = simulate 11
> See (6) and (19)
d (n-1ttol
Compute RNDs: W'tn-iital a inl
(X (tn-ittl))
See (12) and (31)
5: Update weights: Wn Wn- P
See (13)
6: Resample: X.,, W. resamplee (,,, W.)
See Algorithm
return Samples Xx x(:K) approximately from Prarget
attempts, we will show how transition densities corresponding to SDEs yield a principled solution that, moreover, allows us to leverage recent advancements of diffusion models.
Diffusion bridges. To this end, let us consider the stochastic process X (Xx)) (e(o,T) defined by the SDE
xx;: = u(x; t)dt + (()aW, XØ ~ Ppriors
(6)
where E C(R" x [0, is control function, o E C([O, R) the diffusion coefficient, and W standard Brownian motion. This process uniquely defines forward transition density Pale and falls into the framework stated in Section 2.1 for any time steps 0 = to < 11 < ..... tv = T. In fact, we can leverage the ideas from CMCD (Vargas et al., 2024) and learn such that the transport happens along prescribed density in time, i.c., such that the density px= (:,t) of X² is equal to a prescribed target density <(: t), connecting the prior and the target, for every t E T]: cf. Lemma 2.2 below. We will that the knowledge of the marginals allows for natural integration within SMC frameworks. Now, similar to the importance sampling framework from Section 2.1, the general idea is exploit time-reversed dynamics that starts in the desired target density. To be precise, we may further define a related reverse-time SDE
dY = "YY", + att)W,,
YI Ptarget
which depends the control E C(R x (0, RR, and where dw, denotes backward integration of Brownian motion. Now, if u and e are learned such that X" and Y" are time-reversals of each other, then Px- py", i.c. the two processes transport the prior to the target and vice However, in this general setting, there are infinitely many such bridging processes, all fulfilling Nelson's identity (Nelson, 1967), i.e.,
o2 logpx" = = V Py".
Since our goal is to satisfy px- pyv RT, can incorporate this constraint via the ansatz v = L' log म. leading to the SDE
dY; (u o² log (Y," t)dt + ot) YT Ptarget
suggested in Vargas et al. (2024), noting that the process now also depends the control u. Consequently, under mild conditions, this constraint leads to a unique gradient field representing the solution u² to the time-reversal problem (Vargas et al., 2024, Proposition 3.2). We comment more general, learnable density evolutions in Remark A.1.
Measures in path space. The task of learning the time-reversal spective of measures on the space of continuous trajectories C((0, T), Rd), also called path space. Loosely speaking, path space measure IP = Pst.Pocior of the process (6) can be thought of as the joint density Pr......ty _xK..... X (3) when - oo, i.e., evaluated along infinitely many time instances (Baldi, 2017, Corollary 11.1).
be approached via the per-
In analogy to importance sampling described in Section 2.1, we may now consider change of measure in path space, i.e.,
E Ex-- (e(X))xXXX) (X)) = Ey (<(Y))) = Exspeagge [x(z)),
(10)
"See Vargas al. (2024, Appendix A) for details and assumptions.
5

---

Published as a conference paper at ICLR 2025
where w and P = P preege is the path space measure associated to (9). Furthermore, we can di ar formulate the time-reversal task as the minimization problem
u = arg min ((IP"..peer I.lPPtaree))
(11)
where D a divergence and U c C(TRd x [0, TR.) the set of admissible controls, cf. Richter & Berner (2024). If we can bring the divergence to zero, we have indeed achieved time-reversal between the forward and backward transitions and, thus, perfect sampling. Both for (10) and typical divergences in (1I), it is essential to have a tractable expression for the likelihood ratio between the measures of the forward and the reverse-time process, also called the Radon-Nikodyy derivative (RND). This is given by the following lemma; see Vargas et al. (2024) for the proof.
Lemma 2.2 (Likelihood ratio between path measures). Let IP [e.8] and IP [s.8] be the path space mea- sures of the solutions to the SDEs in (6) and (9) on the time interval [s,t] C. [0, 1"I where we assume X x(,,)) and Y T(.,t). Then for a generic? process X holds
(o.)) (1) = dP [1
(X) = ((x..)
=(X,,)) exp (1e ||all Ilu log al1 T)dr
20a
(12)
+ S " log (X,, ax, S-- 2x ()xx))
As can be seen from Lemma 2.2, path space measures can be readily employed for sequential al- gorithms that operate on the time grid that we introduced before. In particular, we may divide our trajectories X x and y into subtrajectories and thus our path space measure into multiple chunks. To be precise, we may write
1o db
Pa. PN- = W[to.t1] ... WiiN 1..]]-
(13)
(1 [NN-11N]
Different from the framework in Section 2.1. we note that Lemma 2.2 offers an explicit formula for computing the weights (1(.) in continuous time. As can be seen in the importance sampling identity (10), the weights can be interpreted as correcting for a potentially imperfect rime-reversal. For convenience, we state Algorithm for a simplified, high-level overview of combining SMC with diffusion models and refer to Algorithm 2 in Appendix A.3 for a more detailed exposition. Further, we note that the suggested setting relates to the usual SMC algorithm (such as in Dai et al. (2022)) by taking a different forward transport step (where our Markov kernel is implemented by an SDE) and by adopting the weighting step (using the Radon-Nikody derivative in place of the likelihood ratio). Using the target density (-, (n), we can also add MCMC refinements at each time In: see Section 2.4.
2.3 LOSS FUNCTIONS AND OFT-POLICY TRAINING
We can adapt the idea of learning the optimal control u² to our sequential setting by considering divergences on each subinterval [fn-1, tn] separately, in consequence bringing losses of the form
C(u) = Dm D [1 [n-11d)) (--1..)))
(14)
where n. := x-; (.,ttt) t.). We stress that with (14) optimization can in principle be conducted globally in spite of the resampling happening sequentially. However, depending on the choice of the divergence, this comes with additional challenges.
KL divergence. A classical choice is the Kullback-Leibler (KL) divergence D DKL, i.e.,
DKL [ [bn-1:fa]] (ba=i.tal) -xx= 2Min [log (w/
(15)
(G--1-5 1.fa]
where [6 is defined as in Lemma 2.2 and the minus originates from the reciprocal impor- tance weights in the logarithm. However, for computing the expectation we need X - resampling has been employed in the previous iteration (at time in-1; see Algorithm 1), potential mismatch in the expectation is automatically corrected. Alternatively, we may correct with impor- tance sampling in path space. To this end, let fm (with tm < fn-1) be the last time resampling has been conducted, i.c., the last time the weights have been reset; see Algorithm 5. As suggested in
NNte that the Padon-Nikodyee derivative is only defined almost surely w.r.t IP P...-- In particular, it only depends on X on the time interval [s.

---

Published as a conference paper at ICLR 2025
Matthews et al. (2022), we can then consider the importance weight (am and compute
DKL Pu Pu.
[t [tn-1stn]] (5t-1)))) = E Puut (fm.fal
[log (w),,, 2.ful (X") (tmoi u] (X"),, (16)
for which X does not need to be distributed according to nn-1 anymore. However, the im- portance weights potentially introduce additional variance into the loss, particularly in high dimen- sions. This observation is stated rigorously in the following proposition, cf. Nüsken & Richter (2021, Proposition 5.7), and proved in Appendix A.2.
Proposition 2.3 (Relative error of KI. divergence). Denote by Dx² the x2-divergence and by ((K) = Var (Va( y12 DDL the relative enror of the Monte Carlo estimator D(K) of the KL diver- gence in (16) with sample size K. Moreover, let tm. be the last resampling time and let B&i and 1 1n5.)] be the I-fold product measures of identical copies of PP (tn-1.8ul and IP [t (ta-
[n-ttall
spectively. Then there exists a constant c >> O, such that for any it holds that
r(K) (188L [t-1,
ISST ((n-1sdai) 2 c (D2e (Pitmott -11F (Pm.tt-il) + +1) 1/2
(17)
Given path measure P of dimeeniiaaa process, note that P is measure the product C( TT. In particular, for D (corresponding to indepen- dent components), we can clearly identify d == I as the dimension of the considered problem. This means that the relative error of the estimator of the KI. divergence (16) is expected to scale exponen- tially in the dimension, which is illustrated in Figure 11 in Appendix A.6.10. As shown in Nüsken & Richter (2021), the log- variance (LV) divergence does not exhibit this unfavorable property.
space 8 C[,,
LV divergence and off- off-policy training. An alternative divergence can be defined by
DI [
50,50 (a-1,.a)) Varx-Q [log ( (.. (X))]
(18)
which, in fact, is family of divergences parametrized by reference measure Q) P -....-1 can be chosen with arbitrary controls and initial distributions - (also called off off-policy training, see Remark A.2 for details and connections to reinforcement learning). In particular, we do not need Xr,. ~ Tn-1 anymore, and thus reweighting such as in (16) is not necessary, irrespective of the fact that resampling at time t,, might not have been conducted. We summarize the training procedure for both divergences in Algorithm and present details in Appendix A.3.
that
(t.. 1.6..]
2.4 ALGORITHMIC REFINEMENTS AND IMPLEMENTATIONAL DETAILSS
In this section, turn theoretical considerations from Sections 2.1 to 2.3 into implementable algorithms. We collate these changes in Algorithm 2 in Appendix A.3, representing a practical version of Algorithm 1.
Loss Function. We focus on the log-variance divergence in the sequel and refer to Appendix A.6.10 for comparison to the KI. divergence. We choose ū u (or previous versions when using buffer, see "replay buffers" below) and simulate X in (18) starting from the prior, so 7 corresponds to the SDE marginal. However, since we do not take gradients w.r.t. the control ū of the reference measures, we detach the trajectory X, in line with Richter & Berner (2024). In particular, we do not need differentiate through the SDE integrator.
Time discretization. In practice, we choose N equidistant resampling times, i.c. t, - t--1 T, for every n E {1,..., N.. where the number of subtrajectories N may change across applications. We discretize the SDE (7) via the Fuler- scheme, containing I. evenly spaced steps per subtrajectory, i.e.,
X - X2 + + (i - 1)h)h + a(ii 1)h) & &~,N(
(19)
f E (1,.... NL} with h = T/L. of the Radon-Nikodym derivative from Lemma 2.2 for computing the importance weights Wit.. 1.47]-
L. We refer to (31) in Appendix A.3 for the resulting discretization
Annealing path. For the prescribed density curve we consider
क (x, 1) XX Pprior (1:) 1-1(1) Ptarget ( ((1)
(20)
7

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of different methods in terms of ELBOs, i... lower bounds the log- nog-oommaization constant log Z. We use this metric for tasks where we not have access groundtruth metrics. We report NA if all considered hyperparameter choices diverged.
ELBO (1)
Seeds (26d)
Sonar (6ld)
Credit (25d)
Brownian (32d)
LGGPP (1600d)
SMC
74.63=0.14
111.50=0.
589.82+6.75
2.21
385.75+7.66
SMC-ESS
-0..07_060
- -109.1020
-505.5710.1
0.49
497.85
SMC-FC
74.07-0.02
108,93=
-005,30+.02
1.91=
778.10+2.20
CRAFT
-73.75=002
-108.97
-108.97=0.1
-518. 118.2522.00 52
0.90=0.10
480.87
DDS
-75.21_021
-121.22
-514.74:1.22
0.56
NA
PIS
-88.92_2.5
-142.87
-846.5712.42
NA
10.
CMCD-KL
-3.51=0.0
1009.09=..01
507.23++.40
0.86=0.01
478.75
CMCD-LV
73.67=0m
-109.50
504.90
0.54.0.0
472.79a0.44
SCLD (ours)
-73.4520.01
-108.17
-604.4620.00
1.00
486.77
where 3: [0, T] + [0, 1] is a monotonically increasing function fulfilling 8(0) and B(T) 1. We choose to learn the function 33 to attain a smoother transition; see (35) and Appendix A.6.5.
Resampling. There is wealth of literature (Webber, 2019; Doucet et al., 2001; Douc & Cappé, 2005) regarding designing SMC resampling schemes. However, for a fair comparison to CRAFT (Matthews et al., 2022), we utilize the common multinomial resampling scheme. Resampling can, however, reduce particle diversity by introducing identical particles in its output. As such, it is common to trigger resampling at a time t, only when the Effective Sample Size (ESS), a measure
of particle quality defined by ESS = ΣE Σ (wr is below a certain threshold, where wh, are the
importance weights at time tu (as in Algorithm 2). In line with prior works (Matthews et al., 2022; Phillips al., 2024), we pick the threshold to be 0.3K where K is the number of particles.
MCMC refinements. In order to cope with sub-optimal controls u during the course of optimiza- tion, we add some MCMC refinmment steps after each subtrajectory at time t,s using Markov kernel with invariant measure (T., t.). ). In line with Matthews et al. (2022), after every subtrajectory, we use one Hamiltonian Monte Carlo (HMC) step with 10 leapfrog steps.
Replay buffers. Replay buffers are known to prevent mode collapse and improve sample efficiency for sampling tasks (Vemgal al., 2023; Midgley et al., 2022; Sendera et al., 2024). such, we uti- lize prioritized replay buffer during training time. At a high level, maintain a fixed-size rolling cache of paths generated by previous versions of the policy, i.c., learned control u. For the gradi- updates, we then take half of the samples from the current policy and the other from the buffer using NikodyN derivatives as weights for prioritization, see Algorithm in Appendix A.3 for details. We note that this procedure easily feasible with the log-variance divergence since this divergence does not rely on an evaluation along the current policy (see Section 2.3).
EXPERIMENTS
We empirically demonstrate the performance of the proposed SCLD sampler on a wide variety of sampling benchmarks.* We consider a combination of practical and synthetic examples taken from Blessing et al. (2024), the full descriptions of which are contained in Appendix A.4:
Examples from Bayesian statistics: The Seeds, Sonar, Credit, Brownian, and LGCP tasks.
Synthetic targets: 40-mode Gaussian mixture model in 50d (GMM40), 32-mode Many- Well task (MW54) in 5d, the popular 10d Funnel henchmark, and a 50d Student mixture model (MoS). Many of these are relatively high dimensions and with many well-separated modes.
The Robotl and Robot4 tasks: Inspired by robotics control problems, these synthetic 10- dimensional targets model the distribution over the configurations of a 10-joint robotic arm in the plane. They have multiple sell-separated and sharp modes.
As baselines, we consider a representative selection of related sampling methods and refer to Ap- pendix A.I for descriptions. We study two metrics used frequently by previous works, such as in Blessing et al. (2024); Vargas al. (2023). When groundtruth samples available, we report the Sinkhorn distance (an optimal transport distance) to set of generated samples (Cuturi, 2013), and otherwise consider the ELBO metric (i.e., lower bound on log Z).
*Our code can be found /tttp:://gitaii com/ cowm///ooyoouu3111m1m 41/CLD.
8

---

Published as a conference paper at ICLR 2025
Table 3: Comparison of different methods in terms of Sinkhorn distances. We present all tasks where have
access to samples for evaluation. report NA considered hyperparameter choices diverged.
Sinkhorn (L) Funnel (10d)
MW54 (5d)
Robot1 (10d)
Robot4 (10d)
GMM40 (50d)
MoS (50d)
SMC
149.35
20.71
24.02=1.0s
24.08
3297. :
SMC-ESS
117. 487.78
1. 1.11=0.15
1.82=0.00
2. 2.11=0.a1
24240.68 68
SMC-FC
211 33 os
2.03+0.17
0.37++0.08
1.23=0.02
39018. 27 130.82
3200, 10=
CRAFT
133.4222.o
11.4720.0
2.9220.01
4.14
28960. 109900.70.34...
1918. 1918.114,188.
DDS
142 20...
0. 0.632..24 24
11.44g12.50
5.38=2244
5435. 18 20
2154.88 86
PIS
NA
0.42
1.54
2.02.2.58
(0405.75:244
2113.17 1331177
CMCD-KL
124.89
0.57
3.70
2.62
22132. 28 1es.
1848.89 1.18.891.32.... 2.32, 56
CMCD-LV
139.07++9.8
0.51+0.0s
28.49=0.07
27. 27.00=a.07
4258. 57..577 #777.16
1945.71++4s
SCLD (ours)
+34.23
0.44+0.06
0.31
0.40
3787.73:224.77
656.10
CMCD-KL
CMCD-LV
CRAFT
SCLD (ours)
Groundtruth
Figure 2: Samples from our considered methods and the groundtruth for the GMM40 (50d) (top) and Robot4 (10d) (bottom) tasks. Our SCLD method accurately finds all modes and avoids low probablity regions.
We took great care to ensure the fairness of our experiments and refer the reader to Appendix A.5 for full experimental and reproducibility details and to Blessing et al. (2024) for a discussion on bench- marking samplers. We also include numerous additional experiments and metrics in the appendices, such ablation studies in Appendices A.6.1 and A.6.2, runtime information in Appendix A.6.3, study on log estimation in Appendix A.6.4, the effect of learning priors by variational inference in Appendix A.6.6, comparison to PDDS in Appendix A.6.7, and comparison of KL and LV training in Appendix 1.
3.1 RESULTS
Our SCLD method exhibits strong performance on both ELBO and Sinkhorn benchmarks (Tables and 3). Indeed, among all tasks except Funnel, we are able to achieve the top performance or come a close second when measuring performance by Sinkhorn distances (when it available). For ELBO estimation, SCLD can utilize a large number of resampling steps to attain the strongest performances all but one task. In particular. SCLD can surpass the outcomes of CMCD-KL and CMCD-LV with 40000 gradient steps using only 3000 steps. In the following, we comment on different aspects.
Avoiding mode collapse. We visualize the samples for GMM40 and the Robot4 task in Figure 2. For GMM40, we plot the first two dimensions of samples against the true marginal distribution. In all attempted hyperparameter settings, we found that CRAFT suffers from mode collapse (see also Appendix A.6.4) and that CMCD-KL gradually collapses to few modes, covering low proba- bility regions. CMCD-LV and SCLD perform much better, and indeed the samples from SCLD are virtually indistinguishable from the groundtruth. For Robot4, visualize the sampled robot arm positions. Observe that for the Robot4 task, CMCD-KL and CRAFT both collapse onto mode. CMCD-LV does not experience mode collapse but nevertheless does not sample accurately for any mode. Only our SCLD Method is able to identify and sample relatively precisely from all modes.
Improved convergence properties of SCLD. We found that the SCLD algorithm demonstrates superior convergence properties. As SCLD is effectively initialized as an SMC sampler and is trained to improve upon it, expect good initial performance even before training and, thus, an improved starting point for optimization. As visualized in Figure 3, SCLD consistently attains better ELBOs for any given training time budget on all tasks when compared CMCD-KL and CMCD-LV. While in some cases SCLD is initially worse than CRAFT, it always manages to catch up quickly and surpasses SCLD and CMCD steps require similar amounts of time for these tasks (see Appendix A.6.3), and thus SCLD offers 10-fold decrease in training time well as iteration count compared to CMCD. See Appendix A.6,9 for an alternative visualization.

---

Published as a conference paper at ICLR 2025
Crudit
1.0
Brownsan
Beada
-108
Suner
Deu
-73.9
0.5
-110
-920
-74.0
SCLD CMCD KL
-330
0.0
112
-0.5
74.5
-114
Long Run CMCD (Seal)
CIRAFT
-550
200
400
-1.0g
1.50
200
-75.0
107
200
100 Elopood
Edpoodd
Elapsed
Figure 3: ELBOs during training for several tasks. We visualize the ELBO estimates attained by 4 methods as function of the training time clapsed (until SCLD finished after 3000 iterations), running 3 seeds for cach task. We mark the long run CMCD ELBOs (best out of KL and LV loss), corresponding to running for 40000 gradient steps as the main table. Methods leveraging Sequential Monte Carlo (SCLD and CRAFT) generally exhibit improved convergence speed, but whereas CRAFT plateaus quickly, our SCLD method often achieves state-of-the-art performance in about 5 minutes.
(900)
-7.5
110
2500
-7.0
2000
5MO Stops
114
1900
12854
-75.0
118
1000
16 32 64128
2 4 8 16 64128
16 61128
32 64128
6BMC Steps (Teh)
SMC Steps (Teh))
# BMG Stops (Tah))
//MC Stops (Trak)
Figure 4: Performance of SCLD for different numbers of SMC steps training and evaluation time for several tasks. Better results are shaded darker. note taking "zero" SMC steps corresponds to the CMCD method. Using more SMC steps has generally a beneficial effect during training. Our method allows us to select different number at training and during inference.
Choice of number of SMC steps. Here, we study the effect of varying the number N of subtrajec- tories used in the SCLD sampler, i..., SMC steps where we apply resampling (if ESS is lower than the threshold) and MCMC steps, and offer practical advice on choosing this value. For this study, we fix the number of gradient steps for training to 8000 but otherwise retain the same experimental design. The results are illustrated in Figure 4, where we visualize the relevant metric for four tasks and demonstrate the effect of varying the number of SMC steps used at training and evaluation. For most tasks, we found advantageous to use as many SMC steps as possible at both training and evaluation time. Particularly for the Seeds and Sonar targets, the outcomes look strikingly similar. For these tasks, it is also shown that using a smaller number of SMC steps at training or even only adding SMC steps at evaluation already improves upon stand-alone diffusion- samplers.
While is well known that resampling can potentially lead to mode collapse and loss of sample diversity on highly multimodal tasks (Doucet et al., 2001), found that even for such tasks, sampling, when used sparingly, was still beneficial during training. This is clearly reflected in the multimodal Robot4 task, where using SMC steps training significantly improves sample quality. In line with the previous paragraph, this suggests that our SCLD training setup can help improve training convergence. Informed by our observations, we opt use subtrajectories only training for all synthetic tasks except Funnel and MoS, and, for all other tasks, we utilize 128 subtrajectories both training and evaluation time for the main experiments. These choices, while not necessarily optimal, are robust and work well across our diverse set of benchmarks.
CONCLUSION
We have developed a framework for combining diffusion-based samplers with Sequential Monte Carlo algorithms and propose simple yet effective methods for training. Our framework culminates in novel sampler, termed Sequential Controlled Langevin Diffusson (SCLD), in principle offering a great amount of design freedom. In particular, SCLD allows for accelerated training, flexible pa- rameterizations, end-to-end training with prioritized replay buffers, and injection of resampling and MCMC steps at arbitrary times in the generative process. We provide careful ablation studies of our design choices and empirically show state-of-the-ae performance diverse range of benchmarks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Francisco Vargas, Lee Cheuk-Kit, and Dinghuai Zhang for very helpful discussions. J.C. was supported by a Summer Undergraduate Research Fellowship at the California Institute of Technology. D.B. is supported by funding from the pilot program Core Informatics of the Helmholtz Association (HGF). J.B. acknowledges support from the Wally Baer and Jeri Weiss Postdoctoral Fel- lowship. A.A. supported in part by Bren endowed chair, ONR (MURI grant N00014-23-1-2554) and the A12050 senior fellow program at Schmidt Sciences. The research of L.R. was partially funded by Deutsche Forschungsgemeinnchatt (DFG) through the grant CRC 1114 "Scaling Cas- cades in Complex Systems" (project A05, project number 235221301). We thank the anonymous reviewers for their careful reading of our manuscript and their insightful comments and suggestions.
REFERENCES
Michael S. Albergo and Eric Vanden-Eijnden, NETS: A non-equilibrium transport sampler, 2025.
URL https::tp///o // www.i.orgg.ooo..oooo.oooooooooot abs/2410 02711.
Michael Arbel, Alex Matthews, and Arnaud Doucet. Annealed flow transport Monte Carlo. In International Conference Machine Learning, pp. 318-330, 2021.
Oleg Arenz, Mingjun Zhong, and Gerhard Neumann. Trust-region variational inference with gaus- sian mixture models. Journal of Machine Learning Research, 21(163):1-60) 2020.
Paolo Baldi. Stochastic calculus. Springer, 2017.
Julius Berner, Lorenz Richter, and Karen Ulirich. An optimal control perspective on diffusion-bart based generative modeling. Transactions on Machine Learning Research, 2024.
Espen Bernton, Jeremy Heng, Amaud Doucet, and Pierre E Jacob. Schrödinger bridge samplers. arXiv preprint 1912: 13170, 2019.
Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statis- tics). Springer-Veriie Berlin, Heidelberg, 2006.
Denis Blessing, Xiaogang Jia, Johannes Esslinger. Francisco Vargas, and Gerhard Neumann. Be- yond ELBOs: A large-scale evaluation of variational methods for sampling. In Forty-first Inter- national Conference on Machine Learning, 2024.
Benjamin Boys, Mark Girolami, Jakiw Pidstrigach, Sebastian Reich, Alan Mosca, and O Deniz Akyildiz. Tweedie moment projected diffusions for inverse problems. arXiv preprint arXiv:2310.0672., 2023.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miine and Qiao Zhang. JAX: composable transformations Python+NumPy programs, 2018. URL http: //github. com/ jax-ma/jaxx ml/jax.
Alexander Buchholz, Nicolas Chopin, and Pierre Jacob. Adaptive tuning of hamiltonian monte carlo within sequential monte carlo. Bayesian Analysis, 16(3):745-771, 2021.
Alberto Cahezas, Adrien Corenflos, Junpeng Lao, Rémi Louf, Antoine Carnec, Kaustubh Chaud- hari, Reuben Cohn-Cirrdon, Jeremie Coullon, Wei Deng, Sam Duffield, et al. BlackJAX: Com- posable Bayesian inference in JAX. arXiv preprint 47X7:2240..1099, 2024.
Souray Chatterjee and Persi Diaconis. The sample size required importance sampling. The Annals of Applied Probability, 28(2):1099-1133 2018.
Tianrong Chen. Guan-Horng Liu, and Evangelos Theodorou. Likelihood training of Schrödinger bridge using fations, 2022.
SDEs theory. In International Conference on Learning Represen-
Eungchun Cho, Moon Jung Cho, and John Eltinge. The variance of sample variance from a finite population. International Journal of Pure and Applied Mathematics, 21(3)3389, 2005.
11

---

Published as a conference paper at ICLR 2025
Nicolas Chopin. A sequential particle filter method for static models. Biometrika, 89(3):53--552) 2002.
Nicolas Chopin, Omiros Papaspiliopoulos, et al. An introduction to sequential Monte Carlo, vol- ume 4. Springer, 2020.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209. 2022a.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 35:25683 25696, 2022b.
Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Advances in Neural Infor- mation Processing Systems, volume 26, 2013.
Marco Cuturi, Lactitia Meng-Papaxanthon Yingtao Tian, Charlotte Bunne, Geoff Davis, and Olivier Teboul. Optimal transport tools (OTT): JAX toolbox for all things wasserstein. arXiv preprint orXiv:2201.12331 2022.
Chenguang Dai, Jeremy Heng, Pierre Jacob, and Nick Whiteley, An invitation to sequential Monte Carlo samplers. Journal of the American Statistical Association, 1576539) 2022.
Paolo Dai Pra. A stochastic control approach to reciprocal diffusion processes. Applied mathematics and Optimization, 23(1):313-322 1991.
Valentin De Bortoli, James Thortton, Jeremy Heng, and Arnaud Doucet. Diffusion Schrödinger bridge with applications to score-base generative modeling. Advances in Neural Information Processing Systems, 14:17665--177000 2021.
Pierre del Moral. Mean field simulation for Monte Carlo integration. Monographs on Statistics & Applied Probability. Chapman&Hall, 2013.
Pierre Del Moral, Amaud Doucet, and Ajay Jasra. Sequential Monte Carlo samplers. Journal the Royal Statistical Society Series B: Statistical Methodology, 68(3):41 1-43,, 2006.
Kieran Didi, Francisco Vargas, Simon Mathis, Vincent Dutordoir, Emile Mathieu, Urszula J Ko- morowska, and Pietro Lio. A framework for conditional diffusion modelling with applications in motif scaffolding for protein design. arXiv preprint arXiv:2312.09255 2023.
Carles Domingo-Enrich. A taxonomy of loss functions for stochastic optimal control. arXiv preprint arXiv:24100003355 2024.
Carles Domingo-Enrich, Jicqun Han, Brandon Amos, Joan Bruna, and Ricky TQ Chen. Stochastic optimal control matching. arXiv preprint arXiv:2312.02027, 2023.
Carles Domingo-Enrice Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint Ttt.:2409.08866..1 2024.
Randal Douc and Olivier Cappé. Comparison of resampling schemes for particle filtering. In Pro- ceedings of the 4th International Symposium Image and Signal Processing and Analysis, pp- 64-69, 2005.
Arnaud Doucet, Nando de Freitas, and Neil J. Gordon (eds.). Sequential Monte Carlo Methods in Practice. Statistics for Engineering and Information Science. Springer, 2001.
Arnaud Doucet, Will Grathwohl, Alexander G de G Matthews, and Heiko Strathmann. Score-based diffusion meets annealed importance sampling. Advances Neural Information Processing Systems, 2022.
Marylou Gabrié, Grant M Rotskoff, and Eric Vanden-Eijnden. Efficient Bayesian sampling using normalizing flows to assist Markov Chain Monte Carlo methods. arXiv preprint 107.v:2107.08001 2021.
12

---

Published as a conference paper at ICLR 2025
Marylou Gabrié, Grant M Rotskoff, and Eric Vanden- Adaptive Monte Carlo augmented with normalizing flows. Proceedings of the National Academy of Sciences, 119(10):2109920119, 2022.
Tomas Geffner and Justin Domke. MCMC variational inference via uncorrected Hamiltonian an- nealing. In Advances in Neural Information Processing Systems, 2021.
Tomas Geffner and Justin Domke. Langevin diffusion variational inference. arXiv preprint arXiv:2208. 077443, 2022.
A. Gelman, J.B. Carlin, H.S. Stern, D.B. Dunson, A. Vehtari, and D.B. Rubin. Bayesian Data Analysis, Third Edition. Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis, 2013.
Neil J Gordon, David J Salmond. and Adrian FM Smith. Novel approach to Setiteearrnon-Gauusis Bayesian state estimation. IEE Proc. Radar Signal Proc., volume 140, pp- 107-113, 1993.
Paul Lyonel Hagemann, Johannes Hertrich, and Gabriele Steidl. Generalized normalizing flows flows via Markov chains. Cambridge University Press, 2023.
Carsten Hartmann and Lorenz Richter. Nonasymptotic bounds for suboptimal importance sampling. SIAM/ASA Journal on Uncertainty Quamtification, 12(2):309-346, 2024.
Jeremy Heng, Adrian N. Bishop, George Deligiannidis, and Arnaud Doucet. Controlled sequential Monte Carlo. The Annals of Statistics, 48(5), 2017.
M.F. Hutchinson. A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communication in Statistics- Simulation and Computation,
1076, 1989.
Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional dif- fusion for molecular conformer generation. Advances in Neural Information Processing Systems, 35:24240 242553, 2022.
Takeshi Koshizuka and Issei Sato. Neural lagrangian Schrödinger bridge: Diffusion modeling for population dynamics. In The Eleventh International Conference Learning Representations, 2023.
Guan-Horng Liu, Tianrong Chen, Oswin So, and Evangelos Theodorou. Deep generalized Schrödinger bridge. Advances in Neural Information Processing Systems, 35:9374-9388 9388, 2022.
Guan-Horns Liu, Yaron Lipman, Maximilian Nickel, Brian Karrer, Evangelos Theodorou, and Ricky TQ Chen. Generalized Schrödinger bridge matching. arXiv preprint arXiv:2310.02233, 2023.
Kanika Madan, Jarrid Rector-Brooke Maksym Korablyov, Emmanuel Bengio, Moksh Jain, An- drei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning GFlowNets from partial episodes for improved convergence and stability. In International Conference on Machine Learning, pp. 23467-23483, 2023.
Alex Matthews, Michael Arbel, Danilo Jimenez Rezende, and Arnaud Doucet. Continual repeated annealed flow transport Monte Carlo. In International Conference on Machine Learning, pp- 15196-15219, 2022.
Laurence Illing Midgley, Vincent Stimper, Gregor NC Simm, Bernhard Schölkopf, and José Miguel Hemández-Lobato. Flow annealed importance sampling bootstrap. arXiv preprint arXiv:2208.01889 2022.
Volodymyr Mnih. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Jesper Møller, Anne Randi Syversveen, and Rasmus Plenge Waagepetersen. Log Gaussian Cox processes. Scandinavian Journal of Statistics, 25(3):55--82, 1998.
Radford M Neal. Annealed importance sampling, Statistics and computing, 11:125-139, 2001.
13

---

Published as a conference paper at ICLR 2025
Radford M Neal. Slice sampling. The annals of statistics, 31(3):705-763) 2003
Kirill Neklyudov, Rob Brekelmans, Daniel Severo, and Alireza Makhzani. Action matching: Learn- ing stochastic dynamics from samples. In International conference on machine learning. pp- 15858-258888 PMLR, 2023.
Nelson. Dynamical theories Brownian motion. Press, Princeton, NJ, 1967.
Nikolas Nüsken and Lorenz Richter. Solving high-dimensional Hamilton- using neural networks: perspectives from the theory of controlled diffusions and measures on path space. Partial differential equations and applications, 2(4):48, 2021.
PDEs
Angus Phillips, Hai-Dang Dau, Michael John Hutchinson, Valentin De Bortoli, George Deligianni- dis, and Amaud Doucet. Particle denoising diffusion sampler. arXiv preprint arXiv:2402.0632,, 2024.
Lorenz Richter and Julius Berner. Improved sampling via learned diffusions. In The Twelfth Inter- national Conference on Learning Representations, 2024.
Marcin Sendera, Minsu Kim, Sarthak Mittal, Pablo Lemos, Luca Scimeca, Jarrid Rector-Brooks, Alexandre Adam, Yoshua Bengio, and Nikolay Malkin. On diffusion models for amortized inference: Benchmarking and improving stochastic control and sampling. arXiv preprint arXiv:2402.05098, 2024.
Yuyang Shi, Valentin De Bortoli, Andrew Camphell, and Arnaud Doucet. Diffusion schrödinger bridge matching, Advances in Neural Information Processing Systems, 36, 2024.
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse guided diffusion models for inverse problems. In International Conference Learning Representations, 2022.
Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In International Conference on Machine Learning, pp. 32483-32498. 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Jingtong Sun, Julius Berner, Lorenz Richter, Marius Zeinhofer, Johannes Müller, Kamyar Aziz- zadenesheli, and Anima Anandkumar. Dynamical measure transport and neural PDE solvers for sampling. arXiv preprint arXiv:2407.07873 2024.
Alexander Tong, Kilian FATRAS, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research, 2024.
Suriyanarayanan Vaikuntanathan and Christopher Jarzynski. Escorted free energy simulations: Im- proving convergence by reducing dissipation. Physical Review Letters, 100(19):19001, 2008
Francisco Vargas, Pierre Thodoroff, Austen Lamacraft, and Neil Lawrence. Solving schrödinger
bridges via maximum likelihood. Entropy, 23(9):1 134, 2021.
Francisco Vargas, Will Sussman Grathwohl, and Arnaud Doucet. Denoising diffusion samplers. In International Conference Learning Representations, 2023.
Francisco Vargas, Shreyas Padhy, Denis Blessing, and Nikolas Nüsken. Transport meets variational inference: Controlled Monte Carlo diffusions. In The Twelfth International Conference on Learn- ing Representations, 2024.
Nikhil Vemgal, Elaine Lau, and Doina Precup. An empirical study of the effectiveness of using a replay buffer mode discovery in GFlowNets. arXiv preprint arXiv:2307.07674 2023.
14

---

Published as a conference paper at ICLR 2025
Siddarth Venkatraman, Moksh Jain, Luca Scimeca, Minsu Kim, Marcin Sendera, Mohsin Hasan, Luke Rowe, Sarthak Mittal, Pablo Lemos, Emmanuel Bengio, et al. Amortizing intractable in- ference in diffusion models for vision, language. and control. arXiv preprint arXiv:2405.20971, 2024.
Robert J Webber. Unifying sequential Monte Carlo with resampling matrices. arXiv preprint 17Xi::9933.1258,, 2019.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on International Conference Machine Learning, pp. 681-688, Madison, WI, USA, 2011.
Hao Wu, Jonas Köhler, and Frank Noé. Stochastic normalizing flows. Advances in Neural Informa- fion Processing Systems, 33:5933-5999 5944, 2020.
Luhuan Wu, Brian Trippe. Christian Naesseth, David Blei, and John P Cunningham. Practical and asymptotically exact conditional sampling in diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Bingliang Zhang, Wenda Chu, Julius Berner, Chenlin Meng, Anima Anandkumar, and Yang Song- Improving diffusion inverse problem solving with decoupled noise annealing. arXiv preprint arXiv:240..01521, 2024.
Dinghuai Zhang, Ricky Tian Qi Chen, Cheng-Hao Liu, Aaron Courville, and Yoshua Bengio. Diffu- sion generative flow samplers: Improving learning signals through partial trajectory optimization. arXiv preprint arXiv:2310.02679, 2023a.
Qinsheng Zhang and Yongxin Chen. Path integral sampler: A stochastic control approach for sam- pling. International Conference on Learning Representations, 2022.
Xuan Zhang, Limei Wang, Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, et al. Artificial intelligence for science in quantum, atomistic, and continuum systems, arXiv preprint arXiv:2307.0842. 2023b.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
CONTENTS
A.1
Related works
16
A.2 Proofs and theoretical remarks
19
A.3 Algorithmic details and pseudocode
A.3.1 Computation of the Radon-Nikodym derivative
21
A.3.2 A practical algorithm
22
A.4
Benchmark target distributions
22
................................
A.4.1
Bayesian statistics tasks
22
A.4.2
Synthetic targets
24
A.5
Experimental details
25
A.5.1
Metrics and evaluation
................................ ................................
25
A.5.2
Design choices
26
A.5.3
Hyperparameter selection
................................
27
A.6
Additional experiments
29
A.6.1
Ablation studies of SCLD
A.6.2
Removing MCMC components
29
A.6.3
Timings Timings.........
30
A.6.4
Estimations of the normalizing constant
31
A.6.5
The learned annealing schedule
32
A.6.6
Mean field prior for SCLD
32
................................
A.6.7
Comparison to PDDS
A.6.8
Comparison with advanced SMC schemes
34
A.6.9
Convergence of different methods by iteration count
A.6.10 KL-based training of SCLD
35
A.1
RELATED WORKS
Adding to Section 1.1, this section provides additional related works.
SMC. SMC methods (Chopin, 2002; Del Moral al., 2006) describe general methodology sample sequentially from a sequence of (annealed) distributions. They rely on forward and backward kernels in order move from one distribution another and leverage resampling steps in between. Popular choices for the kernels include MCMC. However, while enjoying theoretical guarantees, they suffer from drawbacks such long mixing times and tedious tuning (Dai al., 2022).
SMC with learned kernels. To make the transition kernels more flexible and reduce the amount of manual tuning, previous approaches have been proposed to learn them (Wu al., 2020: Geffner & Domke, 2021). Combinations with SMC include the works by Bernton et al. (2019); Heng et al. (2017). While they propose learned SMC transitions, they not utilize neural networks (partially due to tractability issues). Bernton et al. (2019) build on the prior work of Heng et al. (2017), which uses ideas from optimal control to iteratively modify the prior distribution and transition kernels through an approximate dynamic programming approach. However, this requires the prior distribution to be conjugate with respect to the policy the underlying optimal control problem, among other drawbacks discussed in Bernton et al. (2019). The latter work, in turn, proposes the Sequential Schrödinger Bridge Sampler (SSB), which produces trained SMC sampler by applying sequential approximate iterative proportional fitting (IPF) learn the forward and backward kernels.
16

---

Published as a conference paper at ICLR 2025
Whereas the paper works in discrete time, we take continuous time perspective and, in doing so, obtain a family of simpler, unbiased training procedures. as well as reveal additional design choices like the ability choose the integrator. We also note that our objective is fundamentally different from IPF and, in particular, yields a different solution for a finite numbers of steps (see Vargas et al. (2024, Proposition 3.4)).
Methods combining SMC with neural networks include Annealed Flow Transport Monte Carlo (AFT) (Arbel et al., 2021), as well as its improved version Continual Repeated Annealed Flow Transport Monte Carlo (CRAFT) (Matthews et al., 2022). Those works use normalizing flows to transition between adjacent annealing steps. While achieving improved performance. the determin- istic nature of the transitions requires MCMC steps after the resampling steps to avoid particles collapsing to the same location. Moreover, the log-determinant of the Jacobian (or divergence of the drift for continuous time) is required. To avoid costly computations in high dimensions, one either needs to place architectural restrictions on the architecture or require the use of noisy estimators (such as Hutchinson's 's trace estimator (Hutchinson, 1989) for the divergence). We remark that there is also a series of works that combines normalizing flows with MCMC methods (Midgley et al., 2022; Gabrié et al., 2021; 2022; Hagemann et al., 2023).
Diffusion-based samplers. Works on diffusion-basse samplers such as Path Integral Samplers (PIS). Denoising Diffiusion Samplers (DDS), Time-reversed Diffusion Samplers (DIS), and others introduced by Zhang& & Chen (2022); Berner et al. (2024); Vargas et al. (2023; 2024); Sendera et al. (2024); Sun et al. (2024) have focused transporting prior the target distribution using con- trolled stochastic differential equations (SDEs), where the control is learned by minimizing suitable divergences between induced measures on the SDE trajectories; see the framework described in Section 2.2. As a historical note, some of the ideas of diffusiin sampling were anticipated in earlier works such as Vaikuntanathan & Jarzynski (2008). In this work, we aim to harness their flexibility together with the power of SMC. Orthogonal to our work, techniques from diffusion models have been employed to approximate the extended target distribution needed in AIS methods (Doucet et al., 2022; Geffner & Domke, 2022).
Subtrajectories. In our work, we utilize the idea of dividing a path measure into sequential sec- tions. This bears resemblance to the concept of subtrajectories setting in the context of GFlownets (Zhang et al., 2023a; Madan et al., 2023), and thus we will also use this term. While conceptually similar, the latter work only proposed subtrajectories an alter- native training loss, whereas we use them to facilitate integration with SMC methods. Additionally, their formulation requires learning the evolution of the SDE marginals, whereas adapt recent Controlled Monte Carlo Diffusions (CMCD) (Vargas et al., 2024) to get rid of this requirement.
introduced in discrete-time
SMC with diffusion-based samplers. To the best of our knowledge, the only prior work on combining diffusion- based methods with SMC steps is the Particle Denoising Diffusion Sampler (PDDS) (Phillips et al., 2024), where the backward kernel is chosen to be the noising diffusion and the forward kernel the approximate (learned) time-reversal. While also inspired by diffusion-based samplers, PDDS significantly differs from our approach. First, we take a more general continuous- time perspective, allowing us more freedom in design choices while still recovering the (discrete- time) setup of PDDS as a special case (i.e., where we use one Fuler- Marumaya step per subtrajec- tory). Next, their setup requires learning potential functions and relies on automatic differentiation to compute the control, which can be unstable and challenging to optimize. Indeed, PDDS was em- pirically found to require variational approximations for the prior distribution to train stably, which has certain drawbacks (see Appendices A.6.6 and A.6.7). Moreover, it uses an alternating train- ing setup that uses (approximate) samples from the partially trained model, whereas we train model end-to-end, i.e., our setup is the same during training and inference. We empirically com- pare methods and discuss the impact of this difference in training methodology in Appendix A.6.7. We additionally compare different SMC-based methods in Table 1. Lastly, we note that the recent concurrent work by Albergo & Vanden-Eijnden (2025) also utilizes resampling steps as part of a learned SDE-based sampling algorithm.
Diffusion- based generative modeling. As outlined in our introduction, sampling problems are sub- stantially different from problems in generative modeling, where samples from the target distribu- tion are provided. However, many successful techniques from diffusion-basee generative modeling, such as SDE integrators, noise schedules, and probability flow ODEs, can be translated to diffusion- based samplers. Loosely related to CMCD, and thus SCLD, are (entropic) action-matchnng ap-
17

---

Published as a conference paper at ICLR 2025
proaches (Neklyudov et al., 2023), where the intermediate distributions are prescribed via samples as compared to (unnormalized) densities in our setting. In both settings, there exist unique gradi- ent fields representing the optimal controls, which can be characterized solutions to infinitesimal Schrödinger bridge problems at the intermediate distributions, i.e., minimizers of the kinetic energy (see Vargas et al. (2024, Proposition 3.4) and Neklyudov et al. (2023, Appendix B.3)). As described in Remark A.I, we could replace the CMCD framework with more general bridges, i.c., arbitrary, learnable density evolutions as considered in Richter & Berner (2024), at the cost of learning the (un- normalized) marginals with a separate model. A corresponding objective in generative modeling has been considered by Chen et al. (2022). While such approaches do not exhibit unique solutions, one can additionally minimize the KL divergence of the learned path measure to a reference measure, typically given by a Brownian motion, which leads to dynamic Schrödinger bridge problems (i.e., entropy-regularizee optimal transport). This has been explored by, e.g., Vargas et al. (2021); De Bor- toli et al. (2021); Shi et al. (2024) in the context of generative modeling, and we refer to Koshizuka & Sato (2023); Liu et al. (2022; 2023) for extensions beyond kinetic energy minimization (related to mean-field games).
Finally, we mention that generative modeling frameworks that allow likelihood computations can also be used for sampling problems. Specifically, one can optimize objectives from generative mod- eling (e.g.. score-matching objectives) using approximate samples from the target distribution ob- tained from the partially trained model together with importance sampling based on the likelihoods of the samples. This can be viewed as a version of the cross-entropy method and is used, e.g., in Jing et al. (2022. Section 3.6) for diffusion models and in Tong et al. (2024, Appendix C.2) for flow matching. However, a mismatch of the high-probability regions of the proposal (given by the partially trained model) and target distributions often leads to high variance in high-dimensional settings. We note that PDDS can be viewed as very elaborate version of such an approach, counter- acting the aforementioned problems by incorporating SMC steps into the proposal as well as training with a combination of target-matching and score-matching objectives. We compare to PDDS in Ap- pendix A.6.7.
Diffusion-based posterior sampling and stochastic optimal control. For our considered sam- pling problems, only assume minimal to no prior knowledge of the properties of the target distribution. However, for sampling from posterior distributions arising from Bayesian inference
problems, one can decompose the target as Ptarget = Px r( pxprix(ul)) (ul-) where y is given measurement and px and yy|x are the prior and likelihood, respectively. In our Bayesian statistics tasks (see Appendix A.4), the prior px is given by simple, tractable distribution, and we do not incorporate knowledge about the prior into our framework.
However, for certain problems, the prior can also be more complex, e.g., in inverse problems on image, audio, or video distributions. Assuming - different from our setting - that samples from the prior px are given, recent methods leverage diffusiion priors, i.e., diffusion models pre-trained on Px. to simplify sampling from Ptargeti see, e.g., Chung et al. (2022a:b); Song et al. (2022; 2023); Boys et al. (2023); Zhang et al. (2024). Using the decomposition of Ptarget, they draw approximate samples from Ptarget based on approximations of the likelihood score (i.e., the difference of the score for the noised posterior and prior distributions) during inference. For instance, the common reconstruction guidance approximates this score by the (scaled) gradient of the log-likelihood eval- uated at the denoised sample obtained via Tweedie's 's formula and the pre-trained model. While such plug and play approaches yield impressive results for high-dimensiona distributions without additional training, they typically lack thcoretical guarantees and typically suffer from instabilities and mode collapse.
At the cost of simulating multiple particles during the generative process, the bias originating from approximating the likelihood score can be eliminated (in the limit of infinitely many particles) by leveraging ideas from SMC, i.e., by computing importance weights and interleaving the generative process with resampling steps (Wu et al., 2024). Taking into account an additional training phase, one can also obtain theoretical guarantees by writing the likelihood score as solution to an stochas- tic optimal control (SOC) problem (as in DDS, however, with the pre-trained diffusion model as a reference process; see Didi et al. (2023, Section 2.4) and also Venkatraman et al. (2024). The SOC problem can then be solved using, e.g., the varivane divergence. While such posterior sampling
Whiile Jing et al. (2022) use the probability flow ODE to obtain likelihoods, one could alternatively obtain importance weights in path space; see the references diffusion-based samplers ahove.
18

---

Published as a conference paper at ICLR 2025
Prio
Pprior
Peyl0
P1
Pigiti
P2
PT|t2
Ptarget
Figure 5: Illustration of annealed importance sampling along geometric path, where we consider either one (top arrow) three (bottom arrows) transition steps from the prior the target.
approaches assume more structure than our considered sampling problem and rely on pre-trained diffusion prior, one could also adopt the idea of SCLD to such settings (see also Remark A.I), which we leave to future work. This would basically correspond to combination of the approaches by Wu et al. (2024) and Didi et al. (2023), where the likelihood score is learned but training is facilitated by leveraging SMC steps.
We note that ideas similar Didi et (2023) have recently also been used for fine-tuning diffusion models (where PrIx(yl>) corresponds to a reward function), using adjoint matching to minimize the KL divergence instead of, e.g., the log-variance divergence, to solve the SOC (Domingo-Enre) et al., 2024). While we propose to use the log-variance divergence to allow off-policy training and reduce variance (see Section 2.3), we note that adjoint matching and related approaches (Domingo- Enrich et al., 2023; Domingo-Emiiiiiie 2024) could also be used for SCLD solve the SOC problems in each subtrajectory.
A.2 PROOFS AND THEORETICAL REMARKS
In this section, we provide additional remarks
theory and the proof of Proposition 2.3.
Remark A.1 (Generalizations)) We note that, in principle, prescribing an annealing, i.e. px= n, not strictly necessary, and one could instead consider general bridges allowing for arbitrary density evolutions between the prior to the target. This, however, would come with the additional challenge of learning the (unnormalized) log-density lo of the controlled process, see, e.g., Richter & Berner (2024, Appendix A.7), which could make optimization potentially more difficult. Moreover, we can only use the approximate densities for the MCMC refinements as compared to using the target density in case of preseribed annealing. While the general bridges do not exhibit unique solutions, one can consider the case studied in diffusion models, where the control vo of the reverse- time process in (7) is fixed such that Yσ is approximately distributed as Pprior- Nelson's 's identity in (8) shows that it is sufficient to learn the log-density log py- and the optimal control can be computed using automatic differentiation, as leveraged in Phillips et (2024); Richter & Berner (2024). However, this can potentially be unstable and computationally more expensive.
Remark A.2 (Connections to reinforcement learning). The objectives of diffusion samplers be viewed as stochastic optimal control problems; see, e.g., Dai Pra (1991); Zhang & Chen (2022); Berner et al. (2024). More generally, stochastic optimal control problems can be under- stood as versions of maximum entropy reinforcement learning in continuous time and space; see, e.g., Domingo-Enren Enrich et al. (2024, Appendix C). Specifically, the prior distribution Pprior together with the control u define policies and transitions via the SDE (6) (or, in discrete time, via the tran- sition kernels in (32) given by the Euler Maruyama scheme). This allows the transfer of successful ideas from reinforcement learning to diffusion-based samplers. Motivated by previous work (Zhang et al., 2023a; Richter & Berner, 2024; Sendera et al.. 2024), we propose to off aff-policy training with prioritized replay buffers for SCLD, which is enabled by the log- variance loss (see Section 2.3).
19

---

Published as a conference paper at ICLR 2025
Proof of Proposition 2.3. We follow the proof ideas from Nüsken & Richter (2021, Proposition 5.7), however, need be careful since the reweighting of the measure P [ta-iifa] T u. 1,6]] is done w.r.t. a measure on the previous time interval [tm, tn- Let us recall the KL divergence (16), namely
D = DKL (1 (Prnn-id ( -Ex (i-idi [log (1,
-
x~ -- (mm. [log
wp (itmttnni(()
di ((3)) leid
where we abbreviate W[x,d] =
Using the analogous abbreviation w
of for the product [(a.t]
ap [e.1]
measures, we note that
Var DKr)
(1 poo [ta-1,t]] (fa=1)ta) )
S1 log (wer (1 (x)) (tmdnn)) (x)]
= K
Var
x War
Pmital
(21)
Mr D2
K
where
M, = E Rest
log² (innniion) (X)) (
(1 Memitn (2) (x)) (
30
and
log (wser 2itml (x]](x)) wol
(mmtt ((x)]
Dj -E -xx
(22)
--Exppa [in-1:fa]
[log ( (n- (ex-) = DKT. (on-sita)
11 1881 (on..taa) ID.
Moreover, can compute
log (8) ( (2))
(woe (mm ( (x)))
Mj E X 1881
log? (wferr (x)) (imme .1 ,x(x) )
E Bor jor (m))en)
(23)
Tog (wl) (x)) log (wit) (x)) (w
(mm (x)) (x)
E X (mnctte
IMC + I(I 1(1-1)
where un denotes the weight for the i-th factor of the product measure and we abbreviate
[log² (wie. (X]] (wim.en ,xx)) D²
M = Ex-Fpmm..
(24)
(mm.tn)
and
[(uppmtth Ex (emit-all (x)))
C = E Ex~P (emital tul
(25)
xx² PP (tmit 1] ||IP PPimeete .)) +121
Combining the definition of the relative error with (21), (22), and (23), we obtain that
((K) P 2.1..
) o Ite
s.I..)
M₁ D2
C1/2
MC + D2(T - C²ID2
KD}
VR
C
which, in view of (24) and (25), proves the claim.
As already stated in the main text, note that the log- variance divergence, defined in (18), does not scale exponentially in the dimension, as already proved in (Nüsken Richter, 2021, Proposition 5.7). For convenience of the reader, let explicitly verify that this statement also holds in
20

---

Published as a conference paper at ICLR 2025
setting. To this end, first note that
DWv
IPo In sraa)) Var x-QQ [1og (wee 1. (x)]
(26a)
Pnn 15f,
Σ Varx [log
(x = ID ( (Bnn-ee P pn (di)))))
(26b)
where we recall that Q an arbitrary reference measure. Following Cho et al. (2005), the sample variance satisfies
Var
[DN
(K)
p (tn
1 ( ( ) = K n4 K - DRV
K -3 o0 (F (P(
1 2
(t-- 1.t.)
1 [ stt)
(27)
where
[
(1 (o) - Exwqor [log (x))))))))((())) 17
14 = Ex
(28)
We can calculate
14 Ex-Qor Σ
log (wite (x))) Ex- [log (1)
(29a)
[[log (w)(a-) - Exno [log (wren (wnee-ninn (x)())
TEx
(29b)
2 Ex~o (log (1(())))))())) - Ex-o [log (w (m)(a=,.,,
+ +
(29c)
where we have used the fact that, for instance,
(1) (x)) - -xxwe [log (x) (loR
Ex~q0>
(30)
(10g
(w) (t (x. - Exno [log (wtt (W 1itm.) (x)
= 0,
for j.. Combining this with (26), it follows that Var [
p
1000
(tn-1: ) = 0(13).
[tn-1
Recalling the definition of the relative error, ((K) = Var ((k LV scale exponentially in
(K)
1/2 DEv, we see that it does not
A.3 ALGORITHMIC DETAILS AND PSEUDOCODE
We first provide formulas compute the Radon-Nikodym derivative (RND) and the forward and backward kernels in discrete time. Then, we give an implementable method in Algorithm 2 and provide details the resampling step and training with buffer. Note that uniform discretizations within subtrajectories by adapting the times hi, i = O,..., N, accordingly.
also non-
A.3.1 COMPUTATION OF THE RADON-NIKODYS NIKODYM DERIVATIVE
As in Vargas et al. (2024), we obtain an approximate, computable formula for the Radon-Nikodym derivative in Lemma 2.2 between the (n - 1)-th and n-th time step, given by
* (n) dP [n- (x) ~
nL II
(X- X..). (31)
Wit. 1 l.,., = dP (n-1)1)))
TT (((n-117 11-1)
where the transition densities for the forward and reverse-time SDEs, coming from the Euler- Maruyama discretization in (19), are given as
p (X.IX.) (Xx X, u(X,, Fu (X, = (XX + (o2V log ग - u), t) - s), o - =)).
s). 02) - -))
(32)
In practice, in line with Vargas et al. (2024), we parameterize the control as
U + o log ,,
(33)
2
21

---

Published as a conference paper at ICLR 2025
where Ui is parametrized by neural network. When üe is initialized the zero function, recover an annealed form of Langevin dynamics (Welling starting point for optimization.
Teh, 2011), providing an improved
A.3.2 A PRACTICAL ALGORITHM
In Algorithm 2, we give a practical and detailed version of Algorithm 1.
Algorithm 2 SCLDD-oorwardaus
Require:
Target Prarget (learnable) prior Pprior N (μo, diag(exp(2(s)) number of subtrajectories N, steps per subtrajectory L and step size h, annealing schedule Bo as in (35), noise schedule Ø, control u given by neural network & in (33), number of particles K
1: Sample from prior (by reparamerrization): X (1:K), ~ Pprior
> Independent for each particle
Initialize (unnormalized) importance weights: w/ (1:K)
3: Evaluate control and prior: ul uxx0 (8(18),0
,0) and Porior (X, (8))))))
4: for 10 do
Note that nLh
5: for = (n 1)L
to TL do
D Consider the time interval [(i 1)/, ihi]
6:
Suler-Marayama simulation: 8 ((1:K) ~ 19 (-1 as in (32)
> Sec (19)
7:
Evaluate control: u(X ((X())),,00 th)
8:
Evaluate (unnormalized) annealing: n (((()))) ,bn) (pprior
()) ( (1())))))
Ptarget
See (20)
9:
Compute RNDs: (1:K) ((n-1. as in (31)
For every k. use Xah 8((0)
10:
Update weights: wa
(1:K) = wn-1 w tn)
(1:K) (1:K)
11:
Resample: {(1:K), (1:K) = resample(( nt. ((.K) wn (..K)
at. wun
D. See Algorithm 5
12:
MCMC step: Update ((1:K) with tn
(n, invariant kernel
13:
return RNDs (w..)) (., ) weights (w! )o trajectories (1::k)
) log estimate DN log E (*) w (n-) ELBO Em= E (4) log (eg(wi (w)))
(nn-taaa))
Prioritized replay buffer. We give the exact algorithm our replay buffer in Algorithm 4. We note that there are many alternative possibilities for choosing the buffer priority (including by importance weight), which we leave to future exploration. Moreover, as traditional replay buffers (Mnih, 2013), there is an option to perform multiple gradient steps per simulation reduce computation costs.
Resampling. The work of Webber (2019) shows that there is great scope to design resampling meth- ods. However, in line with prior work, we opt to use the simple adaptive multinomial resampling for which pseudocode is provided in Algorithm 5.
Annealing Schedule While we choose to learn the annealing schedule, an alternative approach that takes advantage of the flexibility of our continuous-time perspective is set /3(1) / and instead adaptively choose the time discretization. For example, the constant-ESS scheme annealing schedule Buchholz et al. (2021) is commonly used scheme in the SMC literature and can be easily adapted to the SCLD setting for both training and sampling. We leave the investigation of this idea future work.
A.4 BENCHMARK TARGET DISTRIBUTIONS
Here, we introduce the target densities considered in our experiments more formally. Most of these are standard benchmarks taken from, e.g., Heng et al. (2017); Arbel et (2021); Geffner & Domke (2022); Richter & Berner (2024); Blessing et al. (2024).
A.4.1 BAYESIAN STATISTICS TASKS
For these tasks, groundtruth samples available.
22

---

Published as a conference paper at ICLR 2025
Algorithm 3 SCLD-Training
P See Algorithm 4 for training with buffers.
Require: Number iterations 1, initial parameters 0 optimizer update update, inputs Algorithm
1: for to do
Run Algorithm (w ((n-1it))
(1:K) )am (wn
(
-= SCLD-ForwardPase (0(9)
3: LV then
Trajectories {(1:k) detached during forward pass
4: Compute loss: L E=E
E k=
log (4)
K Σ ii log w/ (tal1.ta))
in-1tal
5: else if KL then
6:
Compute loss: -= En n1
Σ detach(w (4) log wi [nn-166]
R
Compute gradient w.r.t. parameters: GO
V otc
8:
Optimizer step: g(i=1) = update(0 (3
We use Adam
return Optimized parameters ((1)
Algorithm 4 SCI.D-Buffer-Traiine
Require: Buffer (B, (B,,),. for every sutrajectory, inputs for Algorithm
I: for to I 1 do
2:
Run Algorithum 2: (1 1.h)) R= ) ((L:K) SCuD-ForwardPass
(8(1) with K particles
3:
n to do
4:
Store subtrajectories: (8 (8(1:))) (1= into with weights w ( [f,]" replacing oldest entries
(1:K)
5:
Sample from buffer: (1:K/22 B. with probability proportional to buffer weights
6:
Recompute RNDs: (1::K/2) (fn-116 for detached (1:K/22 using (31) and current parameters g(1)
7:
Update buffer: Set (In-1da) weights for (1:-//2)
(1:K/22)
Updating all B particles is too slow
8:
Sample other half from simulation: Wi (in-itt))
,=(K/2+1:K) from w (1:K)
(nniit uniformly without replacement
Compute log- tog-variance loss: L En' Σ
log (T(k)
lt, 1.61 # E log (7()) Bt 1.fml)
Compute gradient w.r.t. parameters: G Vote C
Optimizer step: g1++1) = update((()) (GG))
We use Adam
12: return Optimized parameters ((1)
Bayesian Logistic Regression (Sonar and Credit). We used two binary classification problems our benchmark, which have also been used in various other works compare different state-of- the-art methods in variational inference and MCMC. Specifically, we assess the performance of Bayesian logistic model with
Ptarget (x) p(z) II Bernoulli sigmoid u.))
on two standardized datasets ((u,, 8:)) y namely Sonar (d 61) and German Credit 25) with n = 208 and n = 1000 data points, respectively. We choose p = N (0, I) for Sonar and for Credit (in line with the code Blessing al. (2024) which omitted the prior).
Random Effect Regression (Seeds). The Seeds (d 26) target uses random effect regression model given by:
Gamma (0.01, 0.01)
01 02 ~ ~N0, 10)
b. ~ N 0,
i = 1,...,21.
logits; = An + 01 + 023/ + a12 + b, rt Binomial (logits,, N.),
i == 1, 21,
1,...,21. 21.
The goal is to do inference observed values for F;, Vr. and N; from a dataset modeling the germination proportion of seeds; Geffner & Domke (2022) for details.
the variables т, 0u, 01 02 012 and b, for i 1,... 21. given
23

---

Published as a conference paper at ICLR 2025
Algorithm 5 Adaptive Multinomial Resampling
Require: particles X (1:K), unnormalized weights w (1.K)
1: Normalize: W (k) Σ w((, k L....,K
Compute ESS: ESS = 1/ Σ 1 (WrM))2
if HSSS- aK then
b take a 0.3
k 1 do
Sample index from categorical distribution: i fI,..., K} with probabilities W(lk)) Define resampled particle: (*) X()
7:
Reset weights:
W(1:K) 1/K
8:
clse
9:
Keep particles:
X(::) ((=k)
10: return resampled particles ( updated and normalized weights W(I:))
Time Series Models (Brownian). The Brownian (d 32) model corresponds to the time dis- cretization of a Brownian motion with Gaussian observation noise:
Qinn (ogNormal 2).
Olobs Nogmarrall00 2),
I1 NO0 arinn);
([ Cinn); 3/1 (xxx; Clob);
i = ...,.,3 30,
, 30.
Inference is performed over the variables Ciiu Oobs and {a, j given the observations {y:) and {ys 12 (i.e., the middle observations are missing); see Geffner & Domke (2022).
Spatial Statistics (LGCP). The Log Gaussian Cox process (LGCP) is popular high- figh-dimensional task in spatial statistics (Møller al., 1998), which models the position of pine saplings. Using d = 40 40 1600 grid, we obtain the unnormalized target density by
cxp (x.))
Prarget No; E)
II exp Till
where y is given dataset and μ and Σ the mean and covariance matrix of the given prior. We use the more challenging unwhitened version; see Heng et al. (2017); Arbel et al. (2021) for details.
A.4.2 SYNTHETIC TARGETS
For these tasks, groundtruth samples are available.
Robot. The Robot targets (Arenz al., 2020) (Robot1, Robot4) aim at learning joint configurations of a 10 degrees-of of-freedom planar robot, parameterized by
(ai....,
such that it reaches a desired goal position while enforcing smooth configurations. The target density given by
Purget (a) Peont (a)
where Pooof enforces smooth configurations and Pean penalizes deviations from the goal position. Peeaf is modeled as zero-mean Gaussian distribution with diagonal covariance matrix, where the angle ai of the first joint has variance of 1 and the remaining joint angles Q.g have variance of 4 x 10 -
Formally, define the locations of the robot joints by
I;(a)
cos(
0, O,..., .....,,... 10,
j=1
vi(a) Σ sin(a;);
0, ..... 10.
24

---

Published as a conference paper at ICLR 2025
In the Robotl task there is goal at (7, 0). and we specify
Peart =N ((((()))) ((110(a)) ( 10,10-11 (0).10--))
(no((a))
(34)
i.e., Gaussian distribution centered the Cartesian coordinates of the goal position, with variance of 10-4 in both directions.
In the Robot4 task there mum over the four respective Gaussian distributions as in (34) (up to constant of proportionality) Groundtruth samples are generated by long slice sampling runs (Neal, 2003) and taken from the repository of Arenz et al. (2020).
goals at (=7,0) and (0, 27), and Pein is given by the maxi-
Mixture distributions (GMM and MoS). For the GMM and MoS tasks, we define a mixture dis- tribution with m components as
m
Σ
Ptarget
Pr.
777
i=1
The Gaussian Mixture Model (GMM), taken from Blessing al. (2024), consists m 40 mixture components with
Pt - (u, I),
f4: 44( 40).
where Ul(l, u) refers uniform distribution on [t, u], We take d = 50 for the main experiments.
The Mixture Student's 1-distributions (MoS), taken from Blessing et al. (2024), comprises m 10 Student's s t-distributions 12, where the refers to the degree of freedom. Specifically, we use
= t2 +
H4 4a 10).
where μ4 refers to the translation of the individual components, and take d 50. For both the GMM and MoS tasks, the H4 's are fixed throughout experiments, i.e., selected with the same random seed.
Funnel. The Funnel target introduced in Neal (2003) challenging funnel-shaped distribution given by
Prarget (I) = Ax11; 0, o²)
I10: 0, exp(x1))),
with 0² 9 for any number of dimensions d 2 2. We take d = 10 in main experiments.
Many-Well (MW). typical problem in molecular dynamics considers sampling from the station- ary distribution of Langevin dynamics. In our example we shall consider a d-dimensional many-well potential, corresponding to the (unnormalized) density
d
Purget (x) = exp
8)2
2
In line with Berner al. (2024); Sun al. (2024), we take d m 5, and 4, leading 2m = 32 well-separated modes. Groundtruth log Z and samples can be obtained by noting that the distribution factors over dimensions.
A.5 EXPERIMENTAL DETAILS
In this section, we describe the experimental setup and evaluation protocol. We also discuss design choices for our main experiments well how our hyperparameters are selected.
A.5.1
METRICS AND EVALUATION
Maximization of the ELBO. The ELBO refers a lower bound on log Z. This is a classic benchmark for samplers, and higher ELBOs usually associated with precise sampling from discovered modes. However, the ELBO is not necessarily indicative of mode collapse; see Bless- ing al. (2024) and Appendix A.6.4 for details.
Minimization of the Sinkhorn distance. The Sinkhorn distance W2 is an optimal transport (OT) distance. When computed between of generated samples and groundtruth set of samples
25

---

Published as a conference paper at ICLR 2025
from the target (when the latter is available), this gives estimate of the OT distance from the distribution generated by the sampler to Ptarget- As discussed further in Blessing et al. (2024), low OT distances are associated with good mode coverage (i.e., avoiding mode collapse).
For both ELBO and optimal transport evaluation, we follow the protocol of Blessing et al. (2024). In particular, we use the Sinkhorn distance as implemented in Cuturi al. (2022) and use standard formulas for the ELBO computations of our baselines. For SCLD, the ELBO computation is stated in Algorithm 2. We compute all performance criteria 100 times during training using 2000 samples, applying rumning average with a length of 5 over these evaluations to obtain robust results within single run. To ensure robustness across runs, we use four different random seeds and average the best results from each run. As we use the same evaluation protocol as Blessing et al. (2024), we re-use their results for DDS and PIS whenever available.
As discussed in Blessing et al. (2024), ELBO metries are insensitive to mode collapse and, as such, may not accurately reflect the quality of samples on multimodal tasks. As groundtruth samples are available for the synthetic tasks considered and due to their generally multimodal nature, we report Sinkhorn distances for these tasks.
A.5.2
DESIGN CHOICES
We follow the following principles:
SCLD follows design choices of other methods when these are shared.
SCLD reuses the hyperparameter choices of baseline methods when shared such that it is not tuned excessively.
Baseline methods should be given as much or more computational budget compared to SCLD.
General remarks. For SCLD, CMCD, DDS, and PIS we take the convention that T = 1 as rescal- ing time is equivalent to rescaling the noise level. Since the objectives of DDS and the Time-Reversed Diffusion Sampler (DIS) (Berner et al., 2024) only differ by choice of the reference process (see also Berner al. (2024, Appendix A.10.1), Richter & Berner (2024, Section 3), and Vargas et al. (2024, Appendix C.3)), we do not explicitly compare against DIS in this work.
CMCD and SCLD. As SCLD and CMCD share numerous design choices, mostly follow the choices of CMCD as in Vargas et al. (2024). In particular, we opt to learn the prior as well as the annealing schedule. For the former, we define Pprior N(ue diag(exp(2(,))
In other words, we parameterize the Gaussian prior through its mean pa I and logarithmic standard deviations lg E IR, initialized to (000, I), i.e., Hi = 0 and (fo): = log( for some a > 0 (referred to as initial scale) be tuned. We update μe and lo via the parameterization trick as training progresses. For learning the annealing, we parameterize the schedule in (20) for every j E {I...., NL} by
softplus(0.)) (0,)
Bo(ih)
(35)
Di= softplus(0.)
NL
where 0, e R are learnable parameters. We choose the buffer size to be 20 times the training batch size, i.e., B = 20K. Moreover, we parametrize the control as in (33). For SCLD, we use the subtrajectory settings from Section
CRAFT. We use the implementation by Blessing et al. (2024), following the standard settings of Matthews et al. (2022). Specifically, we employ diagonal affine flows as the transport maps.
SMC operations We use the same resampling strategy and MCMC kernel for CRAFT, SMC, and SCLD. In particular, every SMC step consists of adaptive resampling with threshold of 0.3K, followed by one Hamiltonian Monte Carlo (HMC) step with 10 leapfrog steps. For details on the advanced SMC schemes (SMC-ESS and SMC-FC), we refer to Buchholz et al. (2021) and Ap- pendix A.6.8.
Optimization and batch size. We utilize the Adam optimizer for all methods that require learning. We also found that clipping gradients to 1 was important for stable training all diffusion-based methods. We use batch size 2000 for training except for LGCP, where batch size 300 is used. We always evaluate with K 2000 particles.
26

---

Published as a conference paper at ICLR 2025
Number of annealing / diffusion steps. For SMC, DDS, PIS, CMCD, and SCLD in the main experiments, we fix 128 steps. In particular, we have = 128/N /N for SCLD. For CRAFT, we sweep over [4, 8, 128] annealing steps (which also define the number of SMC operations).
Number of training iterations. We select the number of training iterations such that all methods given roughly the same number of target function evaluations (NFEs) for given number of SMC operations or subtrajectories N, evaluations per SMC operation M, and annealing or diffusion steps per subtrajectory L. In our setup, M = 10 due the 10 leapfrog steps in HMC, (N, L) 128) for DDS, PIS, and CMCD, and N E (4, 8, ]28} for CRAFT (with = 1) and SCLD (with L = 128/N). As a reference value, we use 40000 iterations for DDS and PIS as in Blessing et al. (2024). We report the chosen number of iterations for each method in Table 4.
Table 4: Number of training iterations for our considered method depending on the number of SMC operations or subtrajectories N. The last rows show the approximate number of target function evaluations (NFFs) per particle in each iteration w.r.t. the number of evaluations per SMC operation M and annealing or diffusion steps per subtrajectory L..
CRAFT
SCLD
DDS, PIS, CMCD-KL, CMCD-LV
N
x 10a
105
2.5 104
N = 8
5 x 104
N 128
3 10
3 x 10
Approx. NFEs per particle
MN
MN +
Our setup
L = 1, M = 10 LN = 128, M = 10
N = 1, = 128
We note that all baselines converged satisfactorily within the given iteration budget. Moreover, the generous budget of 40000 iterations for DDS, PIS, and CMCD required running for 4 - 20 times as long SCLD's training process equivalent architecture for our considered tasks (see also Table 9).
A.5.3 HYPERPARAMETER SELECTION
General remarks. We follow the spirit of experimental design in Blessing et al. (2024) to fairly compare SCLD with our diverse range of baselines. We describe the search space and selection procedure below. We select the best configuration based on the target metric and single seed. We note that alternative experimental setups such as done in Vargas et (2024) are possible, leveraging the ability of CMCD and SCLD to learn further hyperparameters end-to-end or use variational mean field approximations (see Appendix A.6.6) instead of a grid-search.
Prior scale. For all methods that require (O, o²1) prior, we sweep over in [0.1, 10] for tasks where we have no information about the target. For GMM40 and MoS tasks, we know that the initial scale should be around 40 and 15, respectively, by construction of the problem, we fix these values for all methods. Similarly, for the Robot tasks, we know that the coordinates correspond to radial angles, so we the initial scale
cover the [-,, range.
Diffusion noise schedule. For iiffusion-based samplers a noise schedule a as in (6) needs be spec- ified. For PIS, we use linear noise schedule as in Zhang Chen (2022), and for DDS, CMCD, and SCLD we use a cosine schedule as in Vargas et al. (2023). Both noise schedules are parameterized by "minimum diffusion" and a "maximum diffusion" coefficient. We set the minimum diffusion noise level to 0.01 for all tasks and methods except the Robot tasks, where we set it to 0.001. For all methods and tasks we perform grid searches over the maximum diffusion parameter. For all tasks except the Robot and GMM40 tasks we search in [0.1, 1. 10). Due to the large initial scale of GMM40, we search the maximum diffusion parameter over [5, 10, 20). For Robot, we search it in [0.003, 0.03, 0, 3] instead of the usual grid due to the constructed sharpness of the modes.
Architecture. For hyperparameter selection on CMCD-KL, CMCD-LV and SCLD, we use the PIS- GradNet architecture (with detached score and 2 hidden layers of 64 units) for all diffusion-based methods as in Vargas et al. (2023). However, for CMCD-KL, we found that using the simpler MLP architecture deseribed in Vargas et al. (2024) (which we term PISNet) gave significantly better per- formance than PISGradNet on most tasks. As such, to ensure strong basclines for CMCD-K KL and CMCD-LV, we also select the best architecture among PlSGradNet and PISNet (with 2 hidden layers
27

---

Published as a conference paper at ICLR 2025
of 90 units ensure similar parameter counts), re-sweeping learning rates necessary. For SCLD we use PISGradNet on all tasks.
CMCD-KL, CMCD-LV, DDS, and PIS. We jointly grid search the initial scale and maximum diffusion along with the learning rates. We use one learning rate for the model üg and prior Pprior: and another for the annealing schedule B. We sweep over the learning rate of the model
in [10-3, 1 9] and learning rate of the annealing schedule in [10- 10-3]. We perform model selection using 8000 gradient steps instead 40000 due to the large grid.
SMC. For all tasks not present in Blessing et al. (2024) (namely the Robot tasks and MW54), search the same parameter grid used for other methods for the scale of the prior, jointly with HMC step sizes. For all tasks present in the benchmark,
re-use their results and SMC configuration for SCLD and CRAFT. We tuned the step size of HMC, using different step sizes for < T/2 and t > T/2 (where time corresponds to annealing steps in CRAFT) in the same fashion as Blessing et al. (2024). We search step sizes in the set [0.001, 0.01, 0.5, 0.1, 0.2]
CRAFT. We sweep over [4, 128 for the number of annealing steps, jointly with the prior scale 10-1,11--0 10-9)) and choose the best value. As in Blessing et al. (2024), we re-use the HMC step sizes that were tuned for SMC. Our results uniformly reproduce or improve upon those presented in the aforementioned paper due to the extended search space.
and the learning rate (also in [10-3 10-1,
SCLD. To ensure fair comparison with baseline methods, we reuse the chosen scale and diffusion parameters of CMCD-LV as well as the HMC step sizes tuned for SMC. The only grid search we perform for SCLD is over the learning rate of the model [10 101 and the learning rate of the annealing schedule in [10-2 10-$1. However, reflected in Table 5, setting all learning rates to 10-3 typically turned out to be a robust choice.
Table of hyperparameter choices. In Tables 5 and 6 we present the tuned hyperparameters we obtained. Please note that "PGN" refers to the PISGradNet architecture, whereas "PN" refers to the PISNet architecture. We refer to Blessing et al. (2024) for further details and design choices for PIS and DDS. In Table 6, we specify the hyperparameters for DDS, PIS, and SMC on tasks not present in Blessing et al. (2024).
Table Hyperparameter choices of our considered methods for the tasks in Blessing et al. (2024).
GMM40 (504)
Amwnian (324)
LGGPP (16004)
Funnel (10d)
(S))((d)
Rabot! (108)
(104)
(504)
Sands (26d)
Sonar (61d)
Credit (258)
CMCD-KL
Initial Scale
1.0
1.0
2.0
2.0
40.0
15.0
1.0
0.1
10.0
0.1
1.0
Max Dillusion
10.0
1.0
0.03
0.03
10.0
10.0
1.0
1.0
1.0
1.0
10.0
Architecture
PN
PGN
PGN
PGN
PN
PGN
PN
PN
PGN
PN
PGN
Model LR
0.001
0.0001
0.001
0.001
0.0001
0.001
0.001
0.001
0.001
0.001
0.0001
Annealing Schedule LR
0.01
0.001
0.001
0.001
0.01
0.001
0.01
0.001
0.01
0.01
0.01
CMCD-LV
1.0
1.0
Maximum diffusson
1.0
10.0
0.03
0.03
20.0
1.0
1.0
1.0
0.1
1.0
10.0
Architecture
PN
PN
PGN
PN
PGN
Model LR
0.001
0.0001
0.0001
0.001
0.0001
0.001
0
0.001
0.001
0.001
0.0001
Annealing schedule T.R
0.01
0.01
0.01
0.001
0.001
0.01
0.01
0.001
SCLD
Model LR Annealing schedule L.R
0.001
0.0001
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0.01
0.01
0.01
0.001
0.001
0.001
0.01
0.01
0.01
0.001
0.001
CRAFT
Number of steps
128
128
128
128
LR
0.001
0.00001
0.001
0.001
0.00001
0.001
0.0001
0.0001
0.0001
0.001
0.001
initial
1.0
1.0
Experimental details. Here, we provide additional details on the experiments in the main part of the paper.
Improved convergence (Figure 3). All experiments were performed on a single Nvidia RTX4090 GPU using the same settings the main experiments.
28

---

Published as a conference paper at ICLR 2025
Table 6: Hyperparameter choices of DDS, PIS, and SMC for the tasks not present in Blessing al. (2024).
Robot1
Robot4
MW54
SMC
Initial scale
2.0 [0.001, 0.01]
2.0
1.0 [0.01, 0.001]
HMC step sizes
[0.01. 0.001]
DDS
Initial scale
2.0
2.0
0.1
Maximum diffusion
0.3
0.3
10.0
LR
0.001
0.001
0.00001
PIS
Maximum diffusion
0.3
0.3
10.0
LR
0.00001
0.00001
0.00001
Varying the number SMC steps (Figure 4). For this study, we train for 8000 gradient steps in all instances and vary the number of subtrajectories training and evaluation time. Apart from that, we use the same hyperparameters and procedures as the main experiments. In particular, the total number of annealing steps is fixed 128.
A.6 ADDITIONAL EXPERIMENTS
In this section, we present additional experiments.
A.6.1 ABLATION STUDIES OF SCLD
In Figure 6, we study the effect of removing various parts of SCLD on several tasks. We inves- tigate the use the buffer, resampling, and MCMC steps. For this experiment, other design choices are kept the same as in the main experiments. In particular, the reported results for the full SCLD algorithm here coincide with those the main experiments up to variation due seeds. On the other hand, the "No Peatter,eeammlinggMMMM. subtrajectories.
Algorithm corresponds to CMCD-LV with
In all studied cases except the Seeds task, the addition of each component (MCMC, resampling, and buffer) improves performance (we use a logarithmic scale for clarity on the Robot task). In the case of the Seeds task, the performances of all choices effectively the same (note the small range of the y-axis). In summary, this study shows that none of our components are redundant.
A.6.2 REMOVING MCMC COMPONENTS
Here, we investigate the effect of not using MCMC steps during training. This is an interesting question because, unlike SMC methods with deterministic transitions like CRAFT, where MCMC steps are needed to remove the partiele degeneracy caused by resampling steps, our stochastic tran- sitions do this automatically. As such, is possible to remove MCMC steps from the SCLD training procedure, and we investigate the effect of doing here, as it offers potentially accelerated training.
Table 7: ELBOs attained by SCLD when removing MCMC steps during training and evaluation.
ELBO (t)
Seeds (26d)
Sonar (61d)
Credit (25d)
Brownian (32d)
LGCP (1600d)
SCLD
-73.45
-108.17
-504.4610.00
1.00
486.7710.70
SCLD-NOMCMC
-73.4810.0
109.3991..0
-504.72.o..
1.
415.83119.88
Table 8: Sinkhorn distances attained by SCLD when removing MCMC steps during training evaluation.
Sinkhorn ())
Funnel (10d)
MW54 (5d)
RobotI (10d)
Robot4 (10d)
GMM40 (50d)
MoS (50d)
134.23+a.50
0.44+0.00
0.31
0.40:a.01
3787. 7787.73+20.75
656.10+sa.97
SCLD-NOMCMC
147.38
0.44
0.31
0.41a0.1
1929.52-ma..r
1252.87=1es
Using the same experimental setting as the main experiments, we compare the effect of omitting SMC steps during training and evaluation in Tables 7 and 8. Unsurprisingly, removing MCMC steps has an adverse effect on performance. However, in many cases, the difference is not too big. In partieular, tasks where smaller number of 4 subtrajectories have been used (Robotl, Robot4,
29

---

Published as a conference paper at ICLR 2025
Robo04
Sinkhor Distaines
10"
10"
No (Butfor (e.oo.Resammiing..MM.)
No (Buffer, Rastarins
No Butter
SCLD
MoS
Sinkhorn Distanoe
1200
1000
800
600
No (Bu [B..hr.Rssamminngg.....
No (Buffer, Rasfarr/
No Buffer
SCLD
Seeds
-73.435
-73.445
-73.450
No (Buffer hww.fe/eeesmmpigggMMM. No (Buffer, Restfer.essmmpiing)
No Buffer
SCLD
Sonar
ELBO
108.4
No ReddleerResammiigggMCCGC
No (Bufler Resafpin)s
No Buffer
SCLD
Figure 6: Ablation study of the different components of SCLD on four tasks. sequentially add MCMC steps, resampling, and a prioritized relay buffer to LV-CMCD with subtrajectories (corresponding to the "No SoufferReeampling.MMM.
method) to arrive at our proposed SCLD method. We observe that most tasks, each of these components improves performance.
GMM40, MW54). the effect was negligible. as MCMC steps did not feature prominently in the training process in the first place. On the other tasks, where 128 SMC steps have been employed, the impact on performance was larger. However, the performance was still competitive with other approaches, noting that we did not increase the number gradient steps. In all, using SCLD without MCMC steps is shown to be viable possibility. It is also plausible that increased noise levels could help compensate for the lack additional randomness.
A.6.3 TIMINGS
In Table 9, we report the timings on each task for each the methods in the main table with regards time taken per gradient step (except SMC, which does not require training), using the same hyperparameters as for the main experiments. We worked in the JAX framework and used jitting, discarding the first iteration (Bradbury et al., 2018). We average across seeds on a single Nvidia RTX4090 GPU for 500 iterations. Dynamical memory allocation via was required for CMCD-KL. on GMM40 fit within the memory limit, resulting in slower runtimes.
XLA PYTHON CLIENT ALLOCATOR=plate
Table 9: Average time per gradient step for all considered methods and tasks.
Time (s)
Brownian
Credit
LGCP
Seeds
Sonar
Funnel
GMM40
MW54
Robot1
Robot4
MoS
CMCD-KL
0.21
0.14
0.39
0.12
0.13
0.14
0.58
0.10
0.24
0.24
0.20
CMCD-LV
0.34
1.41
0.42
0.13
0.18
0.10
0.14
0.09
0.11
0.12
0.11
0.15
1.48
0.07
0.12
0.08
CRAFT
0.004
0.89
0.04
0.02
0.01
0.01
0.05
0.04
DDS
0.04
0.03
0.11
0.03
0.03
0.03
0.04
0.03
0.04
0.04
0.03
PIS
0.04
0.03
0.10
0.03
0.03
0.03
0.04
0.03
0.04
0.04
0.03
30

---

Published as a conference paper at ICLR 2025
The dimension of the target, the number of SMC operations, well as the difficulty of evaluating the target all significantly influence the computation time. It may seem strange that SCLD, with the added complexity of SMC steps, was generally faster than the CMCD variants. This can be attributed to two points. First, SCLD detaches the trajectory due to the use of the off-policy log log-variance loss, unlike CMCD-KL,, which results in simplified computation graph, saving both time and memory. We refer to Richter & Berner (2024) for a full discussion on using detaching in the log-variance loss. Due to our use of the subtrajectory-based LV loss, the gradients for each subtrajectory can be computed independently and in parallel, improving speed over CMCD-LV. Please note, however, that timings are highly dependent on implementational details.
A.6.4 ESTIMATIONS OF THE NORMALIZING CONSTANT
When the true normalizing constant for a density is known, another benchmark often used to evaluate sampler is study how accurately it estimate Z log Z. It is, however, known that for multimodal tasks, methods that achieve good log Z estimates often do so at the expense of mode collapse. Conversely, methods that avoid mode collapse sometimes yield poor log Z esti- mates (Blessing et al., 2024). Indeed, applying (tuned) CRAFT to the GMM40 (50d) task achieves an log Z estimate of 3.663 (the true value is log 0) and one of the better-performing methods for the task. While this may sound impressive, it is realized that -3.63 & log 40 corresponds to sampling perfectly from exactly of the 40 modes achieves relatively good estimates of the true log Z, it performs poorly as a sampler. Likewise, when SCLD optimized for Sinkhorn distances, often has worse estimation errors but achieves significantly better sample quality.
evidenced by Figure 7). Thus while CRAFT
Figure 7: CRAFT only samples from one mode of GMM40 (50d).
Acknowledging this trade-off between log Z estimation and mode collapse, we present two sets of results for CRAFT, CMCD- -LL CMCD-LV, and SCLD corresponding to the log Z estimation error when methods are optimized for Sinkhorn distances (named CRAFT-SD, SCLD-SD, CMCD- KL-SD, CMCD-LV-SD) and when methods are optimized for log Z estimation (named correspond- ingly). In Table 10, we present errors of normalizing constant estimations on a selection of tasks where true log Z values are available, averaged over 4 seeds and using the same evaluation protocol as the main experiments. For this experiment, results for DDS and PIS are also taken from Blessing et al. (2024) when available.
Table 10: log Z estimations for different tasks.
A Alog (L)
Funnel (10d)
MW54 (5d)
GMM40 (2d)
GMM40 (50d)
MoS (50d)
SMC
0.19
1.45-1.58
0.08 0..03
761.93+21.56
3.88
PIS
0.92+0.60
0.36=
0.27=
7.12+0.63
12.25
DDS
0.19
3.34= 0.08
0.01=0.00
1.74
1.95
CRAFT-SD
0.10#p.02
0.16
0.0222..2
1295.255144.711
(). 0.75220.1
CRAFT-logZ
0.10#p.02
0.16
0.02=o.01
3.63
0.75
CMCD-KL-SD
0.04+0.01
1.65=0.10
0)01=0.00
3.53+0.12
2. 2.72+0.45
CMCD-KL-llogZ
0.04+0.01
1.65=a.10
0.01=0.00
3.53+0.12
2. 2.19 191
CMCD-LV-SD
0.24++.10
0.01+
0.01=0.00
1.45++0.8
3. 3.04+0..11
CMCD-LV-logZ
0.18
0.01+
0.00+0.00
1.45++.35
3.04+0.41
SCLD-SD
0.09
0.14=o.ua
0.02=0.03
7.10+4.00
0.05+...3
SCLD-logZ
0.09
0.01
0.02-0.01
0.77
0.05+10.03
31

---

Published as a conference paper at ICLR 2025
We found that using SMC evaluation time (with the same configuration during training) consis- tently improved log estimate quality for SCLD and consequently used it for all tasks. We maintain the same subtrajectory settings we did for the main experiments. SCLD significantly outperforms all other methods on the GMM40 (50d) and MoS tasks and is best or a close second on the other tasks. This illustrates that method can also be adjusted target better log Z estimates.
A.6.5 THE LEARNED ANNEALING SCHEDULE
For CMCD-KL, CMCD-LV, and SCLD, we found that using learned annealing schedule as in (35) is crucial to obtaining good results. We illustrate this in Figure 8 with a case study SCLD, visualizing the linearly interpolated annealing schedule, i.c., (20) with 3(t) = t/T, and the learned annealing schedule in (35) for the 2-dimensional GMM40 task.
Learned Annealing
3
8
t=0
t 0.52T
0.25T
t=0.775T
0.5T
Linear Annealing
Figure 8: We compare the uniform annealing schedule with the annealing schedule learned by SCLD for S S T/2. SCLD able to learn more gradual annealing schedule, which potentially allows transitions between adjacent densitics to learned more casily.
A.6.6 MEAN FIELD PRIOR FOR SCLD
While we opt for prior of the form (0,, o²1) for SCLD in our main experiments, an alternative approach is to initialize it using diagonal Gaussian trained using Mean Field Variational Inference (MFVI) (Bishop, 2006). We study this design choice experimentally here.
We use 50000 iterations of MFVI with batch size 2000 and constant learning rate 10-3, initializing with (0, I). We retain the same experimental setup and hyperparameter settings as for the main experiments, except for the max diffusion coefficient, where we divide the values from the main experiments by 10. This is because MFVI is mode seeking, and so aims to cover a high probability region of the target distribution tightly, leading to a prior with smaller support. We compare the attained ELBOs in Table II and also report results for SCLD-MFVI at initialization (i.e., without training the control), termed "NoTrain".
Table Performance of SCLD when fitting the diagonal the prior covariance matrix using MFVI at initial- ization ("NoTrain") and after training ("SCLD-MFVI").
ELBOs (f)
Brownian
Credit
LGCP
Seeds
Sonar
NoTrain
1.07
-513.70
500.42+0.37
-73.4840.0b
-114.89
1.35
SCLD
1.00
-60..4620.09
486.77.0.70
-73.45+0.01
-108.17+0.2
SCLD-MFVI
1.14
-504.5940.15
500.56+0.12
-73.44+0.01
-108.93
Impressively, SCLD often achieves cear-state-of-the-ar results even without training when initial- ized with MFVI, such as on the LGCP task. We can attribute this SCLDD being initialized as an SMC sampler with Unadjusted Langevin Annealing (ULA) transition kernels well as MCMC steps, which, in conjunction with the mode-seeking behavior of MFVI, leads to high ELBO values. SCLD-MFVI attains competitive performances on all tasks. Given that performed no re-tuning on SCLD-MFVI it is probable that with more careful setting and hyperparameter choices, even higher ELBOs could be attained.
However, using MFVI-fitted priors in practice often carries serious drawbacks. In line with the experiments of Blessing al. (2024), we found that using MFVI priors leads to mode collapse
32

---

Published as a conference paper at ICLR 2025
(due the mode-seekingg nature of MFVI training restricting the sampling to subset of the target modes), and thus potentially poor sample quality. We illustrate this in Figure on the GMM40 (50d) target, where we use the same hyperparameters in the main experiment except for the prior.
Figure 9: The samples drawn by SCLD when an MFVI-fitted prior is used. MFVI obtains a prior density that covers exactly one mode of the GMM40 distribution. As such, SCLD is unable to discover the other modes and experiences complete mode collapse. This is in contrast to Figure 2 where SCLD samples are visually indistinguishable from the target density.
A.6.7 COMPARISON TO PDDS
In this section, we empirically compare the PDDS and SCLD methods. We employ the exact ex- perimental methodology of Phillips et al. (2024). In particular, we train for 20000 gradient steps, refreshing the model every 500 steps. We employed 50000 gradient steps to train the mean field prior. We note that this corresponds to significantly higher iteration budget than was allocated to SCLD. In line with the findings of Phillips et al. (2024), we found that sweeping over the prior seale opposed using variational approximation significantly degraded performance (and indeed on several tasks, such as Robot and GMM40 could not train at all). One reason for the degraded performance might be that PPDS is unable to further optimize the prior during training (as is done in SCLD). We thus opt to use variational approximations (by mean field Gaussians) to initialize the prior for all tasks. Benchmarking was done exactly as in the main experiments, and we analyzed the performance of PDDS with and without MCMC steps.
all tasks
For all tasks present the benchmark of Phillips et al. (2024) (including the Gaussian mixture tasks), we used the pre-tuned MCMC step sizes. For the other tasks, we chose linearly interpolated step size schedule from
to T where step sizes times 0 and T are taken from the grid [0.1, 0.3, 3, 10J since the method for tuning MCMC step sizes was not specified. We select the best parameters directly based on the target metric and present the results in Tables 12 and 13.
Table 12: Comparison of SCLD against PDDS (Phillips al., 2024) in terms ELBOS.
ELBOs (f)
Brownian
Credit
LGCP
Seeds
Sonar
PDDS
1.
-502.80
499.35
-73.4820.21
-108.61100
PDDS-MCMC
1.04
-502.9020.28
499.83
-73.4720.19
-108.67
SCLD (ours)
1.00
-504.462..00
486.7 780.7770070
-73.4520.01
-108.17+00
SCLD-MFVI (ours)
1.1440.05
-504.5940.15
500.562o.12
-73.4420.01
-108.93.20
Table 13: Comparison of SCLD against PDDS (Phillips et al., 2024)
terms of Sinkhom distances.
Sinkhorn ({)
Funnel
GMM40
MW54
Robotl
Robot4
MoS
PDDS
+
42157.92+3468
1.
3,36+
3.09+a.16
3119.83+9s.co
PDDS-MCMC
+51.02+25.00
42157.92+346.21
1.07 20.25
3.35
3.08
3108.75: +
SCLD (ours)
134.23+s.29
3788.733+240..7
0.44+0.06
0.31+0.04
0.40+a.01
656.10+s8.07
PDDS attains comparable ELBOs to SCLD on the Bayesian statistics tasks. This is due to both meth- ods being initialized as SMC samplers with prior obtained by the same variational approximation
33

---

Published as a conference paper at ICLR 2025
(for SCLD-MFVI), We also observed, in line with the findings of Phillips al. (2024) and similar to Appendix A.6.6, that often relatively little training is required to achieve optimal performance, so the gap in performance between the initial, untrained SMC scheme and the trained sampler is small.
However, PDDS consistently presents significantly worse Sinkhorn distances (on all tasks where this is available) than SCLD. This is due to the reliance of PDDS using MFVI prior, which, as discussed in Appendix A.6.6, is prone to mode collapse. On the other hand, SCLD is able to operate stably without relying on using the MFVI prior, avoiding mode collapse.
A.6.8 COMPARISON WITH ADVANCED SMC SCHEMES
In the section, we compare SCLD against two advanced SMC schemes implemented in the frame- work by Cabezas et al. (2024). We consider adaptive tempered SMC, which utilizes the constant- ESS method for choosing the annealing schedule as seen in Buchholz et al. (2021). We term this method SMC-ESS. In line with SCLD, utilize a single HMC step for the SMC kernel with 10 leapfrog integration steps, and apply the same tuning procedure for HMC step size as we did for our own SMC method. We additionally sweep over the ESS threshold Due to the large search grid, we run 10 seeds per task to mitigate outliers. Unlike SCLD, which uses multinomial resampling (for fair comparison (see, e.g.. Chopin et al. (2020, Chapter 9)) for SMC-ESS, which we found led to best performance. We consider another method from Buchholz et al. (2021), utilizing the full-covariance tuning ap- proach for Independent Rosenbluth Metropolis- Sirropoli--Hastinee Hastings (IRMH) proposals (on top of using adaptive tempered SMC). We use 100 MCMC steps per step and term this method SMC-FC.
(0.3, 0.5, 0. 75, 0.9, 0.95, 0.99).
other baselines). use systematic resampling
We report results in Tables 2 and 3, using the same evaluation protocol (in particular, using 2000 particles). For reference, we also compare all SMC methods with SCLD Tables 14 and 15.
Table 14: Comparison of SCLD against advanced SMC methods (Buchholz et al., 2021) in terms of ELBOs.
ELBOs (t)
Brownian
Credit
LGCP
Seeds
Sonar
SMC
-2.21
-589.82
385.75.7.0
-74.63=0.14
-111.50=0.0 969
SMC-ESS
0.49
-505.57 10.18
497.85+0.11
-7..07
-109.10#0.1.
SMC-FC
-0.9
50..30+0.02
-88.10422
-74.07=0.02
-108.93+0
SCLD (ours)
1.00
-504.46+o0
486.77+0.m
-73.45+0.01
-108.17+0.25
Table 15: Comparison SCLD against advanced SMC methods (Buchholz et al., 2021) terms of Sinkhorn distances.
Sinkhorn (1)
Funnel
GMM40
MW54
Robot1
Robot4
MoS
SMC
14
16370.342197
20.71 33
24.02_1.06
24.0820.6
2997.2882184..4
SMC-ESS
117.48
24240.68 50.62
1.11+0.15
1.82
2.11
1477,04+ 133.60
SMC-FC
211.43_aa 330000000
39018. 1109.022
2.03
0.37 13772.00 09
1.23
3200. 102.00a..... 35
SCLD (ours)
134.23
3787.73
0.4420.06
0.31
0.40zo.01
656.10
The full-covariance tuning and the ESS-based scheme for selecting the annealing schedule sig- nificantly outperform our baseline implementation of SMC at the expense of longer and variable (possibly unbounded) sampling times. Nevertheless, all considered SMC methods are superseded by SCLD in performance on all but two tasks. While SCLD uses a relatively simple version of our baselines, our framework enables the usage of more advanced
SMC for fair comparisons techniques, such as those used for SMC-ESS and SMC-FC. Thus, we expect that the performance of SCLD can be even further improved.
A.6.9 CONVERGENCE OF DIFFERENT METHODS BY ITERATION COUNT
In Figure 10, we visualize the same data as Section 3.1 but plotting by the number of elapsed gradient steps. In this perspective, the same conclusions hold that SCLD exhibits superior conver- gence properties, attaining the best ELBOs on each task for all numbers of gradient steps. Note that CRAFT was competitive the Credit task in this perspective.
34

---

Published as a conference paper at ICLR 2025
Credit
1.0
Brownian
Seeds
-106
Sonar
-510
-73.5
0.5
-110
520
-74.0
CMCD-KL
-630
CMICDALV
0.55
-14.5
114
Long Run
CMGD (Best)
-6500
-1.0
1030
-75.0g
G.radiinnt
Gradiont Stupa
Gradiintt Slaps
Figure 10: The same experiments as in Figure plotted instead by iterations.
A.6.10 KL-BASED TRAINING SCLD
We compare KI. and LV-based training of the SCLD algorithm, using the family of Funnel distri- butions with E {10, 20, 30, 40, 50} as case study. We train SCLD using KI. and LV losses with 4 and 128 subtrajectories as described in Section 2.3 for 3000 gradient steps using the same hyper- parameters and settings (including learning the annealing schedule and prior) as in the for the main experiments. In Figure 11, we visualize the ELBOs attained (using the same settings during evaluation as for training) alongside CMCD-KL and CMCD-LV.
case
ELBOs Across Varying Funnel Dimensions SCLD Variants
ELBO
CMCD-k
CMCD-Iv
SCLD-KL-4-Subiraa
SCLD-LV-4-Subtra)
SCLD-KL-128-Subrra
-10
SCLD-LV-128-Subbt
10
26
26
30
36
40
00
Funnel Dimension
Figure 11: ELBOs across Varying Funnel Dimensions for different SCLD-variants
All methods except SCLD with LV loss and 128 subtrajectorics experience some form of perfor- mance degradation as dimensions scale. For the LV loss, adding subtrajectories reduces the amount of performance degradation. This may be due to SMC steps counteracting increased dimensionality by focusing computation high-density regions. For the KL loss, however, increasing the number of subtrajectories resulted in worse performance, especially as the dimension increased. Indeed, SCLD-KL with 128 subtrajectories scales the most poorly of the methods tried d increases. As discussed in Section 2.3, this may be due to the use of importance sampling to estimate the loss function. Indeed, a of importance weights required for each subtrajectory to estimate the loss, and thus, using more subtrajectories demands greater reliance on importance sampling. As the variance of importance sampling can increase significantly with dimension, this may account for the decreased performance of KL-based subtrajectory losses. In summary, this supports the hypothe- sis that losses avoiding importance sampling, such as the log-variance loss, are more suited to the training of SCLD on higher-dimension tasks.
35


Reviews:
[{"review_id": "pe9RpYD00i", "reviewer": "Reviewer_6QXs", "summary": "Sequential Monte Carlo and diffusion-based samplers both intend to simulate samples from an un-normalized target distribution via a continuously-evolving sequence of distributions, yet their drawbacks are markedly different: SMC requires no learning but lacks flexibility, which might lead to slow convergence; diffusion samplers are potentially more adaptive but it requires training. \n\nThis paper proposes sequential controlled Langevin diffusion (SCLD), a combination of the above two methods that uses diffusion-based samplers to adaptive transition between different stages of SMC. As a result from Girsanov's theorem, we can use the path measure likelihood ratio to construct a sequence of importance sampling weights, which can be used as part of resampling for the SMC procedure. The paper illustrates the results with theoretical and empirical findings that showcase SCLD as a strong alternative to SMC-based samplers.", "questions": "- Equation 12: the right hand side looks like the log-density ratio instead of the density ratio itself -- is this correct? \n- Figure 5 seems to suggest that the more SMC steps, the better the performance, even though the paper noted potential downsides of excessive resampling (line 521). Have you tried applying an unusually large number of resampling steps, and seen if the results would degrade overall?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and I find the results to be of significant interest to the Bayesian statistics community -- SMC has become an important statistical tools for simulating samples from intractable densities, and I think merging SMC with diffusion gives useful insights in building more flexible SMC samplers. \n\nWhile some of the paper's contributions seem incremental, the resulting algorithm seems more useful in practice when compared to its predecessor work CMCD, mainly thanks to its use of SMC resampling techniques for more accurate sampling.", "weaknesses": "I have a positive assessment of this paper, and I enjoyed reading it overall. \n\nThe overall contribution of this paper seems somewhat limited. It is somewhat difficult to pinpoint exactly what is new from this paper, although I think the overall readability and SCLD's potential usefulness make up for this weakness.\n\nI think the main weakness of this paper is that the combination of continuous and discrete time seems somewhat arbitrary. It seems to me that given enough training iterations, SCLD or CMCD will be able to learn the optimal control function eventually. In that case there would be no need for resampling and MCMC steps as a well-calibrated SDE solver would be able to simulate a sample from 0 to T directly. Therefore, the main appeal of SCLD is that it is able to correct small imperfections of the control function via resampling (as evidenced empirically by SCLD requiring fewer iterations to achieve good performance). However, when to resample depends crucially on where the discrete $t_n$s are placed. This framework has some flexibility in this regard in that it learns a smooth transition function, but the question of how many discrete steps to take remains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VLff4K4knu", "reviewer": "Reviewer_Kjto", "summary": "This paper considered the task of sampling from unnormalized densities and proposed a framework that combines the sequential Monte Carlo (SMC) method with diffusion-based samplers. The main idea behind such a framework is to view both SMC and diffusion-based samplers under the continuum time limit and consider measures on path spaces. This yields the Sequential Controlled Langevin Diffusion (SCLD) method, which has achieved competitive performance on multiple real-world and synthetic examples.", "questions": "The reviewer's main concern is that a thorough comparison between the SCLD method and the method proposed in [4], whose training stage is based on score matching, needs to be included. Could the authors provide some intuition on how the training losses based on diffusion bridges compare to the score-matching loss when combined with SMC? Empirically, it would be necessary to add the algorithm in [4] as a baseline and compare it with SCLD.\n\nReferences:\n\n[1] Shi, Y., De Bortoli, V., Campbell, A. and Doucet, A., 2024. Diffusion SchrÃ¶dinger bridge matching. Advances in Neural Information Processing Systems, 36.\n\n[2] De Bortoli, V., Thornton, J., Heng, J. and Doucet, A., 2021. Diffusion schrÃ¶dinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34, pp.17695-17709.\n\n[3] Domingo-Enrich, C., Han, J., Amos, B., Bruna, J. and Chen, R.T., 2023. Stochastic optimal control matching. arXiv preprint arXiv:2312.02027.\n\n[4] Wu, L., Trippe, B., Naesseth, C., Blei, D. and Cunningham, J.P., 2024. Practical and asymptotically exact conditional sampling in diffusion models. Advances in Neural Information Processing Systems, 36.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is presented clearly and in detail, making it easy for readers to follow. The proposed framework also allows the authors to design suitable loss functions compatible with off-policy training, which has greatly reduced the use of the training budget in practice. Extensive numerical experiments are provided to validate the effectiveness of the SCLD method.", "weaknesses": "1. As the SCLD algorithm relies on the diffusion bridge formulation, the reviewer thinks that it might be necessary to include a short literature review on related work [1,2,3] combining diffusion bridges/stochastic optimal control with generative models for the sake of completeness. However, this is missing in the current version of the manuscript. \n\n2. Though the framework based on path measures proposed in this paper leads to effective sampling methods, the idea of combining SMC with diffusion-based samplers isn't new. In fact, this has been explored in earlier work [4] for the posterior sampling problem, which the authors should have cited and discussed. Specific concerns about the comparison between the SCLD method and [4] will be further elaborated in the \"Questions\" section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nh7nCWxVHs", "reviewer": "Reviewer_TwR8", "summary": "This paper proposes Sequential Controlled Langevin Diffusion (SCLD), a novel sampling framework that combines the strengths of Sequential Monte Carlo (SMC) and diffusion-based sampling methods. The goal is to leverage the resampling efficiency of SMC while taking advantage of the adaptive flexibility of diffusion models. The authors further introduce a log-variance divergence loss function to address the variance issues often encountered with KL divergence estimators, aiming for more stable optimization.", "questions": "- Could the authors elaborate on the finite-time convergence analysis of SCLD mentioned in Table 1? Unfortunately, I did not find a related analysis in the submission.\n- The method claims to use a continuous-time setting inspired by Langevin diffusion, but the implementation appears to discretize time with fixed intervals (section 2.4). Could the authors clarify how the continuous-time formulation benefits the analysis or experimental results? How does it compare to related methods such as CMCD from [1]?  \n\n[1] Transport meets Variational Inference: Controlled Monte Carlo Diffusions. ICLR 2024", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The integration of SMC and diffusion-based sampling represents a novel and promising approach to tackling challenges in high-dimensional sampling.\n- The paper is generally well-structured, with clear segmentation between theory, algorithmic details, and experimental evaluation.\n- The proposed log-variance divergence loss function could have meaningful applications in high-dimensional probabilistic modeling, particularly for tasks prone to mode collapse and instability.", "weaknesses": "While the proposed method is conceptually reasonable, the writing is quite sloppy. It is hard for me to understand the contribution and advantages of the proposed SCLD. \n- The authors claim that the KL divergence estimator suffers from exponential error growth (Proposition 2.4) and the authors instead use log-variance divergence (Equation (18)). However, there is no formal proof or quantitative scaling analysis of this claim using the log-variance divergence.\n- The use of \"off-policy training\" seems inappropriate and underexplained. This term is typically associated with reinforcement learning rather than Langevin dynamics and diffusion models. \n- Section 1.1 is loosely connected to the content presented in Table 1. For example, the Particle Denoising Diffusion Sampler (PDDS) is mentioned in the appendix without proper discussion in Section 1.1. The differences between existing methods and SCLD are not sufficiently emphasized. Furthermore, the sentence \"limiting them to normalizing flows, as well as MCMC steps to keep sample diversity after resampling\" (lines 125–126) is unclear and requires revision.\n- The log-variance divergence in Equation (18) appears to be a critical component of SCLD, but it is missing from the main Algorithm 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NuAgodNif3", "reviewer": "Reviewer_ZuHY", "summary": "This paper proposes a new algorithm for sampling from unnormalized distributions. The core idea is to combine the capacity of SMC to focus on relevant regions of the space with the ability of diffusion based samplers to adapt to the geometry of the target distribution by learning a control term $u$ of the reverse SDE.\n\n\nThe control term $u$ then parametrizes a divergence between the path space measure of the forward process and the measure of the backward process.\nBy minimizing this divergence the algorithm ensures that the two processes become aligned.\n\n\nTo learn the optimal control $u$ the optimization objective is split on N intervals $[t_n, t_{n+1}]$ over which the divergence can be estimated by running the SMC procedure.\nThe learned control $u$ is then used to guide the proposals in the SMC sampler.\n\nThe method is benchmarked against recent approaches on standard benchmarks.", "questions": "- Are there clear examples where the more flexible proposal is worth the added need for training over running a slightly more sophisticated SMC sampler (eg [1], [3])?\n\n- Are there more complex and less toy examples where SCLD would be applicable ?\n\n-  How sensitive is the procedure wrt the chosen time discretization $[t_n, t_{n+1}]$?\n\n- When sampling, is it possible for SCLD to maintain a principled time discretization guidance similar to traditional SMC use of constant ESS? Does the flexibility in proposal distribution come at the cost of losing these methods?  If so, how does this trade-off impact the overall performance and reliability of the sampler?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper makes a good job at identifying the strengths and weaknesses of SMC samplers and of diffusion based samplers and how they can be combined together. The main concepts needed to understand the method are well introduced and the paper does a good job at explaining them.", "weaknesses": "A significant portion of the paper focuses on presenting the foundational concepts and prior research upon which SCLD is constructed. While this provides necessary context, very little space is used for a detailed discussion of the novel aspects of the sampler itself. As a result, the reader struggles to properly get the contributions.\n\nA comparisons with more traditional SMC samplers to justify the need for the control term and the training procedure would be a nice thing to have. For example with the HMC kernel adaptive procedure of [1] or the adaptive tempered version of [3] (both already implemented in [2]).\n\nIn SMC, the choice of the time discretization is usually motivated by the need to have a constant effective sample size (ESS) per step. \nThis is quite critical for good sampler performance and principled ways like [3] are available to achieve this. When moving to the continuous time/path space measure setting of SCLD it seems to me that we loose these principled ways to choose the time discretization. It is worth comparing if the flexibility gained in the proposal outweighs the loss of this machinery. \nOr if there is a principled way to choose the time discretization for the sampling process of SCLD.\n\n\n[1] - Buchholz, A., Chopin, N. and Jacob, P.E., 2021. Adaptive tuning of hamiltonian monte carlo within sequential monte carlo. Bayesian Analysis  \n[2] - https://github.com/blackjax-devs/blackjax   \n[3] - Jasra A, Stephens DA, Doucet A, Tsagaris T,  2011. Inference for Lévy‐driven stochastic volatility models via adaptive sequential Monte Carlo. Scandinavian Journal of Statistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)