Solving Linear-Gaussian Bayesian Inverse Problems with Decoupled Diffusion Sequential Monte Carlo
Authors: Filip Ekström Kelvinius
Zheng Zhao
Fredrik Lindsten
Venue: ICML 2025 poster (2025)
Keywords: diffusion models
sequential monte carlo
posterior sampling
Primary area: probabilistic_methods->monte_carlo_and_sampling_methods
TL;DR: Sequential Monte Carlo method for solving linear-Gaussian inverse problems

Abstract: A recent line of research has exploited pre-trained generative diffusion models as priors for solving Bayesian inverse problems. We contribute to this research direction by designing a sequential Monte Carlo method for linear-Gaussian inverse problems which builds on ``decoupled diffusion", where the generative process is designed such that larger updates to the sample are possible. The method is asymptotically exact and we demonstrate the effectiveness of our Decoupled Diffusion Sequential Monte Carlo (DDSMC) algorithm on both synthetic as well as protein and image data. Further, we demonstrate how the approach can be extended to discrete data.

Solving Laussinn Bayesian Inverse Problems with Decoupled Diffusion Sequential Monte Carlo
Filip Ekström Kelvinius'
Zheng Zhao Fredrik Lindsten
Abstract
distribution pe (xly) p(y[x))pe(x becomes an appealing alternative. This approach is flexible since many different likelihoods can be used with the same diffusion model prior, without requiring retraining or access to paired training data. Previous methods for posterior sampling with diffusion pri- ors, while providing impressive results on tasks like image reconstruction (Kawar et al., 2022; Chung et al., 2023; Song et al., 2023), often rely on approximations and fail per- form poorly on simple tasks (Cardoso et al., 2024, and our Section 5.1), making it uncertain to what extent they can solve Bayesian inference problems in general.
A recent line of research has exploited pre-trained generative diffusion models as priors for solv- ing Bayesian inverse problems. We contribute to this research direction by designing a sequential Monte Carlo method for Greea-Ga inverse problems which builds on "decoupled diffusion", where the generative process is designed such that larger updates to the sample are possible. The method is asymptotically exact and we demon- strate the effectiveness of our Decoupled Diffu- sion Sequential Monte Carlo (DDSMC) algorithm on both synthetic as well as protein and image data. Further, we demonstrate how the approach can be extended to discrete data.
Sequential Monte Carlo (SMC) (Del Moral, 2004; Naesseth et al., 2019) is a well-established method for Bayesian in- ference, and its use of sequences of distributions makes a natural choice combine with diffusion priors. It also of- fers asymptotical guarantees, and the combination of SMC and diffusion models has recently seen spark of interest (Trippe et al., 2023; Dou & Song, 2023; Wu et al., 2023; Car- doso et al., 2024). Moreover, the design of an efficient SMC algorithm involves a high degree of flexibility while guaran- teeing asymptotic exactness, which makes an interesting framework for continued exploration.
1. Introduction
Generative diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021b) have sparked an enormous interest from the research community, and shown impres- sive results on a wide variety of modeling task, ranging from image synthesis (Dhariwal & Nichol, 2021; Rombach et al., 2022; Saharia et al., 2022) and audio generation (Chen et al., 2020; Kong et al., 2021) to molecule and protein gen- eration (Hoogeboom al., 2022; Xu al., 2022; Corso et al., 2023). A diffiusion model consists of neural network which implicitly, through generative procedure, defines an approximation of the data distribution, (x()). While methods (e.g. Ho & Salimans, 2022) where the model is explicitly trained to define a conditional distribution po (x y) exist, this conditional training is not always possible. Al- ternatively, domain-specific likelihood p(y xx) might be known, and in this case, using (x) a prior in a Bayesian inverse problem setting, i.e., sampling from the posterior
As previous (non-SMC) works on posterior sampling has introduced different approximations, and SMC offers a flex- ible framework with asymptotic guarantees, we aim to fur- ther investigate the use of SMC for Bayesian inverse prob- lems with diffusion priors. particular, we target the case of Gaussian likelihood models. We develop method which we call Decoupled Diffusion SMC (DDSMC) uti- lizing and extending previously introduced technique for posterior sampling based on decoupled diffusion (Zhang et al., 2024). With this approach, it is possible to make larger updates to the sample during the generative procedure, and it also opens up new ways of taking the conditioning on into account the design of the SMC proposal distribution. We show how this method can effectively perform posterior sampling synthetic data, and its effectiveness on image and protein structure reconstruction tasks. We also further generalize DDSMC for discrete data. Code is available online'.
'Department of Computer and Information Science (IDA), Linköping University, Sweden. Correspondence to: Filip Ekström Kelvinius <filip.ckstromm @ liu.se>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'nttpa://githubs com/ fil www.pekttrm/ddseee

---

Decoupled Diffusion Sequential Monte Carlo
2. Background
SDE by solving deterministic ODE (initialized randomly). The solution Xg is sample from (x(x),, and this is used by Zhang et al. (2024) to construct an alternative transition kernel: first step is obtaining sample X0,++1 by solving the ODE starting at X then sample from g(x, X = *o.e+1). This can give samples from the same marginals pa po(x;) as if using the regular kernel po [x++)), ), but for unconditional sampling this is a convoluted way of performing generation as x0,++1 is already sample from the desired distribution. However, as we discuss in Section 3, this approach does provide an interesting possibility for conditional sampling.
As mentioned in the introduction, we are interested in sampling from the posterior distribution po(xly) x p(yl))())) x))p(xx), where the prior px(x) is implicitly defined by a pre-trained diffusion model and the likelihood is linear- Gaussian, i.e., p(y) |x) = ([ylAx, I. We will use the notation (-) for any distribution that is defined via the dif- fusion model, the notation p(-) for the likelihood (which is assumed to be known), and q(-) for any distribution related to the fixed forward diffusion process (see next section).
2.1. Diffusion Models
2.2. Sequential Monte Carlo
Diffusion models are based on transforming data, Xu. to Gaussian noise by a merkoiaia process of the form?
Sequential Monte Carlo (SMC; see, e.g., Naesseth et al., 2019, for an overview) is a class of methods that enable sampling from a sequence of distributions { )==0 which are known and can be evaluated up to a normalizing constant, i.e..
[(x++1|x;) = (xx,+11)
(1)
The generative process then consists in trying to reverse this process, and is parametrized as a backward Markov process
Me (xxT) = (xx.))) )/z,
(4)
T-1
po px(xo:T)= = (x() II (x (=0
(2)
where (xe:) can be evaluated pointwise. In an SMC algorithm, a set of N samples, or particles, are gener- ated in parallel, and they are weighted such that a set of (weighted) particles {(x{_r: ) are approximate draws from the target distribution Te rx(x))) For each t, an SMC algorithm consists of three steps. The resampling step sam- ples a new set of particles {(x2)
The reverse process po is fitted to approximate the reversal of the forward process, i.e., pa (x(xx(x++) z ((x((x+1)), where the latter can be expressed as
iid (.r) )) with at,..., af ~ Categorical { (I....,N] ;; { [ ] (). The second step is the proposal step, where new samples {x(-1T)) are pro- posed as x/-1 ~ (x-1(x{{1 (x)r), x/-1X = (4-1xx))) and finally a weighting step,
((xx|xe+1)= = / ((xx|xe+1, Xo)g(xo)xe+)))))))) )dxo-
While X0) is available in closed form, ((x)(x++1) is not, thereby rendering Equation (3) in- tractable. In practice, this is typically handled by re- placing the conditional (xx)(x++1) with point estimate (x0x) where fe(X++1) is a "reconstruction" of Xy. computed by a neural network. This is typically done with Tweedie's formula, where (x(x++1) = x; + [[x and s + 1) is neural network which approximates the score Vx.+1 log q(x++1) (and using the true score would result in the reconstruction being equal to the expected value). Plug- ging this approximation into Equation (3), the backward ker- nel becomes pa (x, (x-1) == q(x, xx+1, Xo = fo (xx+1)).
wi_1 x (x- (XL ( (5)
To construct an SMC algorithm, is hence necessary to determine two components: the target distributions (xx()))) (or rather, the unnormalized distributions (xx (2)), and the proposals {re-1 (x--1 YE
18_156(X411, + 1)
SMC as a General Sampler Even though SMC relies on sequence of (unnormalized) distributions, it can still be used as general sampler to sample from some "static" dis- tribution Φ(x) by introducing auxiliary variables xo:r and constructing a sequence of distributions over (Xer) As long as the marginal distribution of XD w.r.t. the final distri- bution To (Xo.T) is equal to Φ(xo). the SMC algorithm will provide a consistent approximation of ((x) (i.e., increas- ingly accurate as the number of particles N increases). The intermediate target distributions { (xxx)) )T= are then merely a means for approximating the final target (x(.T).
Probability Flow ODE Song et al. (2021b) described how diffusion models can be generalized as time-continuous stochastic differential equations (SDEs), and how sampling form a diffusion model can be viewed as solving the corre- sponding reverse-time SDE. They further derive a "proba- bility flow ordinary differential equation (PF-ODE)", which allows sampling from the same distribution Pa (x,) as the
(x(x;)
II principle, x, could also be multiplied by some time- dependent parameter, but we skip this here for simplicity.
"hhe resampling procedure design choice. In this paper we use Multinomial with probabilities ( v for simplicity.
2

---

Decoupled Diffusion Sequential Monte Carlo
3. Decoupled Diffusion SMC
and define our backward transition using this as
3.1. Target Distributions
(x1
== === 4r (x,(x, (X Xo)o (xx+1))) (9)
The sequential nature of the generative diffusion model naturally suggests that SMC can be used for the Bayesian inverse problem, by constructing sequence of target dis- tributions based on Equation (2). This approach has been explored in several recent works (Wu et al., 2023; Cardoso et al., 2024, see also Section 4). However, Zhang et al. (2024) recently proposed an alternative simulation protocol for tackling inverse problems with diffusion priors, based on a "decoupling" argument: they propose to simulate (approx- imately) from pe (xo Xx+1. y) and then push this sample forward to diffusion time t by sampling from the forward kernel g(x,) Xu) instead of ((x, Xg, X(+1). The motivation is to reduce the autocorrelation in the generative process to enable making transitions with larger updates and thus cor- rect larger, global errors. The resulting method is referred to as Decoupled Annealing Posterior Sampling (DAPS).
which allows us to smoothly transition between the DAPS (1) = 0) and standard (1) = 1) backward kernels.
Likelihood Having defined the prior as the generalized DAPS prior (²(.).)) we also need to incorporate the condi- tioning on the observation y. A natural starting point would be to choose as targets
m((x)) = (xx
(10)
as this for t O leads a distribution with marginal (x However, the likelihood p(y x,)
I p(y (xo) (xo|xx))d in Equation (10) is not tractable for > 0, and needs to be approximated. As the reconstruction fo (x++1) 1) played a central role in the prior, we can utilize this also for our likelihood. Song et al. (2023) proposed to use a Gaussian approximation oo(xo|xx (x,) == N (xolfo(x)) I). resulting in
To leverage this idea, we can realize that the SMC frame- work is in fact very general and, as discussed in Section 2.2, the sequence of target distributions can be seen as a design choice, as long as the final target No (XD:)) admits the distri- bution of interest as a marginal. Thus, it is possible to use the DAPS sampling protocol as basis for SMC. This corre- sponds to redefining the prior over trajectories, from Equa- tion (2) to (x(o))))= = Po (xx) I (X (x++1) where ([(x|(x(1))) = q(xx x fa (1).
p(y xx) ~ p(y|x,) = (x)))ff(())) (x))) + AA
(11)
as the likelihood is linear and Gaussian. From hereon will use the notation jo for any distribution derived from the Gaussian approximation Oe (xo]Xe). As for = 0 the likelihood is known, we can rely on the consistency of SMC to obtain asymptotically exact samples, even if using approximate likelihoods in the intermediate targets.
(6)
Conceptually, this corresponds to reconstructing xo con- ditionally on the current state X(+1, followed by adding noise to the reconstructed sample using the forward model. As discussed by Zhang et al. (2024), and also proven by us in Appendix A, the two transitions can still lead to the same marginal distributions for all time points t, i.e., I ( sumption that Xo (xx+1) is a sample from po (xx) and that the generative process pe indeed inverts the forward process q. The sample xo can be approximately obtained by, e.g., solving the reverse-time PF-ODE, and using this solution as reconstruction model 1x)41))
Putting it Together To summarize, we define a sequence of intermediate target distributions for SMC according to
I ) under the as-
T-1
(xxr)) = p(y (x) II P8 (x+11)
p(y(x))) Fly xx+1)
p (XelXe+ (741) (12)
where
Generalizing the DAPS Prior By rewriting the condi- tional forward kernel g Xx+1, using Bayes theorem,
p(ylx;) = -V (11A (13)
qxx Xx+1, Xo) oc (x1+1 xx) (xx((x)))
(7)
and, for = 0,
we can view the standard diffusion backward kernel Equa- tion (3) as applying the DAPS kernel, but conditioning the sample on the previous state X++1, which acts as an "obser- vation" with likelihood g(xe (x+)). We can thus generalize the kernel in Equation (3) by annealing this likelihood with the inverse temperature
p9 (xx(xx+)) = N (xxx(((6(xx111)
(14)
We view the reconstruction Lo as a design choice, and try Tweedie's 's formula and solving the PF-ODE initialized at X in the experiments section.
gn (x//xx4 Xx+1, xo) x g(x++1 (x:+) (((e)
*See Appendix D for an expression for general r7.
3

---

Decoupled Diffusion Sequential Monte Carlo
Algorithm 1 Decoupled Diffusion SMC (DDSMC). All
3.2. Proposal
operations for i = N
As the efficiency of the SMC algorithm in practice very much depends on the proposal, we will, previous works SMC (c.g.. TDS (Wu et al., 2023) and MCGDiff (Car- doso et al., 2024)), use proposal which incorporates infor- mation about the measurement y. Again we are inspired by DAPS where the Gaussian approximation po (xx|xx) (x)) plays a central role. In their method, they use this approximation as a prior over Xg. which together with the likelihood p(y XX) form (approximate) posterior
Input: Score model sg, measurement y
Output: Sample Xo
Sample x ~ p(Xr)
for t in T........ ......0 do
Predict xb, = (x()))
Compute p(y|>x) (Equation (13))
if' = T then
Set u² = p(y|>2/)))
else
Set o' ply Mlyyx))n (x]
xx xx+1: y) x p(y Xu po (Xu) Xx+1).
(15)
plyix) Jr (X]1
end if
The DAPS method is designed for general likelihoods and makes use of Langevin dynamics to sample from this ap- proximate posterior. However, in the linear-Gaussian case we can, similarly to Equation (11), obtain closed form expression as
Compute w² w //mm wo
Resample (xt, 86,1,,11" IN
Sample xi_1 ~ T2-1 1x-1(x111/x/ 11x/,) {Equation (19), or Equation (54)for general }
end for
xx (xxliz (i (), (16) with 1i
Compute w' piyix!) ) and w² w0/ Σラ=1 w
Sample X ~ Multinomial ((()moomia((()))))) د {w>) )-²
1 (x++1.y) = M b and
return Xo {Or the full set of weighted samples depending
on application)
M1+1
A A +
I,
(17a)
0g
Pi+1
1
b
AT y + fo (x(+1).
(17b)
4
pi+1
Although this is an approximation of the true posterior, it offers an interesting venue for an SMC proposal by propagat- ing sample from the posterior forward in time. Assuming r = 0 for notational brevity (see Appendix for the general expression) we can write this
Proposition 3.1. The target in Equations (12) and (13) with p = O will, under mild assumptions, give SMC algo- rithm targeting P (xoly) ply (x))% (xo) that is asymp- totically exact, i.e., asymptotically unbiased when the num- ber of particles increases.
ry
= ((xx (xo1x (xx y)xxo. (18)
This follows from general SMC theory, as the unnormal- ized target at 0 becomes "o0 (xx:T) p(y xx) (x(:T), meaning the normalized target mo (xx:7) To (o(x))))) /Zo will have p (xoly) as marginal, i.e., (x dxx
This proposal is similar to one step of the generative procedure used in DAPS, where in their case the tran- sition would correspond using the diffusion forward kerne,, i.e., x(xx|xo) = q(Xx|Xo). We make slight adjustment based on the following intuition. Consider- ing the setting of non-informative measurements, we ex- pect our posterior po(x))y) to coincide with the prior. To achieve this, we could choose {(x:|xo) such that rx xx41) (x.(xxx)1) Using q(x; xx) together with the prior (x, (x44) will, however, not match the covari- ance in pl(x, (x+1),, since the Gaussian approximation Po (xo 1)++) will inflate the variance by term př To counter this effect, we instead opt for covariance X (i.c., q(x, Xu) = A (xt Xg, X)T) such that in the unconditional case, re (x, xx+1) = p (x(xx=1) As everything is Gaus- sian, the marginalization in Equation (18) can be computed exactly, and we obtain our proposal for > 0 as
Po pxooly As SMC provides consistent approximations of all its target distributions (xx:)x )= it will in particular give a consistent approximation at f O.
Remark 3.2. Whether the assumptions needed for the marginals of the DAPS (1) = 0) and standard diffusion (1) = 1) prior to coincide actually are fulfilled does not affect the consistency in Proposition 3. I. If they are not fulfilled, the priors will be different, but the consistency refers to consistent approximations of the posterior induced by the chosen prior. This is in contrast to FPS (Dou & Song, 2023) and SMCDiff (Trippe et al., 2023), which rely on the assumption of the matching of the backward and forward processes for their target to coincide with the induced pos- terior, and hence do not enjoy the asymptotical exactness guarantees unless this assumption is met.
re re(xx = (xx125 (x++1, XP " M (19)

---

Decoupled Diffusion Sequential Monte Carlo
where 22 of - pi+1 and, for 1 = 0, ro ro y) (x (xo). We can directly see that in the non- informative case (i.e., By 00), we will recover the prior as M,11 = P and bi+1 = fo(x+t))
similar to previous work (Kawar et al., 2021; 2022; Car-
doso al., 2024), by writing A in terms of its singular value decomposition A = USVT where U E Rdyxdy and V E R xdz are orthonormal matrices, and S dy x dz- dimensional matrix with non-zero elements only on its main diagonal, we can multiply both sides of the measurement equation by U to obtain
can be noted that, although the proposal relies on approxi- mations (like the approximate posterior in Equation (16)), the SMC framework still provides an asymptotically ex-
act algorithm by targeting the desired posterior po (Xo:r)) yy)
Uº y = SV x x KUTU.
(22)
as established in Proposition 3.1. Approximations in the proposal do not affect that property.
By then defining y' U x' VTx, A' S, and é' = -'= ~ (1 (00, UIU?) = N(O, 1), we obtain the new measurement equation
If using the generalized DAPS prior from the previous sec- tion, we can also replace ((x,|xx) with dn (x, (xx/Xxx+ X Xo) (ef.
y' = A'''' c(,,(..1).
e'~
(23)
Equation (8) and make the corresponding matching of co- variance.
We can now run our DDSMC algorithm in this new basis and use the expressions from the diagonal case by replacing all variables with their corresponding primed versions, en- abling efficient sampling also for non-diagonal A, given an implementable SVD. An algorithm outlining an implemen- tation of DDSMC can be found in Algorithm 1.
Detailed Expressions for Diagonal A If first assuming that A is diagonal in the sense that it is of shape dy x dæ (dy I dz) with non-zero elements (a1....,,a .a,)) only along the main diagonal, we can write out explicit expressions for the proposal. In this case, the covariance matrix M,11 is diagonal with the icth diagonal element ......dz) becoming
3.3. D3SMC - A Discrete Version
We also extend our DDSMC algorithm to a discrete set- ting which we call Discrete Decoupled Diffusion SMC (D3SMC). Using x to denote a one-hot encoding of single variable, we target the case where the measurement can be described by transition matrix Qg
1i t i is dly Pitt
M (M
(20)
Pi+1
i i> dg
This means that the covariance matrix in the proposal is o2
diagonal with the elements of p + afpi4 tog for i
p = Categorical( x Qy),
(24)
dy. and of for i dy. Further, the mean of the i:th variable in the proposal becomes
and when the data consists of D variables, the measure- ment factorizes over these variables (i.e., for each of the D variables, we have a corresponding measurement). This setting includes both inpainting (a variable is observed with some probability, otherwise it is in an "unknown" state) and denoising (a variable randomly transitions into differ- ent state). To tackle this problem, we explore the D3PM- uniform model (Austin et al., 2021) and derive a discrete analogy to DDSMC which uses the particular structure of D3PM. We elaborate on the D3PM model in general and our D3SMC algorithm in particular in Appendix E.
11t1 (x a:p 0.pi4 (x(++11)
of of 030 Se f0(x(+1)s
Di
is dy i> dy-
(21)
We can see the effect of the decoupling clearly by consid- ering the noise-less case, Jy O. In that case, we are always using the known observed value (Xo): = y:/0, as the mean, not taking the reconstruction, nor the previous value (x++1)s ). into account.
4. Related Work
As the proposal is a multivariate Gaussian with diagonal covariance matrix, we can efficiently sample from it using independent samples from a standard Gaussian. We provide expressions for general choices of 1 in Appendix D.
SMC and Diffusion Models The closest related SMC methods for Bayesian inverse problems with diffusion pri- ors are the Twisted Diffusion Sampler (TDS) (Wu al., 2023) and Monte Carlo Guided Diffusion (MCGDifI) (Car- doso et al., 2024). TDS is a general approach for solving inverse problems, while MCGDiff specifically focuses on the linear- Gaussian setting. These methods differ in the choices of intermediate targets and proposals. TDS makes use of the reconstruction network to approximate the like- lihood time t as ply Xp = fxx)) and then add the
Non-diagonal A For a general dia-diagonal A, the pro- posal at first glance looks daunting as it requires invert- ing Mc+1 which is a (potentially very large) non-diagonal matrix, and sampling from a multivariate Gaussian with non-diagonal covariance matrix (which requires a compu- tationally expensive Cholesky decomposition). However,
5

---

Decoupled Diffusion Sequential Monte Carlo
Table 1. Resulls on the Gaussian mixture model experiment when using DDSMC and reconstructing xo using either Tweedie's formula (X+())+) = [x]] or solving the PF-ODE. Metric is the sliced Wasserstein distance between true posterior samples and samples from DDSMC, averaged over 20 seeds with 95% CLT confidence intervals.
TWEEDIR
PF-ODE
dg
dy
17
0.0
7 0.5
17 = 1.0
0.0
1) = 0.5
1) 1.0
1.90 + 0.48
1.78 + 0.44
1.57 + 0.43
1.15 + 0.24
0.88 + 0.33
1.01 + 0.39
8
2
0.69 0.27
0.64 = 0.29
0.50
0.56 0.23
0.34 = 0.20
0.39 +.29
0.28
+ 0.04
0.22 = 0.08
0.00
0.21 0.10
0.09 0.08
0.17 0.06
1.26 0.34
1.07 0.96 0.31
0.69 0.17
0.77 0.16
80
1.14
0.92 0.36
0.35
0.69 + 0.36
0.37 0.20
0.15
4
0.66 0.29
0.50 0.27
0.24
0.41 0.23
0.04
1.73 0.54
1.64 = 0.61
0.63
1.37 0.43
1.68 0.52
2.09 1.51
800
2
1.06 + 0.56
0.90 = 0.65
0.68
0.81 0.40
1.17 0.65
0.67
4
0.35 0.08
0.16 = 0.10
0.56 0.19
0.23 + 0.06
0.47 + 0.22
1.24 2 0.47
17 0.0
7 p--0 0.5
1.0
9 0.0
0.5
92- 1.0
T11
T1
Figure 1. Samples from DDSMC from the GMM experiments (d2 = 800 and d, = 1) using either Tweedie's 's formula (three left-most figures) or the solution of the PF-ODE (three right- most figures) as reconstruction, and using different values of 1) in the generalized DAPS prior. Blue samples are from the posterior, while red samples are from DDSMC. More examples can be found in Appendix G.I.
score of this approximate likelihood as a drift in the tran- sition kernel. This requires differentiating through the re- construction model w.r.t. xg, which can incur a significant computational overhead. MCGDiff instead uses the forward diffusion model to push the observation y "forward in time". Specifically, they introduce potential function at time which can be seen as a likelihood corresponding to the ob- servation model ý: = Ax,, where Y, is a noised version (according to the forward model at time t) of the original observation y. DDSMC differs from both of these methods, and is conceptually based on reconstructing X from the current state Xt, performing explicit conditioning on y at time / = 0. and then pushing the posterior distribution forward to time f 1 using the forward model. We further elaborate on the differences between TDS, MCGDiff, and DDSMC in Appendix C (see also the discussion by Zhao et al. (2024)).
uous (Liu et al., 2024) and discrete (Ekström Kelvinius & Lindsten, 2024) setting.
Posterior Sampling with Diffusion Priors The closest non-SMC work to ours is of course DAPS (Zhang et al., 2024), which we build upon, but also generalize in sev- eral ways. DDSMC is also related to IIGDM (Song et al., 2023), which introduces the Gaussian approximation used in Equations (11) and (15). Other proposed methods include DDRM (Kawar al., 2022) which defines task-specific conditional diffusion process which depends struction fx(x)), but where the optimal solution can be approximated with model trained on the regular uncondi- tional task. Recently, Janati et al. (2024) proposed method referred to as DCPS, which also builds on the notion of inter- mediate targets, but not within an SMC framework. Instead, to sample from these intermediate targets, they make use of Langevin sampling and a variational approximation that is optimized with stochastic gradient descent within each step of the generative model. All of these methods include various approximations, and contrary to DDSMC, none of them provides a consistent approximation of a given poste- rior. Other sampling methods include MCMC methods like Gibbs sampling (Coeurdoux et al., 2024; Wu et al., 2024).
recon-
SMCDiff (Trippe al., 2023) and FPS (Dou & Song. 2023) are two other SMC algorithms that target posterior sampling with diffusion priors, but these rely on the assumption that the learned backward process is an exact reversal of the forward process, and are therefore not consistent in general. SMC has also been used as a type of discriminator guidance (Kim al., 2023) of diffusion models in both the contin-
6

---

Decoupled Diffusion Sequential Monte Carlo
Table Comparison of DDSMC with other methods in the Gaussian mixture setting. Numbers for DDSMC the best numbers from Table 1. *All methods have been run for 20 different seeds, but TDS shows instability and crashes, meaning that their numbers are computed over fewer runs.
dx
dy
DDSMC
MCGDIFF
TDS*
DCPS
DDRM
DAPS
0.88 + 0.33
1.79 + 0.54
9.72 9.88
2.51 + 1.02
3.83 = 1.05
5.63 + 0.90
8
0.34 + 0.20
0.80 = 0.41
1.23+2257
1.35 0.70
0.92
5.93 1.16
0.09 + 0.08
0.26 = 0.21
3.02
0.44
0.29
4.85 1.34
0.50 0.16
1.06 0.36
1.19 0.41
5.19
6.85 1.16
80
0.37 + 0.20
1.04 1 0.49
2.63
1.10 0.55
1.09
8.49 - 0.92
0.21 + 0.13 4
0.80 = 0.42
1.47 = 1.42
0.56 + 0.24
+.25
9.04 + 0.74
1.37 + 0.43
1.64 0.48
2.45 0.96
2.45
+ 0.58
7.15
7.03 - 1.20
800
2
0.81
+ 0.40
1.29 = 0.66
2,84 - 1.32
2.80
+ 0.91
8.21 = 0.88
8.31 + 1.01
0.16 + 0.10
1.21 = 0.94
3.58 - 3.18
1.84 + 0.76
8.66
0.87
9.21 0.83
DDSMC
MCGD
TDS
DCPS
DDRM
DAPS
Figure Qualitative comparison between DDSMC (using PF-ODE as reconstruction and 7) = 0) and other methods on the GMM experiments, d, = 800 and dy 1. More qualitative comparisons can be found in Appendix G.1
5. Experiments
Table 3. LPIPS results on FFHQ experiments. The noise level is oy = 0.05, and the tasks are inpainting box in the mid- dle, outpainting right half of image, and super-resolution 4x. Numbers are averages over 1k images. Lower is better.
5.1. Gaussian Mixture Model
We first experiment on synthetic data, and use the Gaussian mixture model problem from Cardoso al. (2024). Here, the prior on x is Gaussian mixture, and both the posterior and score therefore known closed form² (we give more details in Appendix G.1), enabling to verify the efficiency of DDSMC while ablating all other errors. As a metric, we use the sliced Wasserstein distance (Flamary et al., 2021) between 10k samples from the true posterior and each sampling algorithm.
MIDDLE
HALF
SR4
DDRM
0.04
0.25
0.20
DCPS
0.03
0.20
0.10
DAPS
0.05
0.24
0.15
MCGDIFF
0.10
0.34
0.15
DDSMC-0.0
0.07
0.26
0.27
TWEEDIE
DDSMC-0.5
0.07
0.27
0.20
DDSMC-L.
0.05
0.24
0.14
We start with investigating the influence of n and the re- construction function fo- We DDSMC with N = 256 particles, and use T = 20 steps in the generative process. As reconstruction, we compare Tweedie's formula and the DDIM ODE solver (Song et al., 2021a), see Equation (78) in the appendix, where we solve the ODE using as many steps as there are "left" in the diffusion process. In Table I, we see that using Tweedie's reconstruction requires larger value of 1). This be explained by the fact that 7) 0 (DAPS prior) requires exact samples from pa (xol (x) for the DAPS
DDSMC-0.0
0.05
0.23
0.21
ODE
DDSMC-0.5
0.05
0.23
0.15
DDSMC-100
0.08
0.4
0.36
prior to agree with the original prior (see Appendix A), and this distribution is more closely approximated using the ODE reconstruction. For higher dimensions (i.e., d, 800) and using n 0, Tweedie's formula works slightly better than the ODE solver. Qualitatively, we find that using smaller and/or using Tweedie's 's reconstruction tends to concentrate the samples around the different modes, see Fig- ure 1 for an example of d, = 800 and d, = (additional examples in Appendix G. This plot also shows how, in
"This includes using the DDPM (Ho et al., 2020) "Variance- preserving" formulation of a diffusion model. We have included derivation of DDSMC for this setting in Appendix

---

Decoupled Diffusion Sequential Monte Carlo
UT
2002
SCANC Twendla
DDANC-OOR
DCPS
2CI
DAP5
HORDLE
bI
DiN
piN
pin
bi
bI
hi
hi
Figure 3. Examples of generated images for the outpainting task. The DDSMC samples are ordered with = 0 to the left. vj = 0.5 in the middle, and 1) = 1 the right. Ground truth images FFHQ images ID 100, 103, and 110 (see Table 11 in the appendix for attribution).
high dimensions, using a smaller 7y is preferable, which could be attributed to lower ry enabling larger updates neces- sary in higher dimensions. It hence seems to be interplay between the data dimensionality, the inverse temperature 7) and the reconstruction function.
base of DCPS and use IK images from the validation set and compute LPIPS (Zhang et al., 2018) quantitative metrie (see PSNR results in Appendix G.2). The results can be found in Table 3, in addition to numbers for DDRM, DCPS, DAPS, and MCGDiff. Standard deviations can be found in Appendix G.2. We used 5 particles for our method, and when using the PF-ODE as reconstruction, we used ODE steps. Further implementation details are available in Appendix G.2. It should be noted that LPIPS measures perceptual similarity between the sampled image and the ground truth, which is not the same how well the method samples from the true posterior. Specif- ically, it does not say anything about diversity of samples or how well the model captures the posterior uncertainty, which we could verify in the GMM experiment. Neverthe- less, the numbers indicate that image reconstruction is on par with previous methods. Notably, MCGDiff, which is also SMC method and have the same asymptotic guar- antees in terms of posterior sampling, performs similar or worse to DDSMC.
Next, we compare DDSMC with other methods, both SMC- based (MCGDiff and TDS) and non-SMCC-aseed (DDRM,
DCPS, DAPS). For all SMC methods, we use 256 parti- cles, and accelerated sampling according DDIM (Song et al., 2021a) with 20 steps, which we also used for DAPS to enabling evaluating the benefits of our generalization of the DAPS method without confounding the results with the effect of common hyperparameters. For DDRM and DCPS, however, we used I 100 steps. More details are available in Appendix G.1. We see quantitatively Table 2 that DDSMC outperforms all other methods, even when using Tweedic's reconstruction (which requires the same amount of compute MCGDiff), We provide additional results with fewer particles in the appendix. As our pro- posal is essentially equal to DAPS (if using X² = o² in Equation (19)), running DDSMC with a single particle is essentially equal to DAPS (we verify this empirically in Ap- pendix G.1.4). Therefore, we can conclude that it is indeed the introduction of SMC (multiple particles and resampling) that contributes to the improved performance over DAPS. A qualitative inspection of generated samples in Figure 2 shows how DDSMC is the most resistant method to sam- pling from spurious modes, while DAPS struggles to sample from the posterior. These are general trends we see when repeating with different seeds, see Appendix G.I.
measurement of
Inspecting the generated images, we can see a visible effect when altering the reconstruction function and r7, see Fig- ure 3 and more examples in Appendix G.2. It seems like increasing 1) and/or using the PF-ODE as reconstruction function adds more details to the image. With the GMM experiments in mind, where we saw that lower and using Tweedie's 's formula as reconstruction function made samples more concentrated around the modes, it can be argued that there a similar effect here: using Tweedie's 's formula and lower 1) lead sampled images closer a "mode" of im- ages, meaning details averaged out. On the other hand, changing to ODE-rcconstruction and increasing 17 further away from the mode, which corresponds to more details in the images, For the case of ODE and 1/ = 1.0, we see clear artifacts which explains the poor quantitative results.
5.2. Image Restoration
We now turn our attention to the problem of image restora- tion, and use our method for inpainting, outpainting, and super-resolution on the FFHQ (Karras et al., 2019) dataset (downsampled to 256 x 256), using pretrained model by Chung et al. (2023). We implement DDSMC in the code-
hnttps:// com/ Sali---OUFDD//ccp.
8

---

Decoupled Diffusion Sequential Monte Carlo
ADF3D
ADP3D
DISMM-1
DI35M-1
D06ME-50
DOSMC-AD
DOSMC 100
DOSMC 100
RM50 (A)
RM50 (A)
ADF3D
D0SMCC1
D0SMC
D0SMM-10
10
D4SMMC-9 50
3
D0SMC-100 100
16
16
16
32
Subsamping factor
Subsamping
Subsampling factor
Figure 4. RMSD vs subsampling factor for the protein structure completion problem on the ATAD2 protein (Davison et al., 2022) (PDB identifier 7qu (Turberville al., 2023)) and different noise levels 0 (left), 0.1 (middle), 0.5 (right). Comparing ADP-3D (Levy al., 2024) and DDSMC with different number of particles.
and 3, 0.6, i.e., the observed pixel has the same value as the original image with probability 0.7, and opposite with probability 0.3. We present qualitative results for D3SMC with N = particles Figure 5, and more qualitative re- sults can be found in Appendix G.4. While the digits are often recovered, there are cases when they are not (c.g.. becoming 9). Looking at multiple draws in the appendix, it seems the model in such cases can sample different digits, suggesting a multi-modal nature of the posterior.
9
44447
Figure Qualitative results on binary MNIST using the disercte counterpart of DDSMC, D3SMC. Top is the ground truth, middle measurement, and bottom is sampled image. More results can he found Appendix G.4
6. Discussion & Conclusion
We have designed an SMC algorithm, DDSMC, for poste- rior sampling with diffusion priors which also extended to discrete data. The method is based decoupled diffu- sion, which we generalize by introducing a hyperparameter that allows bridging between full decoupling and standard diffusion (no decoupling). We demonstrate the superior performance of DDSMC compared to the state-of-the-an on synthetic problem, which enables a quantitative evalua- tion of how well the different methods approximate the true posterior. We additionally test DDSMC on image recon- struction and protein structure completion where it performs par with previous methods. In the image case, it performs slightly better than the alternative SMC method MCGDiff, while underperforming compared to DCPS. However, while LPIPS indeed is a useful metric for evaluating image recon- struction, it does not inherently capture posterior approxi- mation quality, which is the aim of our method. We found that the methods performing particularly well on image re- construction struggle on the synthetic task, highlighting a gap between perceived image quality and ability to approxi- mate the true posterior. As DDSMC still performs on par with previous work on images and proteins, and the same time showing excellent performance on the GMM task, we view DDSMC a promising method for bridging between solving challenging high-dimensional inverse problems and maintaining exact posterior sampling capabilities.
5.3. Protein Structure Completion
As different type of problem, we test DDMSC on the pro- tein structure completion example from Levy et al. (2024), where we try predict the coordinates of all heavy atoms in the backbone, observing coordinates of all 4 heavy atoms in the backbone of every l residue. More details are in Ap- pendix F. Data used is from the Protein Data Bank? (Berman et al., 2000; 2003). Figure shows an example of the RMSD for different noise levels a and subsampling factors k. Es- pecially on high noise levels, we are close to or better than ADP-3D (Levy al., 2024), and also clear effects of introducing more particles, indicating the effectiveness of introducing the SMC aspect. Experiment details and more results are in Appendix G.3.
5.4. Discrete Data
As a proof of concept of our discrete algorithm D3SMC (see detailed description in Appendix E), we try denoising on binarized MNIST (Deng, 2012) (i.c., each pixel either 0 or 1), cropped to 24 x 24 pixels to remove padding. As a back- bone neural network, we used U-Net (Ronneberger al., 2015) trained with cross-entropy loss. We use a measure- ment model y xQ, with Q2 (1 By) /d
www. rcsb. org/, www. www.b.
9

---

Decoupled Diffusion Sequential Monte Carlo
Acknowledgments
Priors in Bayesian Inference. IEEE Transactions on Im-
age Processing, 33:3496-35077 2024. ISSN 1941-0042. doi: 10. 00.100/T112222444443333
This research was supported by Swedish Research Council (VR) grant no. 2020-04122, 2024-05011, the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, and the Excellence Center at Linköping-Lund in Information Technology (ELLIIT). Computations were enabled by the Berzelius resource provided by the Knut and Alice Wallen- berg Foundation at the National Supercomputer Centre.
Corso, G., Stärk, H.,, Jing, B., Barzilay, R., and Jaakkola, T. S. DiftDDck: Diffusion Steps, Twists, and Turns for Molecular Docking. In The Eleventh International Con- ference on Learning Representations, 2023.
Cruz, M. R., Cristy, S., Guha, S., De Cesare, G. B., Evdoki- mova, E., Sanchez, H., Borek, D., Miramón, P., Yano, J., Fidel, L.., Savchenko, A., Andes, D. R., Stogios, P. J., Lorenz, M. C.,, and Garsin, D. A. Structural and func- tional analysis of Ent'V reveals 12 amino acid fragment protective against fungal infections. Nature Communica- tions, 13(1)7 October 2022. ISSN 2041-1723. doi: 10. 10.1038/s41460 4407000441147--222333633333
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Cuthbert, J. and Goulding, C. W. Mycobacterium ther- moresistible MmpS5, September 2023.
References
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. Structured Denoising Diffusion Models in Dis- crete State-Spaces. In Advances in Neural Information Processing Systems, November 2021.
Cuthbert, B. J., Mendoza, J., de Miranda, R., Papavinasasun- daram, K., Sassetti, C. M., and Goulding, C. W. The struc- ture of Mycobacterium thermoresistibile MmpS5 reveals a conserved disulfide bond across mycobacteria. Metal- lomics, 16(3) 1, March 2024, ISSN 1756-5901. doi: 10. 1033/mtomcss/maae111...
Berman, H.. Henrick, K., and Nakamura, H. Announcing the worldwide Protein Data Bank. Nature Structural & Molecular Biology, 10(12)9900998, December 2003. ISSN 1545-9985. doi: 10. 10.100/ss22203-9800
Davison, G., Martin, M. P., Turberville, S., Dormen, S., Heath, R., Heptinstall, A. B., Lawson, M., Miller, D. C., Ng, Y. M., Sanderson, J. N., Hope, I., Wood, D. J., Cano, C., Endicott, A.. Hardcastle, I. R., Noble, M. E. M., and Waring, M. Mapping Ligand Interactions of Bromod- omains BRD4 and ATAD2 with FragLites and PepLites- Halogenated Probes of Druglike and Peptide-like Molec- ular Interactions. Journal of Medicinal Chemistry, 65 124:6-511115432 November 2022. ISSN 0022-2623. doi: 2aa.022/acc.jmcccchem......
Berman, H. M., Westbrook, J., Feng, Z., Gilliland, G., Bhat, T. N., Weissig, H., Shindyalov, I. N., and Bourne, P. E. The Protein Data Bank. Nucleic Acids Research, 28(1): 235-242, January 2000. ISSN 0305-1048. doi: 10.1093/ m.r228.1.235.
Campbell, A., Benton, J., Bortoli, V. D., Rainforth, T., Deli- giannidis, G., and Doucet, A. A Continuous Time Frame- work for Discrete Denoising Models. In Advances in Neural Information Processing Systems, May 2022.
Del Moral, P. Feynman-Kac Formulae. Probability and Its Applications. Springer, New York, NY, 2004. ISBN 978- 188-1-441--1922. 978-1-4684-9393.1 978-1-4684-9393-1
Cardoso, G., el Idrissi, Y. J., Corff, S. L., and Moulines, E. Monte Carlo guided Denoising Diffusion models for Bayesian linear inverse problems. In The Twelfth Interna- tional Conference on Learning Representations, 2024.
doi: 10.1007/
Deng. L. Images for Machine Learning Research [Best of the Web]. IEEE Signal Processing Magazine, 2916):141- 142, November 2012. ISSN 1558-0792. doi: 10.1109/ MSP.2012.2211477..
The MNIST Database of Handwritten Digit
Chen, N., Zhang, Y., Zen, H., Weiss, R. J., Norouzi, M., and Chan, W. WaveGrad: Estimating Gradients for Waveform Generation, October 2020.
Chung, H., Kim, J., Mccann, M. T., Klasky, M. L., and Ye, J. C. Diffusion Posterior Sampling for General Noisy Inverse Problems. In The Eleventh International Confer- ence on Learning Representations, 2023.
Dhariwal, and Nichol, A. Diffusion Models Beat GANs Image Synthesis. In Advances in Neural Information Processing Systems, volume 34, pp. 8780-8794 Curran Associates, Inc., 2021.
Coeurdoux, F., Dobigeon, N., and Chainais, P. Plug-and- Play Split Gibbs Sampler: Embedding Deep Generative
Dieleman, S., Sartran, L., Roshannai, A., Savinov, N., Ganin, Y., Richemond, P. H., Doucet, A., Strudel, R.,
10

---

Decoupled Diffusion Sequential Monte Carlo
Dyer, C., Durkan, C., Hawthorne, C., Leblond, R., Grath- wohl, W., and Adler, J. Continuous diffusion for categori- cal data, December 2022.
Janati, Y., Moufad, B., Durmus, A. O., Moulines, E., and Olsson, J. Divide-and-Conquce Posterior Sampling for Denoising Diffusion priors. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, November 2024.
Dix, S. R., Aziz, A. A., Baker, P. J., Evans, C. A., Dickman, M.J., J., Farthing, R. J., King, Z. L. S., Nathan, S., Partridge, L. J., Raih, M.. Sedelnikova, S. E., Thomas, M. S., and Rice, D. W. Structure of the bacterial toxin, TecA, as- paragine deamidase from Alcaligenes faecalis, November 2022.
Karras, T., Laine, S., and Aila, T. A Style-Based Generator Architecture for Generative Adversarial Networks. In 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. doi: 10.
June 2019.
Dou, and Song, Y. Diffusion Posterior Sampling for Linear Inverse Problem Solving: Filtering Perspective. In The Twelfth International Conference on Learning Representations, October 2023.
Karras, T., Aittala, M.. Aila, T., and Laine, S. Elucidating the Design Space of Diffusion Eaffusion-Based Generative Models. Advances in Neural Information Processing Systems, 35: December 2022.
Ekström Kelvinius, F. and Lindsten, F. Discriminator Guid- ance for Autoregressive Diffusion Models. In Proceed- ings of The 27th International Conference on Artificial Intelligence and Statistics, pp. 3403-3411. PMLR, April 2024.
Kawar, B., Vaksman, G., and Elad, M. SNIPS: Solving Noisy Inverse Problems Stochastically. In Advances in Neural Information Processing Systems, volume 34, pp.
21757-21769. Curran Associates, Inc., 2021.
Flamary, R., Courty, N., Gramfort, A., Alaya, M. Z.,, Bois- bunon, A., Chambon, S., Chapel, L., Corenflos, A., Fatras, K., Fournier, N., Gautheron, L., Gayraud, N. T. H., Janati, H., Rakotomamonjy, A., Redko, I.. Rolet, A., Schutz, A., Seguy, V., Sutherland, D. J., Tavenard, R., Tong, A., and Vayer, T. POT: Python Optimal Transport Journal of Machine Learning Research, 22(78):1-8, 2021. ISSN 1533-7928.
Kawar, B., Elad, M., Ermon, S., and Song, J. Denoising Diffusion Restoration Models, Advances in Neural Infor- mation Processing Systems, 35:23593-23666, December 2022.
Kim, D., Kim, Y., Kwon, S. J., Kang, W., and Moon, I.-C. Refining Generative Process with Discriminator Guid- ance in Score-based Diffusion Models. In Proceedings of the 40th International Conference on Machine Learning, pp. 16567-16598. PMLR, July 2023.
Ho, J. and Salimans, T. Classifier-Free Diffusion Guidance, July 2022.
Kong, Z., Ping, W., Huang, J., Zhao, K.. and Catanzaro, B. DiffWave: A Versatile Diffusion Model for Audio Synthesis. In International Conference on Learning Rep- resentations, 2021.
Ho, J., Jain, A., and Abbeel, P. Denoising Diffusion Prob- abilistic Models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates, Inc., 2020.
Levy, A., Chan, E. R., Fridovich-Keil. S., Poitevin, F, Zhong, E. D., and Wetzstein, G. Solving Inverse Prob- lems in Protein Space Using Diffusion-Basse Priors, June 2024.
Hoogeboom, E., Satorras, V. G., Vignac, C., and Welling, M. Equivariant Diffusion for Molecule Generation in 3D. In Proceedings of the 39th International Conference on Machine Learning, pp. 8867-8887. PMLR, June 2022.
Liu, Y., Zhang, Y., Jaakkola, T., and Chang, S. Correcting Diffusion Generation through Resampling. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 87 8713-8723, 2024.
Huegle, M. Crystal structure of BRD4(1) in complex with the inhibitor MPM2, February 2023.
Ingraham, J. B., Baranov, M., Costello, Z., Barber, K. W., Wang, W., Ismail, A., Frappier, V., Lord, D. M., Ng- Thow-Hing, C., Van Vlack, E. R., Tie. S., Xue, V., Cowles, C.C Leung, A., Rodrigues, V., Morales-Perez, L., Ayoub, A. M., Green, R., Puentes, K., Oplinger, F. Panwar, N. V., Obermeyer, F., Root, A. R., Beam, A. L., Poelwijk, F. J., and Grigoryan, G. Illuminating protein space with a programmable generative model. Nature, 62707789) November 2023. ISSN 1476-4687. doi: 10. 1000000//4111 -58--003-677888-8888
Naesseth, C. A., Lindsten, F., and Schön, T. B. Elements of sequential Monte Carlo. Foundations and Trends in Machine Learning, 12(3):307-322, 2019, doi: 10.1561/ 2200000074.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Om- mer, B. Righ-Resslutton Image Synthesis With Latent Diffusson Models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp- 10684-10695, 2022.

---

Decoupled Diffusion Sequential Monte Carlo
Ronneberger, O., Fischer, P., and Brox, T. U-Net: Convolu- tional Networks for Biomedical Image Segmentation. In Navab, N., Hornegger. J., Wells, W. M., and Frangi, A. F. (eds.), Medical Image Computing and Cosputer-Assisted Intervention - MICCAI 2015, pp. 234-241, Cham, 2015. Springer International Publishing. ISBN 978-3-319- 24574-4. doi: 10. 10.1007/978---311
Uria, B., Murray, I., and Larochelle, H. A Deep and Tractable Density Estimator, In Proceedings of the 31st International Conference Machine Learning, pp. 467- 475. PMLR, January 2014.
Warstat, R., Pervaiz, M., Regenass, P. Amann, M.. Schmidtkunz, K., Einsle, O., Jung, M., Breit, B., Hügle, M., and Günther, S. A novel pan-selective bromodomain inhibitor for epigenetic drug design. European Journal of Medicinal Chemistry, 249:115134 March 2023. ISSN 0223-5234. doi: 10.1016fj.jjmech2222.... 115139.
Saharia, C., Chan, W., Saxena, S., Li, I.., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., Ho. J.. Fleet, J.J., and Norouzi, M. Pho- torealistic lext-to-Image Diffusion Models with Deep Language Understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-36494, December 2022.
Wu, L., Trippe, B.., Naesseth, C.,, Blei, D., and Cunningham, J. P. Practical and Asymptotically Exact Conditional Sampling in Diffusion Models. Advances in Neural Infor- mation Processing Systems, 1111172--31433 December 2023.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep Unsupervised Learning using Nonequi- librium Thermodynamics. In Proceedings of the 32nd International Conference Machine Learning, pp. 2256 2265. PMLR, June 2015.
Wu, Z.,, Sun, Y., Chen, Y., Zhang, B., Yue, Y., and Bouman, K. Principled probabilistic imaging using diffusion mod- els as plug-an plug-and-play priors. In The Thirty-Eight Annual Conference on Neural Information Processing Systems, 2024.
Song, J., Meng, C., and Ermon, S. Denoising Diffusion Im- plicit Models. In International Conference on Learning Representations, 2021a.
Xu, M., Yu, L., Song, Y., Shi, C., Ermon, S., and Tang, J. GeoDiff: A Geometric Diffusion Model for Molecular Conformation Generation. In International Conference on Learning Representations, 2022.
Song. J., Vahdat, A., Mardani, M., and Kautz, saaudoinverse-gudder diffusion models for inverse prob- lems. In International Conference on Learning Represen- tations, 2023.
Zhang, B., Chu, W., Berner, J., Meng, C., Anandkumar, A., and Song, Y. Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing, July 2024.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. -csre-Based Generative Modeling through Stochastic Differential Equations. In Interna- tional Conference on Learning Representations, 2021b.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In 2018 IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 586-595, Salt Lake City, UT, June 2018. IEEE. ISBN 978-1-5386- 6420-9, doi: 10.1109/CPR.20018.OO
Stogios, P. J., Eydokimova, E., Kim, Y., Garsin, D., Savchenko, A., Joachimiak, A., Satchell, K. J. F,, and Center for Structural Genomies of Infectious Diseases (CSGID). Crystal structure of EntV136 136 from Enterococ- faecalis, October 2022.
Zhao, Z., Luo, Z., Sjölund, J., and Schön, T. B. Conditional sampling within generative diffusion models, September 2024.
Sun, H., Yu, L., Dai, B., Schuurmans, D., and Dai, H. Score- based Continuous-time Discrete Diffusion Models. In International Conference on Learning Representations, February 2023.
Trippe, B. L., Yim, J., Tischer, D., Baker, D., Broderick, T., Barzilay, R., and Jaakkola, T. Diffusion Proba- bilistic Modeling of Protein Backbones in 3D for the motif-scaafolding problem. In The Fleventh International Conference Learning Representations 2023.
Turberville, S., Martin, M. P., Hope, I., and Noble, M. E. M. ATAD2 complex with Fraglite22, March 2023.
12

---

Decoupled Diffusion Sequential Monte Carlo
A. Proofs
A.1. Equality between marginals
The marginal distributions (x:) and px(x are equal, as follows from the following proposition, inspired by Zhang et al. (2024).
Proposition A.1. Conditioning on a sample Xt p(xx) to sample xo ~ ((xo)xx)), and then forgetting about x, when sampling X p(x{-1|Xxx) Xx)), leads to draw from p(x:-1).
Proof.
/ p(x;-1 p(xo) dxo= = ((0)
p(x,-1)=
(25)
= / p(x)-1b60) /
p(x0 X )p( (())))(()))) ddxx dxo
(26)
(.a)
/ 120 xx)( )xxxxxxx;
(27)
Ex, 1x0).
(28)
Remark A.2. In practice, the forward process is given by 9. meaning p(x,1|(xo) would be replaced by - Xo) above. Hence, for the above proposition hold, it assumes that the backward process is the inverse of the forward process (i.e., p(x+-1 (x)) (xx)-
Remark A.3. Proposition A.1 the same as the proposition in (Zhang et al., 2024), but without conditioning on y. Our proof can be used also for the conditional case (by only adding the corresponding conditioning on y), but different from the by Zhang et al. (2024) does not rely on the assumption of their graphical model.
B. DDSMC for DDPM/VP diffusion models
In the DDPM (Ho et al., 2020) or Variance-preserving (VP) (Song et al., 2021b) formulation of a diffusion model, the forward process can described by
((x++1|x;) =
B:+1))
(29)
which leads
((x/\x0) = -N(xxl (1-x))
(30)
where ā₂ = -I a, with a, = - 3z.
B.1. Target distribution
The target distribution changes as the transition (x,xxxx+1) = changes. The new expression is now (x+(x(+1)) = N (v (0x r)
(Xo)dxo changes when q(x, Xx)
B.2. Proposal
In the DDPM formulation, we reuse the same Gaussian prior over Xo given x, as before, (x = -N((((x))))) and hence the same posterior (xoo|x++1.) (x(1,) M, However, we use ((x(1x)) = -N(x (oxx////aaaa Vo,x XTT) which means that the proposal becomes
xx(x(x++1,))) = / (xxx
(31)
-N(aor (x)+,,,)) 171 a.M.))
(32)
13

---

Decoupled Diffusion Sequential Monte Carlo
To match the prior in case of non-informative measurements, set X 1 dg a oi + pi+1).
For expressions with generalized DAPS prior (i.c., arbitrary value of r)), see Appendix D.
C. Comparison with TDS and MCGDiff
this section, we write down the main components in an SMC algorithm, the target distributions and proposals, for Twisted Diffusion Sampler (TDS) (Wu al., 2023) and Monte Carlo Guided Diffusion (MCGDiff) (Cardoso et al., 2024),
C.1. TDS
Target distributions TDS approximates the intractable likelihood ply xx) by p(y(xx) = ply|xo = fo(x))). The intermediate target distributions for TDS are hence
T II
TDS (X:T) = ply( (x(xr) "
pe ( Xx Xs+1),
(33)
where
po(x, x.-1 = N (x,|Xs+1 + (6+180(xx++1)) 1),82+117
(34)
Bet1
3
=N
Xs
X +
2
fx(x++1) (xxx+1))(++1))
(35)
0x+1
and the last equality comes from rewriting the score (fo(xx+1) - 1x-1111111 x 1/22+1 according Tweedie's formula.
Proposal In TDS, they use the proposal
rt
TDS (xxxx+111) y) = N (xe (x+41 + 130+1(50(xe441 + V Vxe.i log (og((y|x(+1)) Be+1)
(36)
(1
Sc+1
B(+1 fo(x(+1) + Vx log ((1/))++)),..) Bo+1
= N xel
X++1 +
(37)
0741
where again the last equality is from rewriting the score according to Tweedie's formula. This proposal is reminiscent of classifier guidance (Dhariwal& & Nichol, 2021), where the unconditional score is combined with the gradient of some log likelihood, in this case the log likelihood log p(y (x--1) from the target distribution. As this likelihood relies on the reconstruction fo 1), computing the gradient of log p(y xx+1) with respect to x means differentiating through the reconstruction, which could be computationally very expensive. On the other hand, this proposal is limited to the linesr-Gaussian case, but works with any differentiable ply) x++1).
C.2. MCGDiI
formulation of diffusion models when describing their method, and will use here. To discuss MCGDiff, we will work under their initial premises that y Tyc, i.e., A merely extracts the top coordinates of (denoted x), and is potentially noisy observation of these coordinates The variable x represents the completely unobserved coordinates. We also follow their presentation, starting with the assumption of noise-free observation then continuing to the noisy case.
MCGDiff relies on the DDPM or
C.2.1. NOISELESS CASE
Target distributions In the noiseless case, they target for 0
T-1
(X(:r) = (x))
II
(x x4+1)
ly X,) DolyXoo)1x {x1)
(38)
by (xx )pol (x,, x1)
'For general A, they also make of the singular value decomposition of A to make change of basis where the "new" A has this property.
14

---

Decoupled Diffusion Sequential Monte Carlo
where pa is the regular diffusion model. By then defining likelihood for i (ojo(y|xo) xx) dy (Xx) and Toooy ())())(x)))
N(Vory, T) for & > 0, they have target the general form
[y)o(((x;) p(x, (x4 1c+1jo (y/x+11)))))+))))
(xxe))
(39)
Proposal
For 0 < tt T, they construct their proposal
(x x 2 (v
po Xx x ) o ())x,)
7x(x|x+41,)= J (x X Z (y [x, = po (xx,1x441) I
(40)
(x x / xx xx)dx;
and as everything is Gaussian (transition and likelihood), all terms can be computed exactly. In words, they use the "regular" diffusion model (0X,(xx+11) for the unobserved coordinates, and for the observed coordinates, they also weigh in the likelihood the update. For they instead use
ro(xo|x1,y) by (Xo)po (Xo)\1)
(41)
Weighting function We have not written out the weighting function Ya (x(x)) (xtt0///)=/1/141xxx=x111 for DDSMC and TDS, but we do for MCGDiff. With the target and proposal above, the weighting function becomes
f (x x 7r y Se, dxx,
T: (
We Wx(x,, Xx+1)
pe (xr (x++1) =
(42)
(x Xx+ 2ejo (y Xx)
10+1 (y (x+1)
1/g: (xxx(x+x+)
J po (xe Xx++ Aa (y
(43)
(
We do this as this weight does not actually depend on X and can hence be computed before sampling from the proposal. This opens up the possibility for so called fully adapted particle filter, where one can sample exactly from the target and the importance weights are always 1/N.
C.2.2. NOISY CASE
In the noisy case, they assume that there exists timestep r such that o/ = 1-ar, They use as targets for 2 T
They They further define ÿ. = Vory
ar
T-1 II
(((x))) = gi (xx)
po (x, (x+1))
(44)
ge (xx))
Po (xc xx+1 7 (X++1T)
(45)
9t-1 (x++1)
with the potential
gx(x.) = N (xxxx//ers, k) at )1)1)
(46)
For = 0, they target
Yo (xo, xy:T) plyl Xo Xo)pa pa (xo x yoo (x() II x(xx
(47)
ply Ko 1o)
Yo (Xr:7)
(48)
9- 9+(x))
What this means that they between T and r run their algorithm conditioning on y, and then sample unconditionally between 7 and 0.
15

---

Decoupled Diffusion Sequential Monte Carlo
D. Detailed Expressions for General DAPS Prior
generalize the DAPS prior by tempering the "likelihood" g(xe+1 (x),, and denote
On (x xo) x g (xx)
(49)
as in Equation (8).
D.1. Variance Exploding
In the "Variance Exploding" setting that we have used throughout the paper, Equation (49) becomes
(x/(xxx11,x N
Bet1
no?
X(+1,
n17 Bu+1 1)
Be+1
(50)
Bi+1
n1o
+ B4+1
We then get our generalized backward kernel (xxx/xx44) by replacing x with the reconstruction fo fo(x(+1).i.) i..,
py (x /V ( 34+1 fo (x(+1) +
noi
Xt+1, Bi+107 not + 3 1)
(51)
no?? + Br+1
nol + Bo+1
For the proposal, the posterior f(xo 1 y) remains as in Equation (16), but we use generalization
(x(xxXx+1.3) = f
σn (x4 X++1, v)xxo
(52)
where (x,xx, Xo) is on the form
(x =N (
Be+1 Xo +
no?
xx+1, 221
(53)
!107 + Bc+1
noo + Be+1
This gives the proposal
rx (xxx|xx+1,y))
Be+1 Mir bet1
no?
(13 Bet1
2
=N
1441,111+
Miti).
(54)
?oo + But1
no? + Bi+1
no? + Bc+1
where we choose
B4+10?
B441P441
A
(55)
110 + B141
2
!oo?
Br+1
so that the proposal matches the prior in case of non-informative measurements (Ty 00).
D.2. DDPM/VE
Using the same techniques as for VE but replacing q()) with its corresponding expressions, get the transition
p3 (x(xx(x++1) N/ ( VanBe+ (x( 1) / Be+ xx)x++1
n1 nā++1 + Bet1" 1
B141 a.)
(56)
n8 n0 + Be+1
and with the same posterior Xx,, x(+1) and the same procedure before, we get the proposal as
(xxxxxx+1,,)) - -N (VāuSepz M M b + 7 B1+1(1 a1)x++1 121
vo.3141
Mi
- 11 gā+t1 + Be+1
17 1/3141 - mac-1 + B141
(57)
with
X² 1 ā.
(58)
17 n8 na++1 + Bot1
16

---

Decoupled Diffusion Sequential Monte Carlo
E. Discrete Version
E.1. Diserete Denoising Diffusion Probabilistic Models
DDSMC relies on the data Xo being a vector of continuous variables. For discrete data, there are a number of different formulations developed like (Austin et al., 2021; Campbell al., 2022: Dieleman et al., 2022; Sun al., 2023). our work, focus on the Discrete Denoising Diffusion Probabilistic Models (D3PM) model (Austin al., 2021).
To introduce D3PM, we will denote by bold one-hot encoding of single discrete variable. In this case, the forward noise process can be described as
(x++1 (x.) = X,Qe+1:
(59)
and as for the continuous case we have a closed form expression for ((x:(xo) as
g(xe|xo) xoQ.
(60)
where = - --- The backward kernel (x, X Xo) is also known, and as are in the discrete regime, the backward kernel can hence be parametrized by marginalizing over xa as
po (o(xs(Xe+1)
(61)
where xo(xo(xx+1) is categorical distribution predicted by neural network.
Generalizing to D Variables The reasoning above can be extended general D-dimensional data by assuming noise process where the noise is added independently to each the variables, meaning q(-) factorizes over these variables. Similarly, it is also assumed that the backward process pe pe(- (-1x++1) factorizes over the different variables. A single forward pass through the neural networks hence produces probability vectors in D different categorical distributions.
E.2. Measurements
For the discrete setting where we have single variable, we assume that we have a measurement y on the form
p(y xu) = Xn
(62)
For the general D-dimensional case, we hence assume that have D measurements, one for each variable in Xo, and that these are obtained independently from each other.
E.3. D3SMC
In the discrete setting, might be tempting to find an "optimal" proposal
p(xx/xx)/(xxxxxxxx.. 7x(
(63)
p(y(x(+1)
However, is not possible to compute
= E - X0
(64)
in a single evaluation as it requires evaluations of the neural network prior for possible values of x,. If only a single variable is updated at each denoising step, like an order-agnostic autoregressive model (Uria et al., 2014), this could be feasible (Ekström Kelvinius & Lindsten, 2024), but for the D-dimensional case will require dD evaluations (where d is the size of the state space of xx) for D3PM with uniform noise as all D variables are updated each step. Hence, we have created proposal which only requires single forward pass at each sampling step in the general D-dimensional case.
Deriving the D3SMC algorithm is rather straight forward. In the derivation, we will continue using the notation with tilde. However, when we made Gaussian approximations around a reconstruction in the continuous case, we will in the discrete case use the categorical distribution predicted by our neural network, Po x))x
17

---

Decoupled Diffusion Sequential Monte Carlo
Target Distribution In analogy with DDSMC, we write target distribution (x(x))
7x(x,) p(y|x)) ply (11)xx+1)) " (0(0)))6)))))))))1220
(65)
where (x, Xx+1) = Categorical x6, 2x) with xi,44 being the predicted distribution over Xo by the D3PM model (i.e., Po (x( xx-1)),,)))),> )), and p(y[x,) Ex, vly(x))Pu (x x.). The distribution (x(x,, Xx+1) is D3PM analogy to the DAPS
kernel, as we predict and then propagate this forward in time.
Proposal Distribution
The discrete proposal starts, just as for the continuous case, with a posterior over Xo.
p(y ( po (Xu)
Po (xo x)+
(66)
Exo p(yl (e x((1))
yQ O x0.1+1
(67)
Z
where @ means clement-wise multiplication and Ze can be computed by summing the elements in the numerator, With this, we can obtain a proposal distribution in analogy with the continuous case
(x(xx
Σ p(x, x (xx) xx+
Σ Qi ((
(YO7 @ x (xo)T)
(68)
Z--1
F. Protein Structure Completion
Levy et al. (2024) provides a protein toy example where a general observation model
R Nx4x3 is the coordinates of the backbone. They then propose
y Axo + +7, " ~~ N00, E).
(69)
However, the diffusion prior is based on Chroma" (Ingraham et al., 2023), which models the diffusion as
dx 2 3x3,dt + v 3,Rdd,
(70)
which is the regular continuous-time formulations of DDPM/VP process but with correlated noise. Therefore, we can go to a new space z defined as
Rx
(71)
Now, in this space, the diffusion process follows the standard DDPM/VP diffusion process and can any point convert between the two spaces with
-Rz.,
(72)
or in other words, at any point
24 2/220 ~ V(ziVorzz (1 T)
(73)
Going back to the measurement equation, we can write it in terms of the whitened space z as
y A RRz + 17.
(74)
9This includes using pretrained weight parameters, trained by GENERATE BIOMEDICINES, INC. who are the copyright hold- of these weights. The weights have been used with permission according the license https: /hhroma wei ghts. generatebi i c cines. com/
18

---

Decoupled Diffusion Sequential Monte Carlo
Now, Levy al. (2024) make an assumption which is equivalent to the measurement
y = Zz + 1)), îj N0 o la.).
(75)
where de is the dimension of z (which the same as dz). This equivalent to the previous measurement equation with E = UUSTTUT, where U and S are from the SVD of the matrix AR USVT.
Now, to run DDSMC we can rewrite the measurement equation using the aforementioned SVD as
= USV 20 i =
2o S
(76)
the and with change of basis as y' y' 22 A' SS and // = SVT N10, - measurement equation becomes
-((00=+55
A'zy y' A'z + 1f ~~~N(0,
(77)
We can now run DDSMC in this new basis with diagonal matrix A'. One should note here that the covariance do not necessarily have constant diagonal entries, which will slightly change the expressions for DDSMC: instead of relying on single yy. we have to replace this in the detailed expressions with s;a for the dimension i I dy where s; is the corresponding singular value Siz.
G. Additional Experimental Details and Results
G.1. Gaussian Mixture Model
G.1.1. PRIOR AND POSTERIOR
For the GMM experiments, we tried following MCGDiff as closely as possible, and therefore used and modified code from the MCGDiff repository/0. 10, We briefly outline the setup here, but refer to their paper for the full details.
The data prior is a Gaussian Mixture with 25 components, each having unit covariances. Under the DDPM/VP model (see Appendix B), the intermediate marginal distributions g ((x)) = (x))(()))xx)ddxx)) will also be a mixture of Gaussians, and it is hence possible to compute Vx, log (xx) exactly.
The sequence of Be is a linearly decreasing sequence between 0.02 and 10-4.
The mixture weights and the measurement model (A, oy) are randomly generated, and measurement is then drawn by sampling first x (xx)), then NO0 1), and finally setting Ax* oye. From this measurement, it is possible to compute the posterior exactly, and 10k samples were drawn from the posterior. Additionally, 10k samples were generated by each of the algorithms, and the Sliced Wassertstein Distance between the exact posterior samples and the SMC samples were computed using the Python Optimal Transport (POT) package" (Flamary et al., 2021)
G. 1.2. IMPLEMENTATION DETAILS
For the three SMC methods, we used 256 particles for all algorithms, and 20 DDIM timesteps (Song et al., 2021a). In this case, the timesteps were chosen using the method described in MCGDiff.
For DDSMC and DAPS, we used the DDIM ODE solver (see equation below), and solve the ODE for the "remaining steps" in the diffusion process. This means that, at diffusion time 6, we start with a sample X Xx and then update this sample according to the update rule
ap-Lo Xy + av))
ār-1 - V(1- - āy -v-1 ((((-)ar)))))))))))){()) av)) Vx. (xx)(xx)) Cr.
X
V
(78)
Or.
for t' = t,t ,,...,1. 1.
DDSMC
We use
ff
"https://pythhontt.giihbb..
19

---

Decoupled Diffusion Sequential Monte Carlo
Table Comparison of DDRM with 20 1000 steps in the GMM setting.
de
dg
DDRM-20
DDRM-1 000
5.22 + 0.92
3.83 1: 1.05
4.58 + 1.09
2.25 +: 0.92
2.05 + 0.85
0.55 0.29
6.32 + 1.24
5.19 + 1.07
7.60 + 0.90
5.62 1.09
7.74 + 0.84 4
4.95 + 1.25
7.02 + 1.14
7.15 1.11
800
2
8.29 + 0.86
8.21 + 0.88
4
9.07 + 0.73
8.66 + 0.87
MCGDiff
We used code from the official implementation 12, per their paper, we used 10-2
TDS We implemented the method ourselves according to their Algorithm I, making the adaptions for the DDPM/VP setting (also described in their Appendix A.1).
DAPS We implemented the method ourselves. As we in the linuar-Gaussian setting, we replaced their approximate sampling of p( ((xp) y) with the exact expression (given by Equations (16) and (17)). We used the same schedule for r. as for our Pt.
DDRM We used code from the official repository and used the same hyper-parameter 7 and as in their paper. We used 1 000 timesteps to avoid discretization error, but also tested 20 steps as it is argued in their paper that more is not necessarily better. This performed worse for d = 8 and 80, and similar for d₂ = 800, see Table 4.
3, K = 2, and y = 10-2 as specified in their paper. We DCPS We used code from the official repository 14 We used L. -= 3,
used 1 100 timesteps.
1.1.3. COMPLEXITY AND NUMBER FUNCTION EVALUATIONS
As DDSMC relies on multiple particles in parallel, will inevitably require larger computational effort. DDSMC-Tweedie uses one score evaluation per partiele per diffusion step, thus requiring N times more score evaluations per step compared to method like DDRM (with N being the number of particles). This is the same effort MCGDiff, and slightly less than TDS which requires differentiating the score function. DDSMC-ODE and DAPS uses additional steps for the reconstruction function, and DDSMC-ODE hence requires N times more score evaluations per diffusion step than DAPS, which in turn requires 'number of ODE steps" times more score evaluations than a method like DDRM. In the case of the GMM setting, we used 20 diffusion steps, and reconstruction required 20, 19,..., steps, meaning total it required score evaluations just for reconstruction. As noted in the paper, using DDSMC-Tweedie already outperforms all other methods (including MCGDiff, which has the same computational footprint). We investigate further the spending of the computational effort in DDSMC-ODE in Appendix G.1.5.
times more
G. 1.4. DDSMC-ODE WITH SINGLE PARTICLE (COMPARISON WITH DAPS)
As the proposal of DDSMC-ODE with = 0 essentially corresponds to DAPS (if instead using X² =² in Equation (19)), running this setup with a single particle would become essentially equivalent to DAPS. We verify that this also holds empirically Table 5, where we present the numbers for DDSMC-ODE with a single particle. Indeed, the numbers for 7 are more or less identical.
Pnttps: http://///ggttoooooo////aa////// diff phtttps///ge thub. com www.baahatt-kaaarr//aaoaa "https: /tatps://gii thub. www.com////aaaa thut..ccom/Aaar--ooouA/////ggg
20

---

Decoupled Diffusion Sequential Monte Carlo
Table Results on the Gaussian mixture model experiment, using DDSMC-ODE with 20 ODE steps and N 1 particles, i.e., sampling from the proposal. The numbers for 0 are more or less identical to DAPS, verifying that our proposal and their method are essentially equivalent also practice.
dz
dy
17
0.0
17 = 0.5
7) == 1.0
1
5.62 + 0.90
5.85 0.93
6.26 + 0.96
8
2
5.93 + 1.16
6.33
6.92 1.00
4.88 + 1.34
5.42
6.16 + 1.16
6.84 1.16
1.15
7.49 1.10
80
2
8.50 0.92
0.90
9.22 0.90
9.05 + 0.75 4
0.71
9.78 + 0.68
1
7.03 + 1.20
7.13
7.26 1.17
800
2
8.30
= 1.01
8.41 1.02
8.57 + 1.01
4
9.20
= 0.83
9.33
0.84
9.50 + 0.83
Table Results the Gaussian mixture model experiment when using DDSMC-ODE, rj O, with fewer ODE steps and/or fewer particles N. "This setup requires as many score evaluations as DDSMC-Tweedie with N = 256 particles.
ODE STEPS =
ODE STEPS =
ODE STEPS 20
dx
dy
N 90
N 256
N
N 256
N 12
N 25"
N 256
1.89 + 0.57
1.79 + 0.52
1.68 4: 0.50
1.49 + 0.40
0.98 0.29
0.94 0.20
1.15 + 0.24
8
0.63 0.30
0.63 0.30
0.55
0.27
0.53 + 0.26
0.57 0.25
0.48 0.18
0.56 0.23
0.16 - 0.01
0.16 0.01
0,17 0.02
0.17 0.04
0.18 0.04
0.19 1.19:0.0 0.07
0.21 0.10
1.22 + 0.36
1.20 + 0.36
1.14 0.34
1.11 + 0.33
0.33
0.71 0.16
0.70 + 0.17
0.69 0.17
80
1.15 + 0.42
1.08 + 0.41
1.09 + 0.48
0.97 0.42
0.88 + 0.57
0.70 = 0.39
0.69 0.36
4
0.53 + 0.28
0.55 + 0.30
0.59 + 0.29
0.55 + 0.30
0.71 + 0.45
0.40 = 0.19
0.41 + 0.23
1.69 + 0.54
1.66 + 0.53
1.65 + 0.53
1.56 + 0.49
1.56 + 0.49
1.39 + 0.47
1.37 + 0.43
800
2
1.24 1 0,64
1.08 1 0.51
1.42 1 0.95
1.07 1 0.45
1.64 1 0.86
1.23 - 0.78
0.81 + 0.40
0.38
1 0.27
0.25 1 0.08
0.50 1 0.40
0.24 1 0.08
1.18 0.82
0.70 0.60
0.23 0.06
Table Results on the Gaussian mixture model experiment when using DDSMC-ODE, 1 = 0.5, with fewer ODE steps and/or fewer particles N. "This sctup requires as many score evaluations as DDSMC-Tweede with N = 256 particles.
ODE STEPS =
ODE STEPS =
ODE STEPS 20
de
dy
N
90"
N 256
N 40"
N 256
N 12
N= 25"
256
1.92 + 0.73
1.84 + 0.64
1.49 0.52
1.33 0.41
0.70 0.25
0.69 0.25
0.88 0.33
8
0.62 + 0.36
0.61 + 0.35
0.44 + 0.29
0.45 + 0.29
0.44 0.23
0.30 0.20
0.34 + 0.20
4
0.08 + 0.03
0.09 + 0.04
0.05 1 0.02
0.05 0.001
0.07 + 0.05
0.07 1 0.06
0.09 1 0.08
1.01 + 0.33
1.01 + 0.34
0.96 + 0.29
0.92 0.30
0.64 + 0.14
0.53 - 0.14
0.50 + 0.16
80
2
0.91 + 0.39
0.85 + 0.40
0.95
0.77 + 0.36
0.91 + 0.50
0.57 - 0.27
0.37 + 0.20
4
0.38 1 0.25
0.38 1 0.27
0.43
0.36 0.24
0.58 + 0.33
0.31 + 0.14
0.21 + 0.13
1.81 + 0.63
1.76 + 0.60
2.01 + 0.65
1.82 + 0.57
2.20 + 0.63
1.92 - 0.59
1.68 + 0.52
800
1.51 0.97
1.27 0.78
1.96 - 1.11
0.77
2.38 1.00
1.94 0.97
1.17 = 0.65
0.49 + 0.33
0.33 + 0.15
0.77 0.45
0.38 + 0.16
1.70 0.81
1.10 - 0.66
0.47 0.22
1.5. DDSMC-ODE WITH FEWER PARTICLES AND/OR ODE STEPS
As DDSMC-ODE requires additional score evaluations when solving the PF-ODE, we present some results in Tables 6 to 8 where we have used fewer particles and/or fewer ODE steps. Notice that the number of ODE steps is the maximum number of ODE steps, and if there are fewer steps left in the (outer) diffusion process, this lower number is used as number of ODE steps that iteration. Hence, as an example, with the maximum number of ODE steps being 20 and the number of diffusion steps also being 20, the number of ODE steps used in the different iterations would be 20, 19, 19,..., and
21

---

Decoupled Diffusion Sequential Monte Carlo
Table Results the Gaussian mixture model experiment when using DDSMC-ODE, r) 1, with fewer ODE steps and/or fewer particles N. "This setup requires as many score evaluations as DDSMC-Tweedis with N = 256 particles.
ODE STEPS =
ODE STEPS 7
ODE STEPS = 20
de
dy
N
90'
N
N
N 256
N 12
25
256
2.07 0.81
1.99 0.71
1.44 0.51
1.30 0.38
0.85 0.25
0.76 0.25
1.01 0.39
8
0.75 + 0.39
0.69 = 0.38
0.57
0.50 + 0.29
0.70 - 0.22
0.48 =
0.39 =
0.14 0.06
0.13 + 0.06
0.19 : 0.03
0.14 0.02
0.29 + 0.05
0.23 0.05
0.17 + 0.06
1.14 0.24
1.10 + 0.23
0.27
1.06 0.21
1.11 0.14
0.91
0.77
80
2
1.08 + 0.36
1.02 = 0.34
1.18 0.35
0.97 0.30
1.39 - 0.46
0.26
0.69 0.15
0.69 + 0.16
0.63 + 0.17
0.79
0.67 +.15
1.01 + 0.27
0.54 0.15
2.17 + 0.62
2.09 - 0.58
0.64
2,21 + 0.55
2.67 + 0.64
0.51 0.60
800
2
2.05 + 0.95
1.81 0.81
2.56 1.08
1.90 + 0.79
2.96 + 0.99
1.76 2.92
1.18 0.48
1.04 + 0.37
1.50 0.54
1.15 + 0.39
2.38 0.80
1.81 0.70
1.24
DDSMC-ODE would in this case use ~ 10x more score evaluations in total. Therefore, using N = 25 would corresponding to approximately the same number of score evaluations as DDSMC-Tweedie.
In the tables, a general trend seems to be that more ODE steps are more important than more particles under a compute budget, as 20 ODE steps with 25 particles seems to perform better than and 7 ODE steps with 90 and 40 particles, respectively. It also in general performs better than DDSMC-Tweedee (which in principle uses single ODE step). This could lead the conclusion that one should computational budget. However, running single particle does not work well, and it hence seems like both aspects (more ODE-steps and more particles) are needed for optimal performance.
many steps possible, and then add more particles up to the desired
should be noted that we have used number of score evaluations as measure of computational effort, and as mentioned ahove in the discussion on complexity, since particles are run parallel, adding more particles might not have the same effect on computational time as adding more steps (if keeping the number of score evaluations equal). As more particles also improves performance, exactly how to tune the number of steps and particles can depend also on possibility for parallelization.
G. 1.6. ADDITIONAL QUALITATIVE RESULTS
In Figures 6 to 8 we present additional qualitative results to compare DDSMC with different and reconstruction functions. In Figures 9 to 11, we present extended qualitative comparisons with other methods.
22

---

Decoupled Diffusion Sequential Monte Carlo
== 0.0
y =
1.0
4 1.0
Figure 6. Extended qualitative results on GMM experiments, comparing DDSMC with Tweedie (three left-most figures) or PF-ODE as reconstruction, d, dy
2t
1)
11- 1.0
10 1)
Figure 7. Extended qualitative results on GMM experiments, comparing DDSMC with Tweedie (three left-most figures) or PF-ODE as reconstruction, d, 80, d, 1.
23

---

Decoupled Diffusion Sequential Monte Carlo
?
?
10
Figure 8. Extended qualitative results on GMM experiments, comparing DDSMC with Tweedie (three left-most figures) or PF-ODE as reconstruction, d, 800, dy
2.
Tq
Figure Extended qualitative results GMM experiments, 8, d, 1. DDSMC using PF-ODE reconstruction r 0.
24

---

Decoupled Diffusion Sequential Monte Carlo
Figure Extended qualitative results on GMM experiments, de 80, dy = 1. DDSMC using PF-ODE as reconstruction and 17 = 0.
I1
Figure Extended qualitative results on GMM experiments, d. 800, d, DDSMC using PF-ODE as reconstruction and rj =
25

---

Decoupled Diffusion Sequential Monte Carlo
G.2. Image restoration
We perform the image restoration on the FFHQ dataset (Karras al., 2019), downsampled from 1024 1024 to 256 256 pixels.
G.2.1. DDSMC IMPLEMENTATION DETAILS AND HYPERPARAMETERS
For the image restoration tasks, we used N = 5 particles. We also follow DAPS and take on the view of diffusion as continuous process, and the backward process as a discretization of this process at timesteps 4, 12,..., ,r, With this, and t t+ in Algorithm are replaced by f, and f241, respectively.
Time and o, Schedule chosen as
We follow DAPS which uses noise-schedule dr. = t where the timepoints t; (i = 1,...,
,....,777 T) are
&
1//7 mix
T
1/77 min
$1/7 "max
))
7
(79)
T
used mmx 100, min = 0.1 and 200
P: schedule In their public implementation 15, DAPS uses = a. However, we find that for when using the exact solution these values are too high. We therefore lower Pr by using similar schedule as for dz.
T
(PMIT
P1/7 Piixx
)
Pt.
Panxx +
(80)
but with with Pmax 50 and Pmin 0.03.
PF-ODE Again, we followed DAPS and used an Euler solver to solve the PF-ODE by Karras et al. (2022), using steps with according the procedure for the outer diffusion process (i.e., Equation (79)), each step 1 starting with fuux f and ending with mmin
0.01.
G.2.2. IMPLEMENTATION OF OTHER METHODS
For the other methods, we used the DCPS codebase 16 as basis where we implemented DDSMC and transferred DAPS.
DAPS We used the hyperparameters as in their public repository!?,
DCPS We used the settings from their public repository, running the algorithm for 300 steps that was what was specified in their repository.
Just as for DCPS, we used 300 steps. We also tried 20 steps stated in the original paper, but found that this worked worse (see Tables 9 and 10).
DDRM
MCGDiff We used 32 particles.
G.2.3. FULL QUANTITATIVE RESULTS
In Table 9 we present the LPIPS values with standard deviations, and include DDRM with both 20 and 300 steps. In Table 10 we present PSNR values.
G.2.4. FFHQ IMAGES ATTRIBUTIONS
In Table 11, attribution information for the images from FFHQ displayed this paper are provided.
Petttss//githann http:l.ccc/eeaaaaaeeee http://hhh..comm/aaaaagaaaaaaaap hwm///aaanggiinggaagggggg/gggg
"https: /httts://giiee thub. com com//Hadh--MOUAn///ggpp/
Whtpps: /tatps://git thub. htt.://coo//.iigggggniiiggggggg
26

---

Decoupled Diffusion Sequential Monte Carlo
Table LPIPS results on FFHQ experiments. The noise is By 0.05, the tasks are inpainting a the middle, outpainting right half of the image, and super-ress resolution 4x. Numbers are averages and standard deviations over 1k images. DDRM-20 and DDRM- means DDRM with 20 and 300 steps respectively. Lower better.
MIDDLE
HALF
SR4
DDRM -20
0.08 + 0.02
0.29 = 0.05
0.2 0.05
DDRM-300
0.04
0.25 0.05
0.2
DCPS
0.03
0.05
0.03
DAPS
0.05 0.02
0.24 0.05
0.05
MCGDIFF
0.1 0.00
0.34 + 0.06
0.15
DDSMC-0.0
0.07 0.02
0.26 - 0.05
0.27 0.07
TWEEDIE
DDSMC-0.5
0.07 0.02
0.27 = 0.05
0.2 0.05
DDSMC-1.0
0.05
0.24 0.04
0.14
DDSMC-0.0
0.05
0.23 1 0.05
0.21 0.06
ODE
DDSMC-0.5
0.05 0.01
0.23 = 0.05
0.15 0.05
DDSMC-100
0.08 + 0.03
0.4 = 0.05
0.36 0.08
Table PSNR results on FFH experiments. The noise level σy = 0.05, and tasks are inpainting box middle, outpainting right half of the image, and super-resolute reserreeooution 4x. Numbers averages and standard deviations over 1k images. DDRM-20 and DDRM-300 means DDRM with 20 and 300 steps respectively. Higher better.
MIDDLE
HALF
SR4
DDRM-20
29.14 + 2.16
16.19 + 2.51
28.47 + 2.0
DDRM-300
30.68 2.37
16.85 3.21
29.14
DCPS
29.74
16.18
1 2.77
26.52
1.94
DAPS
29.66 2.06
16.18 1 2.75
28.28 1 1.88
MCGDIFF
27.49 1.96
14.39
2.26
27.76 1 2.02
DDSMC-0.0
30.09 2.07
15.86 = 3.1
27.28 + 1.85
TWEEDIE
DDSMC-0.5
29.18 2.15
15.78 + 2.98
27.07 + 1.69
DDSMC-1.0
27.75 2.35
14.46 + 2.39
26.59 1.97
DDSMC-0.0
29.68 + 2.1
15.12 + 2.96
27.51 + 1.84
ODE
DDSMC-0.5
28.85 2.16
15.41 + 2.99
26.81 + 1.86
DDSMC-100
25.47 2.27
11.87 + 1.64
21.41
G.2.5. ADDITIONAL QUALITATIVE RESULTS
We provide additional results the image reconstruction task in Figures 12 14.
27

---

Decoupled Diffusion Sequential Monte Carlo
Table 11, Attribution information for the FFHQ images displayed ground truth in this paper. Images have heen preprocessed (e-g-, aligned and cropped) in the original curation of the FFHQ dataset. We then downsampled these to 256 256 for the experiments. and further downsampled for this paper. More details can found the metadata the FFHQ dataset: NVlabs/ ffhq- dataset.
con/
IMAGE ID
AUTHOR
ORIGINAL URL
LICENSE URL
100
DEBBIE GALANT
HTTPS:/ wWW.RRR.IIC.....
HTTE://CRATTVECOMON 0KG/LICEN88S/8X//2.00
COM/PHOTOS SAB.TTOS/EBGALANC// 25236485093/
102
TENACIOUSME
HCTPS://NW..LLCCKK PROIOS/EENACIOUSNE 4674489614/
#TS::/CCAATVCCMMOS
0KC/LIcSNSes/ex/2.00
103
ALPHALAB STARTUP AC-
ACTPS://WW..FLIC /
RTTS://CRATVVEOMMNNS
CELERATOR
PHOTOS/ALLAAILA//B8222266
CsC//LICESS3//B/22.0/
104
SALESIANO SAN JOsÉ
ATTPS://OW..F1.00R.cco
RTPS://CRRATTVRROOMWN
PHOTOS/970735770004/ 9813413673/
ORG//ICCRNS5/B//2200/
110
CAPTURED BY JOLENE
HTTPS:
PTTPS://NWW.FIICRR
NTTS:://GRATTVE000005
C.P. HOANG
#ROTOS/HOANGTOHANV//
0S//LICEN555/////2.00
38813884045/
111
ARCHIDIECEZIA
HTTPS://MW.FFLICRRCC
HTB::/CCBAA118COMO0N..
KRAKOWSKA BIURO
PRASOWE
40877014205/
126
GHOLZER
HCTPS://WW..FLCCKE
RTB::/CRAATVECCMMNN
PHOTOS/GEORGHOLZEVV 4612924700/
135
RESOLUTESSUPORRMMBIA
HITPS:/
HIZPS://WWW.ELICKK
cON/PHOTOS//SAEMEDE.. 4833372671/
0KG/LICEN885//B//2.00/
137
BERENDS PRODUCTIES
HTTP:://RW..LICC...OM
#TT::/CRRA1IM4CCMM05
#H0TOS/11823838488077 39503180405/
0x6/L11G2888//84-98///.000
146
ELI SAGOR
HTTP::/NWW..LICCR..CO
RTTS:/CCR2X1MCC00005
147
INNOTECH SUMMIT
aTTP::/NNw..LLICee.co.
#TTS://CRATTVEOOOONS
PHOTOS/115363558N022 18076371188/
0sc/LICCNSSs//x/22.00
151
HAMNeR_FOTOS
ATTPS://ww..ft:
RTTS://CRRATVRCCOMOG
PROTO//DDVATHAK_AANRR 3765961686/
ORG/AICCNSRS////2.00/
171
CHOU
MTTP:://Nw..FITCRR.coM
RTTS://CRAATTMCOOMOSS.
ORG/LIIGESS5//YY-0////000/
185
OSSEOUS
NTTS://CBAA1VCCOMOS..
ORG/LICENSES/BX/22.00
189
PHIL WHITEHOUSE
BTPS://ww..LLCCRc.CO//
nTTS:/CUAAAVCCCMME..
HHoTO//EHIIL.TECAAAALAAAC
ORG/LIC3N865/////2.00/
3698708497/
190
UNIVERSITY OF THE
HCTPS:/WWW.FLICCR.CC
HT#S://CRE2111000000...
FRASER VALLEY
PI0TOS/UPV/26175.3.113
DRG/LICENSDS/B//2.00
199
AMISOM PUBLIC INFOR-
HTTPS://Ww..f.TTORR
HTTS://CRAATVRRCCOOOO
MATION
PHOTOS/AU_UNISE DI0DOS/AAALO011122000CTRRAAAAA
02d/200L11000111/2B00001
402/2370250/
28

---

Decoupled Diffusion Sequential Monte Carlo
OCONC
Twwedle
DOANC-OOE
MONDter
Figure 12. Additional results the outpainting task. The DDSMC samples are ordered with 0 the left, 0.5 in the middle, and
? = 1 to the right. Ground truth images are FFHQ image IDs 102, 104, 199, 137, and 189 (see Table for attribution).
Weaf
Z05 Tweetle
CEINC-00E -DE
DPPS
3GRM
TAPS
HOGDLEE
Figure 13.. Additional results the inpainting task. The DDSMC samples ordered with 17 0 to the left, 0.5 in the middle, and
y = 1 to the right. Ground truth images are FFHQ image IDs 151, 171, 135, 110, and 126, (see Table 11 for attribution).
29

---

Decoupled Diffusion Sequential Monte Carlo
OCGNC Tomedlle
DDANS-OOR
DCP2
3CRM
TAPS
MIGOO
SGIMEL.
OSANGEL
ORNEER
OSAQUE
GENNELS
OGINDEL
DERIEL
SLANDE
OSSNNES
OSANBEL
OLRVEL
CANEEL
Figure 14, Additional results on the super- resolutin (4x) ) task. The DDSMC samples ordered with 27 0 to the left, 17 0.5 in the middle, and y = 1 to the right. Ground truth images are FFHQ image IDs 185, 111, 190, 147, 146 (sce Table 11 for attribution).
30

---

Decoupled Diffusion Sequential Monte Carlo
G.3. Protein Structure Completion
G.3.1. IMPLEMENTATION DETAILS
We implement DDSMC in the ADP-3D codebase 18 and used DDSMC-Tweedie with 7) We otherwise follow the ADP-3D experimental setup, but also extend the experiments include non-zero noise levels. For ADP-3D, had to lower the learning rate for the higher noise levels, using 0.1 and 0.001 for 0.1 and 0.5, respectively, We run the algorithms with 8 different seeds. To somewhat compensate for the additional computations when running DDSMC, we use 200 steps instead of I 000 that are used in ADP3D. As the diffusion process in z follows the DDPM/VP formulation, we use the same choice of Pt as in the GMM setting.
G.3.2. EXTENDED RESULTS
In Figure 15, we present results on four additional proteins with PDB identifiers 7pzt (Dix et al., 2022), 7r5b (Warstat al., 2023; Huegle, 2023), 7rea (Cruz al., 2022; Stogios al., 2022), and 8em5 (Cuthhert al., 2024; Cuthbert & Goulding, 2023). In general, designed for this problem, and that the SMC aspect with parallel particles and resampling improves performance.
the same trend that DDSMC is competitive with ADP-3D, which specifically was
com.ax..omm/eeeeee adp-
31

---

Decoupled Diffusion Sequential Monte Carlo
ADP3D
AD930
DD6MC-1
DOSMC-1
DDSMC 5
DOSMCS
DOSMC
18
DDSMC-100
DOSMC-100
107
FONAD
ADF3D
DDSMC-1
x 101
DOSMC
188
D0SMC 100
Subssmping
Subsampting
Subssmpling
(a) 79zt
ADF3D
ADF3D
DDSMCC11
DDSMC4
101
DISMMC-10
DDSMC 300
DDSMC-100
RM53 (A)
RW50 1A
ADF30
D0SMC-1
10
D0SMC-100
LUSMC-S0 50
16
32
32
16
32
Subsampling factork
Subsampling factork
Subsampling factor
(b) 7r5b
ADP3D
15
DUSMC-10
DOSMC-50 50
D055C-100
RM150 (A)
IMSO (A)
(A)
D0SMC-5
DOSMC-5
305MC-10
DDSMC-10
DOSMC 50
3
DOSMC 50
DDSMC110
DM
1.6
16
16
32
Subsemping factork
Subssmping factork
Scbsumpling factore
(c) 7roa
10a
BRSMC-S
DDSMC 50
DDSMC-10
10
DDSNC-100
RMSD (4)
AM
ANSD 5A)
0x
ADP3D
ADF10
DDSNC1
D0SMC
DDSMC5
DOSMC
DD
3x.30
D0SMCC 10
50
DOSMC
10-1
16
Subsampling factor
Subsamsling factor k
Subsampling factor
(d) 8em5
Figure 15. Additional results on the protein structure completion problem in Section 5.3. RMSD subsampling factor & for additional protein with different noise levels a: (left), (middle), 0.5 (right). Comparing ADP-3D (Levy al., 2024) DDSMC with different number of particles.
32

---

Decoupled Diffusion Sequential Monte Carlo
72/09919439069011977
2
1
501597344
/
0 00159955
Figure Extended qualitative results on binary MNIST using D3SMC with N particles. Top row ground truth, second row the measurement, and the bottom 10 rows are samples from different independent SMC chains.
G.4. Binary MNIST
In Figure 16 and 17 we provide extended results when running D3SMC binary MNIST, using the same measurement model as in the main text and N = 5 or N 100 particles, respectively.
33

---

Decoupled Diffusion Sequential Monte Carlo
72/09991590090119771
2
7
01100110000000000000
9
(44470044))522
2 0
009442990040057733
7 2 00/09/42290000069733
Figure Extended qualitative results on binary MNIST using D3SMC with N 100 particles. Top row ground truth, second row
the measurement, and the bottom 10 rows are samples from different independent SMC chains.
34


Reviews:
[{"review_id": "rxwQfoHZay", "reviewer": "Reviewer_psuz", "summary": "The paper introduces Decoupled Diffusion Sequential Monte Carlo (DDSMC), a method for Bayesian inverse problems using diffusion priors. Main contributions include: Leveraging a modified diffusion process (\"DAPS prior\") to enable larger updates during sampling, improving exploration. Combining SMC with diffusion models to provide asymptotically exact posterior sampling, addressing limitations of prior methods that rely on approximations. Extending the approach to discrete data (D3SMC) via discrete diffusion models (D3PM).\n\n## update after rebuttal\n\nBased on the authors' rebuttals, and also the reviews of other reviewers, I would like to maintain my original rating.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to comment on our paper. We have tried to address all your comments below, but there were a few points that we did not quite understand, so we kindly ask you to clarify if we in fact misunderstood some of your points.\n\n## Merging SMC with decoupled diffusion is our core contribution\nAs the reviewer comments _\"algorithms that merge SMC with decoupled diffusion in an innovative manned could have strengthened the paper\"_, we want to highlight that this is exactly the core contribution that we make in our paper! As far as we are aware, this is the first time that SMC is merged with decoupled diffusion, and we are therefore not sure what this comment refers to. We are happy to answer any follow up clarifications on this.\n\nThe reviewer comments that _\"The proposed DDSMC method reduces to existing methods in extreme cases (e.g., inverse temperature eta = 0 and using PF-ODE for reconstruction).\"_  We do not agree with this claim since it misses the point that our core contribution is an **SMC algorithm** building on the (generalized) DAPS prior. Hence, this claim would only be correct if we also restrict DDSMC to using a single particle, but the \"parallel particles\" are a key ingredient in SMC. Running an SMC algorithm with a single particle corresponds to sampling from the proposal, which if using $\\eta=0.0$ and PF-ODE would be more or less equivalent to DAPS (if instead using $\\lambda_t^2=\\sigma_t^2$ in eq (16)). To verify that, we ran DDSMC-ODE on the GMM case with a single particle, and the numbers obtained with $\\eta=0.0$ are essentially identical to DAPS (differing by at most 0.03 from each other). Hence, we can conclude that the **introduction of the SMC framework is the key ingredient that leads to the improved performance over DAPS**. We will incorporate these results in the appendix, and make a comment in the experiment section in the main paper. Next, we do agree that a further analysis of the number of particles would be beneficial, and as part of the response to XuMA, we performed additional experiments with fewer particles using the DDSMC-ODE. \n\n\n## We target posterior sampling, which is verified in the GMM task\nWe agree that our experiments largely depend on synthetic experiments. However, we believe that this is a necessary sanity check for a method such as DDSMC which is designed to target the __correct posterior distribution__, intended to empirically prove that our proposed technique samples from the true posterior.\nIn order to achieve this, having an exact score/ground truth posterior is necessary to control errors.\n\nThe purpose of the image experiments is to first show *qualitatively* that our method works also in high-dimensional, real-world problems, and we included the LPIPS metric to also show this quantitatively: we are in line with SOTA methods, and outperform MCGDiff, which is the *closest comparable* method to DDSMC. \n\nWe emphasize that our focus is on recovering the posterior distribution, and we do not aim to improve the per-sample quality like DDRM, but instead, the population quality. However, a problem is that most image-related tasks lack the ground truth posterior, and even a good approximation thereof. To our knowledge, there are no principled metrics to gauge such performance. As such, the synthetic experiment is the only setting we can rely on to compare the methods' posterior sampling ability in a principled way. We revise our Experiment section to reflect on this.\n\n## We have evaluated DDSMC on protein structure determination\nThe reviewer comments that _\"The main limitation lies in the experiments.\"_ In the response to Reviewer XuMA we have added results for another experiment concerning protein structure determination. We show that DDSMC can outperform the tailored APD-3D method out-of-the-box in (realistic) high-noise scenarios.\nPlease see response to XuMA for further details.\n\n## We can see clear effects of changing the inverse temperature $\\eta$\nAs mentioned above, we view the DDSMC method itself as our primary contribution, and the _generalization of the DAPS prior_ (such that we can interpolate between DAPS and the standard diffusion prior using the inverse temperature $\\eta$), as a secondary contribution. The reviewer comments that we should demonstrate \"how varying $\\eta$ affects the result and that an intermediate value between 0 and 1 offers better performance\". We agree with the observation that $\\eta=0.0$ is the best in high dimensions for DDSMC-ODE, but we emphasize that $\\eta=0.5$ is better for DDSMC-Tweedie in this case, and is also the best choice in lower dimensions for DDSMC-ODE, while $\\eta=1.0$ seems to be better for DDSMC-Tweedie in lower dimensions. It hence certainly seems to be an interplay between the dimension of the data, the choice of reconstruction function, and the choice of $\\eta$. We already discuss this in line 304 col 2 and onward, but will extend the discussion in a revised version of the paper.", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A. No theoretical results in this submission (I did not check the supplementary material).", "experimental_designs_or_analyses": "- The contribution appears incremental when compared to existing methods such as DAPS and the experimental results do not fully demonstrate a clear advantage over current methods. Overall, the novelty and significance of the work may not be sufficient for acceptance at ICML.\n - The main limitation lies in the experiments. A large portion of the results relies on synthetic Gaussian mixture models. Since Gaussian mixture models provide ground truth score information, the results based on them do not capture the challenges encountered by existing diffusion model-based methods in real-world applications. For the real-world FFHQ dataset, experiments in inpainting, outpainting, and super-resolution yield performance of DDSMC that is not clearly superior to other methods. For example, DCPS ranks first in all tasks in Table 3, casting doubt on the practical benefits of the proposed DDSMC method.\n- The proposed DDSMC method reduces to existing methods in extreme cases (e.g., inverse temperature eta = 0 and using PF-ODE for reconstruction). The paper should demonstrate how varying eta affects the results and that an intermediate value between 0 and 1 offers better performance. However, Table 1 shows that with 800 dimensions for x, eta = 0 yields the best performance, and Table 3 reveals similar results for eta = 0 and eta = 0.5. The results presented do not effectively support the claimed benefits of the inverse temperature. Additionally, the experiments omit analysis on the number of particles for SMC. Since other methods run only once, DDSMC appears to gain an unfair advantage, particularly against methods that also involve random sampling. Experiments using a single particle for DDSMC and multiple runs for existing methods are necessary.\n\nAlgorithms that merge SMC with decoupled diffusion in an innovative manner could have strengthened the paper. Using SMC to increase the number of samples for approximation and employing the inverse temperature eta might enhance performance. However, based on the experimental discussions above, the overall advantage of the proposed method appears limited. Because of this, I do not recommend acceptance.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U2JbMVPk0Q", "reviewer": "Reviewer_tiGx", "summary": "The authors consider solving linear inverse problems with diffusion models, but only those where the forward model has a tractable SVD.  They build on the recent decoupled annealed posterior sampling (DAPS) method by Zhang (2024) by replacing its Langevin sampling inner-loop with a sequential Monte-Carlo (SMC) sampler.  They claim that this improves posterior sampling performance and present numerical experiments with low-dimensional synthetic GMM data and high-dimensional image data.  Their method differs from several recent works on diffusion-SMC in the details of the intermediate targets and proposal function.\n\n## update after rebuttal\n\nI appreciate the clarifications given in the authors' response, but I still have concerns about the readability of the paper, since a lot of effort will be required to make it clear and accessible. If you look closely at my review, there are a number of unanswered questions, and responses to other reviewers suggest that there are many typos. Regarding the existence of linear inverse problems without implementable SVDs, there are many, including motion deblurring, multi-coil MRI, computed tomography, any i.i.d. random forward operator like those popular in compressive sensing, etc. With these issues in mind, I am leaving my score as-is.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments, and we address their concerns below (see answer to zGWq regarding complexity and NFEs).\n\n## We will clarify the asymptotical exactness guarantees\nSMC provides consistent approximations of its sequence of targets $\\lbrace \\pi_t(x_{t:T})\\rbrace_{t=0}^T$ under weak conditions. However, since we only care about the final target, it is enough that $\\pi_t(x_{0:T})$ admits the true posterior $p_\\theta(x_0|y)$ as a marginal for the method to be consistent. This is the case for DDSMC by construction; see Eq (7) with $t=0$. The _intermediate targets_ $(\\pi_t(x_{t:T}), t>0)$ as well as the _proposals_ can be seen as design choices that affect the efficiency of the algorithm but not its consistency. The particular ways in which we design these quantities (while ensuring correctness of the _final target_) constitute the DDSMC framework.\nWe mention this general fact in line 92 column 2 regarding the intermediate targets but will add a comment regarding the proposal. \nWe note that the specific questions that you ask regarding approximations in Eq (13), (16), (68) _only affect the intermediate targets and/or proposal_ and thus not the consistency of the final target approximation.\n\nWe realize that this should be further clarified, and we will hence add a \"theorem-like\" paragraph, making it clear how our algorithm will be asymptomatically exact. See also response to XuMA.\n## We view reconstruction function $f_\\theta$ as a design choice\nIn our experiments, we tried two types of reconstruction functions: either using Tweedie's formula (see line 87 col 1) or the PF-ODE. As we view this as a design choice, we have not explicitly written how $f_\\theta$ is computed in Algo 1. It is true, however, that if using the PF-ODE, this requires an inner loop. In eq (66) there is a typo, where $t$ should be replaced by $t'$ (the inner loop time variable), and in line 323 we mean that we start at $x_{t'} = x_t$, then use Eq 66 for $t'=t-1, t-2, \\dots, 0$. I.e., we start from a sample at the diffusion time (outer loop index) $t$, then convert that into a sample $x_0$ using as many steps in the inner loop as there are \"left\" in the outer loop. We will fix these typos and clarify line 323.\n\nThis flexibility also affects the computational cost/NFE, where the Tweedie reconstruction avoids the outer loop and is therefore more efficient. Compared to MCGDiff and TDS, DDSMC-Tweedie has the same number of NFE (but avoids the expensive differentiation of the reconstruction network in TDS), but can still provide better result. The DDSMC framework is general and choice between Tweedie and PF-ODE (or some other reconstruction method) becomes a computational trade-off for the user.\n## While falling under the same framework as TDS and MCGDiff, we provide better empirical performance\nBoth TDS and MCGDiff are SMC methods, and we differ in how we design both the proposal and targets, which empirically shows improved performance. Additionally, we make a secondary contribution in the choice of prior (using $\\eta\\neq 1$), which can further improve performance. \n\n## We evaluate DDRM with fewer steps, gives worse performance\nWe used 300 steps when running DDRM to match that of DCPS, but on the reviewers advice we also tried 20, which shows worse result. We will add that to the appendix. For the GMM case we used 1k steps as more steps would mean less discretization error, and instead using 20 shows worse performance for 8 and 80 dimensions, while being similar at 800 dimensions.\n## We have now evaluated on 1k images, computed standard deviations\nWe took the reviewer's advice and evaluated on a 1k image validation set. The numbers, however, differ only by a maximum 0.01. The standard deviations of the 1k values ranges between 0.01 up to 0.075.\n## We are not aware of linear inverse problems without implementable SVDs\nWe make it very clear that this method tackles linear inverse problems (it is in the title) which is still an open question (e.g., MCGDiff and DCPS tackle exactly this). However, we are not aware of linear inverse problems which do not have implementable SVDs and would gladly appreciate pointers to examples of this if implementable SVDs is a large concern. \n## Other points\n* We will add a paragraph in Related work to cite the suggested papers and discuss related MCMC methods.\n* We agree that the expression for $p_\\theta(x_t|x_{t+1})$ is a definition in the current problem formulation (although based on approximating $q(x_t|x_{t+1})$ when training the generative model) and will clarify\n* In the resampling step, a set of \"ancestor indices\" $`{`a_t^i`}`_{i=1}^N$ are sampled from the Multinomial distribution, and each particle $x_t^i$ is then replaced by $x_t^{a_t^i}$. The circular dependency comes from overloading the notation. We will clarify to avoid confusion\n* The notation in F.2.1. came from following the model formulation used by DAPS, which gave rise to a notational inconsistency. We will make sure to clarify", "claims_and_evidence": "The main claim is that the proposed method is asymptotically exact (see the abstract).  I did not find the explanation convincing because many approximations are made and it's not clear whether they all guarantee asymptotic exactness.  \n* In line 204 col 1, it's acknowledged that (13) is an approximation of the posterior.  Is this relevant to the proposed method or does it pertain only to DAPS?\n* In (16) a proposal distribution is constructed based on several approximations.  Does they interfere with asymptotic exactness?\n* Around (68) we learn that several key hyper parameters like $\\rho_t$ are heuristically adjusted. How does the choice of $\\rho_t$ affect asymptotic exactness?  \n\nAnother big issue with the paper is that the proposed DDSMC method is never clearly described.  Algorithm 1 doesn't capture what is described in the text.\n* It's not clear how $f_\\theta$ on line 171 col 1 is computed.  Based on line 321, this seems to require an inner loop with a DDIM ODE solver, but this is not clearly described.  For example, (66) seems to describe an inner loop, but the time variable $t$ is the same as used in the outer loop of Algorithm 1, making it impossible to understand.  \n* Throughout the paper, the word \"steps\" is used in an ambiguous and confusing way.  Based on Algorithm 1, there seems to be outer \"steps\" but also inner steps used when evaluating $f_\\theta$, but often the authors don't distinguish between them.  And there is a confusing comment in line 323 about \"remaining steps $t,t-1,...,0$\".  What does this mean?\n* In section F.2.1, the $\\sigma_t$ and $t_i$ quantities are described over $M$ steps, but these quantities do not appear in Algorithm 1, nor even the DDIM ODE update (66).  How does $M$ relate to $T$ in Algorithm 1?  How are the $t_i$ in F.2.1 related to $t$ in Algorithm 1?", "methods_and_evaluation_criteria": "It's not clear that the authors evaluated the competing methods under appropriate hyper parameter choices.  For example, the DDRM paper shows in their Table 6 that both PSNR and FID get worse as the NFEs are increased from 20 to 100, suggesting that \"more is not better\".  But in the paper under review, DDRM was evaluated with 1000 or 300 NFEs, which are massively larger than the standard value of 20, without justification.  This may be a very poor choice.", "theoretical_claims": "It would help to have a detailed theorem and proof statement for the \"asymptotically exact\" claim.  Currently the claim is too vague.", "experimental_designs_or_analyses": "One issue with the experimental analysis is that the number of NFEs is never clearly reported for the proposed method.  It seems to grow as the product of the number of particles $N$, the number of outer loop steps $T$, and possibly the number of inner loop steps $M$, but as reported earlier, the proposed method is never clearly described.  In any case, it is imperative to explicitly list the number of NFEs, which seems to be a serious drawback of the method.\n\nAnother issue with the experiments is that the authors use only a 100-image validation set, whereas 1000 is typical in most respected diffusion posterior sampling papers.  Just because DCPS and DAPS used 100 doesn't mean that it is sufficient.  With only 100 validation images, there is likely to be large standard errors on the averaged performance metrics, and no standard errors are even presented in Table 3.", "relation_to_broader_scientific_literature": "Personally, I think that this paper is a relatively minor variation on other recent SMC diffusion posterior sampling works like MCDiff and TDS.", "essential_references_not_discussed": "There are various other MCMC approaches to posterior sampling with diffusion that should be discussed.  For example\n* Florentin Coeurdoux, Nicolas Dobigeon, and Pierre Chainais. Plug-and-play split Gibbs sampler: Embedding deep generative priors in Bayesian inference. IEEE Trans. Image Process., 33:3496–3507, 2024.\n* Zihui Wu, Yu Sun, Yifan Chen, Bingliang Zhang, Yisong Yue, and Katherine Bouman. Principled proba- bilistic imaging using diffusion models as plug-and-play priors. In Proc. Neural Info. Process. Syst. Conf., 2024.\n* Xingyu Xu and Yuejie Chi. Provably robust score-based diffusion posterior sampling for plug-and-play image reconstruction. In Proc. Neural Info. Process. Syst. Conf., 2024.", "other_strengths_and_weaknesses": "I don't think the authors have been forthcoming on the computational burden of their method, as the NFEs used were never clearly stated. I suspect that the computational requirements are massive, making the method uninteresting for practical application.\n\nAlso, I don't think the authors have been forthcoming on the restriction of their method to linear inverse problems with implementable SVDs.  This is a strong restriction.\n\nFurthermore, while the authors dismiss other non-consistent SCM works (see line 249, col 2), they have not clearly established the consistency of their approach.", "other_comments_or_suggestions": "I suggest the authors pay more attention to clearly describing the proposed method and clearly stating and defending the main claims.", "questions_for_authors": "* In line 94 col 1, the expression for $p_\\theta(x_t|x_{t+1})$ is described as an \"approximation\", but it seems instead like a definition, given the goal stated in the first paragraph of the Background section.  In other words, given $q$ and $f_\\theta$, we define $p_\\theta(x_t|x_{t+1})$ from them and then aim to sample from $p_\\theta(x|y)$. Is that correct? Same question applies to (4).\n* In line 076 col 2, I wonder if there is a typo in the definition of the resampling step, because it is a circular definition.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YYaCc2wDly", "reviewer": "Reviewer_XuMA", "summary": "The paper proposes a new SMC method for sampling from the posterior of a Bayesian inverse problem that uses the as prior the time zero marginal of a learned score-based (or diffusion) generative model. The proposed SMC is influenced by the [1] but restricts itself to a Gaussian linear likelihood and instead of Langevin proposes an SMC approach. The paper also has links with [2] and [3], which are other SMCs methods in the literature. The paper also proposes an extension to discrete diffusion which is an unique feature with respect to the other available SMC samplers.\n\nThe proposed algorithm is evaluated both in a toy dataset where a tractable Bayesian posterior distribution is available and also on image datasets. While in the toy example it excels w.r.t the other available methods, in the image datasets it comes as a second best w.r.t [4], even though the authors rightly point out that the available metrics for the image task do not exactly measure the \"correctness\" of the posterior sampler, but rather the visual qualities of the images.\n\n\n[1] Zhang, B., Chu, W., Berner, J., Meng, C., Anandkumar,\nA., and Song, Y. Improving Diffusion Inverse Problem\nSolving with Decoupled Noise Annealing, July 2024\n[2] Wu, L., Trippe, B., Naesseth, C., Blei, D., and Cunningham,\nJ. P. Practical and Asymptotically Exact Conditional\nSampling in Diffusion Models. Advances in Neural Infor-\nmation Processing Systems, 36:31372–31403, December\n2023.\n[3] Cardoso, G., el Idrissi, Y. J., Corff, S. L., and Moulines,\nE. Monte Carlo guided Denoising Diffusion models for\nBayesian linear inverse problems. In The Twelfth Interna-\ntional Conference on Learning Representations, 2024.\n[4] Janati, Y., Moufad, B., Durmus, A. O., Moulines, E., and\nOlsson, J. Divide-and-Conquer Posterior Sampling for\nDenoising Diffusion priors. In The Thirty-eighth Annual\nConference on Neural Information Processing Systems,\nNovember 2024.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read and comment on our paper, which certainly has been useful to make the paper better. We answer concerns and questions below.\n## Assumptions in proposition A.1 concerns DAPS vs standard prior, not asymptotical exactness of DDSMC\nThanks for pointing out the unclarity regarding the assumptions in Proposition A.1., and how they affect the consistency of DDSMC. We emphasize that **Proposition A.1 only concerns properties of the DAPS prior**, not the DDSMC algorithm per se. Specifically, it refers to whether the DAPS prior ($\\eta =0$) and the standard diffusion prior ($\\eta=1$) result in the same (marginal) prior $p_\\theta(x_0)$. If the assumptions do not hold (as they will not in practice), these priors are different. However, regardless of the choice of prior (we view this as a design choice, more about this below),  **DDSMC will have asymptotic exactness guarantees**, i.e., the empirical approximation will converge to the corresponding posterior *induced by the chosen prior*. It is in this last point where SMCDiff and FPS are different: they also start from a diffusion-model prior which (combined with the given likelihood) induce a posterior. However, these algorithms then target _an approximation of the induced posterior_. This approximation, which is the target of their respective SMC samplers, will correspond to the actual induced posterior only when the forward and backward kernels match. This is what we mean by \"SMCDiff and FPS [...] are therefore not consistent in general\".\n\nAs mentioned above, it's important to note that the prior in DDSMC is a design choice. We have developed the method based on a generalization of the DAPS prior, because the decoupling offered by this prior has proven to be useful in prior work. However, the DDSMC method is equally applicable to the \"standard diffusion prior\" (simply set $\\eta=1$) and the algorithm will then provide consistent approximations of the posterior induced by this prior. In this case we do not rely on Proposition A.1 at all.\n\nWe thus believe that the assumptions in Proposition A.1 are used in a fundamentally different way in DDSMC than in SMCDiff and FPS. We thank the reviewer for highlighting this important detail, and we will make a clarification around line 144 col 1 in a revised version of the paper. See also response to reviewer tiGx about asymptotical exactness. If there are any more questions or concerns about any of this, we are happy to answer in a follow-up comment.\n## We have tried GMM experiments with same compute budget as for MCGDiff\nUsing Tweedie's formula as the reconstruction requires just a single evaluation of the score function, meaning in **this case we are already using the same compute budget as MCGDiff in the GMM experiments**. We will clarify that this is what we mean in line 358 col 1 when saying that \"DDSMC outperforms all other methods, even using Tweedie's reconstruction\".\n\nWe do agree, though, that using DDSMC-ODE under same compute budget is also interesting, and have therefore performed additional experiments using DDSMC-ODE with 12 particles (20x less than the other methods) and 25 particles (10x less, as the number of steps in the ODE decreases from 20 to 1, i.e. $\\sim 10$x more score evaluations/particle on average), and the results show that in low dimensions, 12 or 25 particles is still better than the Tweedie (and hence, MCGDiff/TDS), but this changes for higher dimensions, where using Tweedie with more particles seems to be the better choice. This shows how the SMC aspect (multiple particles and resampling) indeed is an important aspect in the performance. We will add these results in the appendix, along with a similar experiment where we change the number of ODE-steps instead, and a comment in the main paper.\n## Additional empirical results\nWe agree with the reviewer that \"an addition of a different source of real data would greatly enhance the evaluation\", and therefore looked at the protein structure completion in ADP-3D [1]. Their method is built for that type of task, and our method performs well out of the box, outperforming ADP-3D on higher, but realistic, noise levels (where we had to tweak their learning rate to give reasonable results).\n\nRMSD on 7qum protein. Columns indicate that every $n$ residues are observed.\n| Model ($\\sigma=0$) | 2 |4 | 16 | 32 | 64 |\n|---| ---| ---| ---| ---| ---| \n| ADP-3D | 0.229|0.378 | 1.690 | 3.590| 7.788| \n| DDSMC | 0.231|0.938 | 2.385| 3.858| 8.552| 13.643 |\n\n| Model ($\\sigma=0.1$) | 2 |4 | 16 | 32 | 64 | \n|---| ---| ---| ---| ---| ---| \n| ADP-3D | 1.371|1.429|3.404|4.540|8.542| 13.318| \n| DDSMC | 1.264|1.568 |2.849| 4.201| 8.927| 13.456 | \n\n| Model ($\\sigma=0.5$) | 2 |4 | 16 | 32 | 64 | \n|---| ---| ---| ---| ---| ---| \n| ADP-3D | 6.704| 7.087| 7.970| 9.283| 14.441|14.387| \n| DDSMC | 6.047 | 6.245| 6.742|7.479| 10.282| 14.79178619|\n \n[1] Levy et al. Solving Inverse Problems in Protein Space Using Diffusion-Based Priors, arXiv, 2024", "claims_and_evidence": "The claims of the paper concerning their performance are supported by clear and convincing evidence.\nHowever, I feel that there is a slight problem with one of the theoretical claims, namely the validity of the SMC sampler under general conditions. Notably, in the text, the authors claim :\n\n\"SMCDiff (Trippe et al., 2023) and FPS (Dou & Song, 2023)\nare two other SMC algorithms that target posterior sampling\nwith diffusion priors, but these rely on the assumption that\nthe learned backward process is an exact reversal of the\nforward process, and are therefore not consistent in general.\"\n\nwhich leads the readers to believe that this is not the case of the current approach. But I have doubts over such claim (see theoretical claims).", "methods_and_evaluation_criteria": "Yes, the benchmarks are well chosen and make sense for the application, even though I feel that an addition of a different source of real data would greatly enhance the evaluation of the current method (such as audio, or video or as in DCPS the ECG).", "theoretical_claims": "Yes, I have checked the theoretical claims and I have an issue with proposition A.1. While the proof is correct, the usage in equation (4) is not correct. Indeed, if we assume that $f_{\\theta}(x_{t+1})$ is a sample of $p_{\\theta}(x_{0})$, the to obtain a sample of $p_{\\theta}(x_t)$, proposition A.1 suggests that one has to use $p_{t|0}(x_t | x_{0})$ which is not equal $q_{t|0}(x_t | x_0)$ except if the forward and backward match, if I'm not mistaken. Thus, to the SMC proposed to be valid, it needs two assumptions: The ODE samples from $p_{0}(x_0)$ and that the \"forward of the backward\" $p_{t|0}$ is equal to the forward of the forward $q_{t|0}$.\n\nWhile this is not a problem per-se and can be considered as an approximation to render the SMC tractable, the SMC is not assymptotically exact in general conditions as is the case of MCGDiff and TDS. Indeed, it would fall under the category of \" SMCDiff (Trippe et al., 2023) and FPS (Dou & Song, 2023)\nare two other SMC algorithms that target posterior sampling\nwith diffusion priors, but these rely on the assumption that\nthe learned backward process is an exact reversal of the\nforward process, and are therefore not consistent in general.\"", "experimental_designs_or_analyses": "I checked the soundness of the experimental designs or analysis, but I have one issue with the current analysis is that the current analysis is not made under a same-budget criterion. Indeed, for example in the mixture of gaussians example, the authors state that they used 256 particles for all SMC samplers. The problem is that this would generate a much much higher NFE (Neural function evaluation) for their algorithm, as they need to solve the whole ODE for each time and each particle, which is not the case of MCGDiff or TDS. Therefore, instead of doing 256x20 NFE as MCG diff and TDS (TDS actually does a bit more) they do approximately 20 times more.. I would suggest increasing the number of particles in MCGDiff and TDS to have a fair comparison in such example.\n\nThis concern however, does not concern the image section where an almost equivalent budget is used for MCGDIff and the proposed method.", "relation_to_broader_scientific_literature": "Yes, the related material and the context of the proposed method are clearly explained. The paper also clearly explain the differences between the different SMC samplers.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "The paper is clearly written and does a clear review of the existing methods in SMC. \n\nBesides the two points raised above (theoretical claims and methodology), I feel however that one item that is lacking for practitioners is an analysis of the parameter sensitivity. While the authors show how the parameter $eta$ influences the performance, it is not clear how one should chose either the number of particles and the number of steps in the ODE. It would be interesting to see the tradeoffs between them in a fixed budget regime.", "other_comments_or_suggestions": "Line 171 second column there is an extra parenthesis in the Gaussian.", "questions_for_authors": "My main question is asked in the theoretical claims section, namely:\n1. Does the SMC holds under the hypothesis that the two joints distributions (backward and forward) do not match? My understanding is that this is not the case. \n2. How do all the algorithms compare with equal NFE in the mixture of gaussian case?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1nYz4dEnBF", "reviewer": "Reviewer_zGWq", "summary": "## Summary\n* This paper is based on previous work of solving diffusion inverse problems using sequential monte carlo [Practical and Asymptotically Exact Conditional Sampling in Diffusion Models]. More specifically, it takes the inner loop part of decoupled posterior sampling [Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing], and blend it with SMC based posteior sampling using an annealing parameter. By adjusting annealing parameter $\\eta$, this new prior can generalize previous SMC based methods.\n* The authors verify the effectiveness of their approach, on GMM and image restoration tasks. The proposed approach seems to be sota on GMM, and achieves competitive performance on image restoration tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments, which have been of great value to improve the paper. \n## Design choices in SMC make big difference in practice, motivating us to construct a new and better algorithm\nThe reviewer correctly points out that TDS and MCGDiff already enjoy asymptotic exactness. However, the choice in intermediate targets and in proposal can make a big difference in practice with finite number of particles. Motivated by previous works on SMC for diffusion priors, we set out to design a new and improved algorithm. The different SMC algorithms indeed show very different performance in the experiments, with DDSMC outperforming both MCGDiff and TDS, providing strong evidence that the design choices made in DDSMC give better practical efficiency compared to the other SMC methods. As a reply to _\"By adjusting annealing parameter $\\eta$, this new prior can generalize previous SMC based methods\"_ we would like to emphasize that DDSMC is a novel SMC method for the problem under study _regardless of the annealing parameter $\\eta$_. In fact, our main contribution is the development of this novel SMC method, and we view the generalization of the DAPS prior (i.e., introducing $\\eta$) as a secondary contribution. See also the reply to psuz. See response to tiGx and XuMA regarding asymptotic exactness\n## We will rephrase the part about PF-ODE for sampling from $q(x_0|x_{t+1})$\nIn the background section (line 104 col 1) we wrote that it is possible to use PF-ODE to sample from $q(x_0|x_{t+1})$, in order to generate a sample trajectory from the prior. We thank the reviewer for pointing out this error, and we agree that this is incorrect\n\nNote that **this paragraph was only included as an \"intuitive explanation\" of how the proposed method works, and the validity of the method does not in any way rely on the PF-ODE sampling from $q(x_0|x_{t+1})$**.\nWhat we actually meant to say with this section was that, conceptually, a (convoluted) way to simulate from the _prior_ backward process would be: \nInitialize $x_T\\sim q(x_T)$. \nFor $t=T-1,\\dots,0$,\n1. Solve $\\hat x_{0,t+1}=\\text{PF-ODE}(x_{t+1})$ from time $t+1$ to 0\n2. Sample $x_{t}\\sim q(x_{t}|x_0=\\hat x_{0,t+1})$\n\nThis would result in samples such that, marginally for any $t$, $x_t\\sim q(x_t)$, but we do not get samples from the _joint_ $q(x_{0:T})$, _nor from the conditionals_ $q(x_0|x_{t+1})$ as the reviewer correctly points out. This sampling process motivates the DAPS prior that we use (and generalize), which is why we mentioned it in the background, but we will of course make sure to update the text so that it is mathematically correct when revising the manuscript.\n## We have evaluated DDSMC on protein structure completion\nThe reviewer writes _\"The evaluation of this paper is a kind of limited.\"_ In the response to XuMA we have added results for another experiment concerning protein structure completion, showing that DDSMC can outperform the tailored APD-3D method out-of-the-box in (realistic) high-noise setting.\n## We have now evaluated using 1k images\nWe reran the experiments on 1k images, with essentially identical results (see response to psuz). We also computed PSNR, where now DDRM is the overall strongest model. However, just as for LPIPS, the standard deviation is rather large. Given that our method aims to recover posterior distributions, PSNR as a per-pixel metric (even stricter than per-sample metric like LPIPS) does not represent an ideal metric here, and for generative models there is often a trade-off between perceptual (e.g., LPIPS) and distortion (e.g., PSNR) metrics [1]. We will supply the PSNR table in the appendix with a comment in the main paper. We don't compute FID as we are focused on sampling from 1k different conditional distributions and not 1 unconditional distribution.\n\n[1] Blau and Michaeli, The Perception-Distortion Tradeoff, CVPR 2018\n## Clarification regarding complexity\nWe agree that a discussion is missing and will add this in the paper. In summary: DDSMC-Tweedie requires N times more (N=number of particles) NFEs per diffusion step compared with DDRM, and DDSMC-ODE has N times the NFE of DAPS. DDSMC-Tweedie has the same complexity as MCGDiff and requires slightly fewer NFEs than TDS (which requires differentiating through the score-function). See response to XuMA for an additional study using fewer particles to obtain the same number of NFEs for DDSMC-ODE as MCGDiff in the GMM case. For images, we are already using fewer NFEs as we are using fewer particles.\n\nThe additional NFEs required for SMC should be viewed as way of trading off improved sample quality with compute. As seen in the additional GMM experiments with fewer particles, this aspect holds empirically as the performance improves when using more particles (especially over using a single particle). For methods like DDRM or DCPS, we have attempted to use as much computation as reasonably possible and they still fail while DDSMC effectively enjoys the compute-quality trade-off.", "claims_and_evidence": "## Claims And Evidence\n* I have some concerns with using the solution of PF-ODE as a approximated sample for q(x0|xt+1). This is approximation is not well justified, as PF-ODE is marginal preserving not distribution preserving. The PF-ODE, starting from random xt+1, has same marginal distribution as q(x0). However, whether it serves as a good approximated sample in q(x0|xt+1) is not sure.\n* I am a little bit confused by the motivation of this papar. It seems that TDS [Practical and Asymptotically Exact Conditional Sampling in Diffusion Models] is already asymptotically exact. DDSMC use approximated sample such as tweedie and PF-ODE, to q(x0|xt+1). Does the approximation error harms the asymptotically exactness? Is there any theoretical advantage of DDSMC, over TDS?", "methods_and_evaluation_criteria": "## Methods And Evaluation Criteria\n* The evaluation of this paper is a kind of limited. The authors only verify their approach on 100 FFHQ images. This limited data makes it hard to compute divergence based metrics such as FID. A relatively larger, more diverse dataset such as 1000 ImageNet images, might strengthen the empirical results.\n* Further, only LPIPS is chosen as benchmark, while a more comprehensive comparsion using PSNR and FID can help readers understand the results better.  \n* The complexity of the proposed approach is quite high, while I find no metrics on this, such as wall clock time or FLOPS. It seems to me that DDRM can be super-fast, and DAPS can also be made fast by adjusting parameters. The Tweedie version of DDSMC can be as efficient as DAPS. However, I am really not sure about whether it is fair to compare PF-ODE DDSMC, as it appears a lot slower than DDRM and DAPS.", "theoretical_claims": "## Theoretical Claims\n* The theoretical claims look correct to me.", "experimental_designs_or_analyses": "## Experimental Designs Or Analyses\n* See Methods And Evaluation Criteria", "relation_to_broader_scientific_literature": "## Relation To Broader Scientific Literature\n* This paper contributes a new algorithm to the diffusion inverse solvers. The main contribution is a more effective SMC based algorithm with the idea taken from DAPS.", "essential_references_not_discussed": "## Essential References Not Discussed\nI find no essential reference missing.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a new sequential Monte Carlo method for linear-Gaussian inverse problems using diffusion models. Following previous works on posterior sampling with diffusion models such as MCGdiff and TDS, the authors use a proposal which uses information about the observations. They also follow DAPS to introduce a proposal which can be evaluated explicitly in the linear and Gaussian setting considered in the paper. The method is evaluated empirically in a Gaussian mixture models setting and compared to various state-of-the-art samplers. An experiment on image restoration is also proposed along with a proof of concept with discrete data.

During the discussion, the authors clarified Proposition A.1 and the theoretical foundations of their method to establish asymptotic guarantees. They also clarified some misleading statements in the original paper (the PF-ODE sampling for instance) which should be included in the revised version to improve the paper.

I believe that the GMM setting is very interesting to assess the performance of the model and aligns with previous works as it introduces a potentially very challenging posterior sampling problem with known ground truth. However, some reviewers were also concerned by the lack of more challenging numerical experiments. The authors proposed an additional experiment from [1] (Levy et al. Solving Inverse Problems in Protein Space Using Diffusion-Based Priors, arXiv, 2024) where their method outperforms the baseline for high noise levels. They also evaluate the experiment on image restoration on 1k images and propose to support the new results with a PSNR table. The fact that their approach outperforms TDS and MCGDiff which are natural SMC baselines is convincing numerically. Although I do not think it is indispensable for publication, I agree that adding a paragraph on the complexity of the method would be interesting.   

For all these reasons, I lean towards acceptance of the paper. 

Best regards.
Decision: Accept (poster)