Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Authors: Idan Achituve
Hai Victor Habi
Amir Rosenfeld
Arnon Netzer
Idit Diamant
Ethan Fetaya
Venue: ICML 2025 poster (2025)
Keywords: Inverse problems
sequential Monte Carlo
Latent diffusion
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: An approach for solving inverse problems in the latent space of diffusion models by augmenting the model with auxiliary observations. Sampling from the posterior is done using a novel SMC procedure.

Abstract: In image processing, solving inverse problems is the task of finding plausible reconstructions of an image that was corrupted by some (usually known) degradation operator. Commonly, this process is done using a generative image model that can guide the reconstruction towards solutions that appear natural. The success of diffusion models over the last few years has made them a leading candidate for this task. However, the sequential nature of diffusion models makes this conditional sampling process challenging. Furthermore, since diffusion models are often defined in the latent space of an autoencoder, the encoder-decoder transformations introduce additional difficulties. To address these challenges, we suggest a novel sampling method based on sequential Monte Carlo (SMC) in the latent space of diffusion models. We name our method LD-SMC. We define a generative model for the data using additional auxiliary observations and perform posterior inference with SMC sampling based on a backward diffusion process. Empirical evaluations on ImageNet and FFHQ show the benefits of LD-SMC over competing methods in various inverse problem tasks and especially in challenging inpainting tasks.

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Idan Achituve Hai Victor Habi Amir Rosenfeld
Arnon Netzer
Idit Diamant
Ethan Fetaya
Abstract
Output
In image processing, solving inverse problems is the task of finding plausible reconstructions of an image that was corrupted by some (usually known) degradation operator. Commonly, this process done using generative image model that can guide the reconstruction towards solu- tions that appear natural. The success of diffusion models over the last few years has made them leading candidate for this task. However, the sequential nature of diffusion models makes this conditional sampling process challenging. Fur- thermore, since diffusion models are often de- fined in the latent space of an autoencoder, the encoder-decoder transformations introduce addi- tional difficulties. To address these challenges, suggest novel sampling method based on se- quential Monte Carlo (SMC) in the latent space of diffusion models. We name our method LD-SMC. We define generative model for the data using additional auxiliary observations and perform pos- terior inference with SMC sampling based on a reverse diffusion process. Empirical evaluations ImageNet and FFHQ show the benefits of LD- SMC over competing methods in various inverse problem tasks and especially in challenging in- painting tasks.
partotees
1-1
4-9
Forward
Buckooed
Figure 1: LD-SMC solves inverse problem tasks in the latent space of autoencoders by utilizing auxiliary observations Yı.r initialized using the DDIM forward process. Then, sampling 2y from the posterior distribution done based on the reverse diffusion process using sequential Monte Carlo. In the figure, D and A denote the decoder and the corruption operator respectively.
and Gaussian denoising. The broad applicability of inverse problems makes them highly significant, as they encom- pass numerous real- world challenges, such as those found in digital image processing (Blackledge, 2005), wireless communication (Chen et al., 2021), seismology (Virieux & Operto, 2009), medical imaging (Song et al., 202 lc; Chung et al., 2023c), and astronomy (Craig Brown, 1986).
1. Introduction
Many important signal processing tasks can be viewed as inverse problems (Song et al., 2021c; Moliner et al., 2023; Daras et al., 2024; Chung et al., 2023b; Cardoso et al., 2023). inverse problems, the objective is to obtain clean signal R from degraded observation y A(x) + xb where A is usually known irreversible mapping and 1/b is Gaus- sian noise vector. Common applications that fit this frame- work include image deblurring. super-resolution, inpainting,
A major challenge in solving inverse problems is the istence of multiple plausible solutions. For example, in image inpainting, the likelihood p(y[x) remains constant regardless of how the absent pixels are filled. However, the desired solution is one that not only fits the observa- tion, but also appears natural, which corresponds to having high probability under natural image prior p(x). This insight naturally leads the approach of sampling from the posterior distribution p(xly) oc p(y|x))p(x) combining the data likelihood and the prior to achieve realistic and data-consistent solutions.
Equual contribution Sony Semiconductor Israel (SSI), Israel dence to: Idan Achituve <Ildnachi gomi..com>.
FFaculty Engincering, Bar-Ilan University, Isracl. Correspon-
With the impressive recent advances in diffusion models -Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021a), there has been significant interest leveraging
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
them as prior image models to solve inverse problems. How- ever, integrating diffusion models into this context is not straightforward because of their sequential sampling process. Specifically, diffusion sampling involves iterative drawing from p(x;-1|x{), while the conditioning on the corrupted image y is defined only in the final step, namely, through p(y Xo). This mismatch makes direct sampling from the joint posterior p(Xo, .... xrly) particularly challenging.
In this study, we make the following contributions: (1) we propose a novel method for combining auxiliary obser- vations with latent space diffusion models; (2) we derive approximate target distributions for the SMC procedure and novel proposal distributions to perform approximate pos- terior sampling specifically tailored for diffusion models; (3) We theoretically show that LD-SMC is asymptotically accurate, namely that it can sample from pe Yo): (4) LD-SMC achieves significant improvements in perceptual quality in inpainting tasks, one of the most challenging inverse problem tasks.
Recently, several studies proposed sequential Monte Carlo (SMC) (Doucet et al., 2001b; b; Del Moral et al., 2012) as effective solution for this task based on pixel-space diffusion models (Cardoso et al., 2023; Trippe et al., 2023; Wu et al.,
2023; Dou & Song, 2024). Specifically, Wu et al. (2023) applies the DPS approximation (Chung et al., 2023b) for p(y xx) with | and uses SMC sampling cor- rect for it. Dou & Song (2024), on the other hand, connected X, to y by introducing sequence of latent variables Y1:T through duplex forward diffusion process and sampling sequentially from p ((XxT)/e (ye)). While this approach has shown great potential, it has two main limitations. First, it does not take into account future observations Va:t- in the sampling process. Second, and more crucially, it is limited to linear corruption models only. As such, it cannot be ap- plied with nonlinear mappings A. let alone common Latent Diffusion Models (LDMs) (Rombach et al., 2022) due to the nonlinearity of the decoder. This is a harsh restriction as many of the recent powerful and efficient models are LDMs (Esser et al., 2024).
2. Background
Inverse Problems. In inverse problems, one would like to recover sample E R" from a corrupted version of it y E RT.. Usually, the corruption model that acted on is known, but the operation is irreversible (Tarantola, 2005). For instance, restoring a high-quality image from a low- quality one. We denote the corruption operator by A(-), and assume that y = A(x) + Φ, where ab ~ (N 12) has a known standard deviation т. In a more concise way, p(y|x) (A((x), ²²)). Common examples of inverse problems are inpainting, colorization, and deblurring. In general, solving inverse problem tasks is considered an ill- posed problem with many possible solutions x with equally high p(y |x) values. Given a prior distribution p(x) over natural images, one standard approach to solving the in- verse problem is to sample from the posterior distribution p(xly) x p(y)x)p(xx)
Both existing approaches have pros and cons. Using the p(y [[(x approximation can be helpful in capturing the large-scale semantics of the image, but it often lacks in capturing the small details. On the other hand, using auxiliary observations YI.T can help capture finer details, but the duplex forward diffusion process is not amenable to LDMs. Here, we propose a method that combines these two approaches and strives achieve the best of both worlds. We define generative model for the data based on re- verse diffusion process, according to which an auxiliary observation y: is generated directly from 24, the latent dif- fusion variable. Then, to apply posterior inference over the variables of all timesteps, Zg:r, we use SMC. To obtain a tractable sampling procedure, we derive an approximate target distributions and define novel proposal distributions for the SMC sampling process tailored for diffusion models. Hence, name method Latent Diffusion Sequential Monte Carlo, or more concisely LD-SMC. An illustration of our approach is shown in Figure 1. We empirically vali- dated LD-SMC on the ImageNet (Russakovsky et al., 2015) and FFHQ (Karras et al., 2019) datasets. We found that LD-SMC outperforms or is comparable to baseline methods on image deblurring and super-resolution tasks, and can significantly improve over baseline methods on inapainting tasks, especially on the more diverse ImageNet dataset.
Diffusion Models. Owing to their high-quality genera- tion capabilities, in recent years diffusion models (Sohl- Dickstein et al., 2015; Ho et al., 2020) have been leveraged as priors in inverse problems (Jalal et al., 2021; Song et al., 2021d). Here, we adopt the DDIM formulation (Song et al., 2021a) for the prior model, although our approach can work with other formulations of diffusion models well. Further- more, since it is costly to apply the diffusion process in the pixel space, a common approach is to apply the diffusion model in the latent space given by an auto-encoder (Rom- bach et al., 2022). Applying diffusion models in latent space allows us to sample high-quality images while reducing the computational resources needed by the model. Hence, de- signing models that effectively solve inverse problems using latent diffusion models is of great importance.
Denote by Zo:T the random variables in the latent space. Let Or.r, $1r be the variance schedule of the diffusion process with 3, := 1 - Qx- Also, denote by oz := I[ ag. The DDIM sampling starts from a prior distribution at timestep T set to p(zr) = (O, I). Then, for > 1 the sampling is done according to pe (1,)) = -N(z,- 1 (2g. 4), E()),
2

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
3. Related Work
Yr
YT-1
y1
yo
Inverse problems have a long and evolving history, with methodologies that have undergone significant advances over the years (Daras et al., 2024). Recently, diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021d) have emerged as effective priors for solving inverse problems in image data (Choi et al., 2021; Kawar et al., 2022; Chung et al., 2023a;b; Rout et al.. 2023; Song et al., 2023; Wang et al., 2023: Dou & Song. 2024; Garber & Tirer, 2024; Mardani et al., 2024; Sun et al., 2024).
A
A
XT-1
X1
X
X
Zr
#1-1
21
Zi
Figure 2: The graphical model of LD-SMC. In gray ob- served variables and in white are latent variables.
In (Song et al., 2021d) it was shown that to sam- ple from the posterior distribution, p(xuly), one can solve stochastic differential equation based on the prior score, Vx, log Po(X,), and the conditional score, Xx, log py(x)).). While the first term is easy to com- pute, the latter term requires integration over the full diffu- sion path from time to O. A useful and casy-to-calculate approximation found in several studies is po(y|x;) x po (y([ [x]]] =s po(y X (xx)), which is readily available at each step (Chung et al., 2023b; Song et al., 2023; Wu et al., 2023). Specifically, Diffusion Posterior Sampling (DPS) (Chung et al., 2023b) uses this approximation for linear and nonlinear inverse problems with Gaussian and Poisson likelihood models. Other methods also utilize the pseudoinverse of the corruption operator A (Tirer & Giryes, 2018). 8). IIGDM (Song et al., 2023) introduces a guidance scheme by matching the denoised output and the corrupted image y. via transformation of both. DDNM (Wang et al., 2023) suggested refining the contents of the null space of A during the reverse diffusion process. A different approach proposed in (Mardani et al., 2024) is to approximate the posterior using variational approach based on the score matching objective presented in (Song et al., 2021b). An additional category of inverse problem approaches that use diffusion models is designed with the objective of asymp- totic exactness (Cardoso et al., 2023; Trippe et al., 2023; Wu et al., 2023; Dou & Song, 2024). These methods utilize SMC techniques, which have also been applied for uncondi- tional sampling (Chen et al., 2025), to target exact sampling from the posterior distribution p(xoly). SMC-Diff (Trippe et al., 2023) was designed mainly for motif-scaffolding It uses the prior distribution as a proposal which is known to require a large number of particles for accurate estimation results (Särkkä, 2013), a severe limitation for latent space diffusion models due to expensive decoder evaluations. We experimented with this proposal in our initial attempts and witnessed that even 150 particles were not enough sample plausible reconstructions. MCGDiff (Cardoso et al., 2023) was designed for linear inverse problems only, which makes it unsuitable for inverse problems with latent-space diffu- sion models due to encoder-decoder involvement. TDS (Wu et al., 2023), a recent baCebased method, solves general inverse problem tasks using the twisting technique. This
where 0 are the parameters of the neural network and,
E(1) = If1
Mo (o( t) = (Zoz) + V
ā - -0 co(zi,t).
(1)
Here we denote the approximate posterior mean of E[zo| k by Zo (Zz) = vae (z - -VI--I---- ā2 (z), (Robbins, 1956;
Efron, 2011; Chung al., 2023b). For timestep t 1, the mean M4 (21, 1) is taken to be (0(2)). As in (Dou & Song, 2024) we fix d 17 . v Be parameter.
1- with 17 being hyper- 1-a,
Sequential Monte Carlo (SMC). SMC is an important technique for sampling in probabilistic graphical models in which exact posterior inference is intractable. The SMC breaks the sampling process down to intermediate steps, allowing efficient sampling through a recursive procedure (Doucet et al., 2001b; Del Moral et al., 2012; Naesseth et al., 2019; Chopin et 2020).
One family of probabilistic models for which SMC is espe- cially known is state-space models (SSMs), also known as Hidden Markov Models (HMMs). In general, the following quantities must be defined in SSMs, (1) a prior distribution over the initial state p(zr). (2) transition distribution that defines the dynamics between states p(ze|Zec+) 1) Vt < T, and (3) measurement model p(y: (Z,) Vt < T. The goal is to sample from the target distribution (Yerr)).-.)).... To do so, SMC starts by sampling N partieles (z)) N from the prior distribution. Then, at each step, given the previous particle set {z,
(6) I
Ji=1 new samples are taken from a pro- (9)
posal distribution _ (2,0 (8) (8) Vi E {l...., N).
The particles are then weighted (and possibly resampled)
according to the new sequences {z (=,-i-T)= The proposal distribution serves an approximation to the target distri- bution. Its support needs to contain the support of the target density. The weighting mechanism then corrects the approx- imation by assigning a weight to each particle to adjust its probability. The resampling step, if applied, aims to remove unlikely particles according to the model (Särkkä, 2013).
3

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Algorithm LDD-MC
the absence of publicly available code, making replication challenging. ReSample (Song et al., 2024), contemporary method alongside PSLD, acts in two stages; it first applies hard data consistency to obtain latent variables that are con- sistent with the observed measurements and then employs a resampling scheme to map the sample back onto the data manifold. Concurrent to this study Nazemi et al. (2024) proposed a particle filtering approach. Their method builds PSLD and DPS update in the proposal distribution. Sim- ilarly to TDS (Wu et al., 2023) the connection to the labels is only through 2o using the approximate mean estimator. Also, unlike LD-SMC it does not enjoy asymptotic guaran- tees. Recently, both DAPS (Zhang et al., 2024) and MGPS (Moufad et al., 2024) have been proposed to address inverse problems both in pixel and latent space. DAPS decouples latents in the sampling trajectory by first sampling from zo/ Z y using MCMC techniques, for some At >> 0, and then sampling 2, MGPS also decomposes the sam- pling, but to an intermediate midpoint and uses variational inference for posterior sampling. Given the similarities of both methods, in our experiments we compare only to the former. SILO (Raphaeli et al., 2025), an additional recent study, first learns NN that mimics encoder operation on labels y which is then used guide the sampling process. In our empirical evaluation, we compare to methods that apply sampling only.
1: Set Zo = ( where Xo = arg min Ilvo - 2
2: Sample Z1:7 ~v p ((11)(/z))) according to the forward
process of DDIM (Eq. 2)
3: For = 1,.,, K:
4: Sample Y1:T II:= ((Dz) 72)
12)
5: Sample Zp:T ~ (Za:7 (Y:T) using SISR
6: SISR(Yo:T):
7: For i = N:
8: Sample 2T pz() and initialize z(x) (ZiT) (3)
(2)
9: Set w/ according to Eq.
10: Normalize {w/ J.v to sum to one
11:
For - 1....,0: 0:
{z [ ]==1 ~ Multi( (2) J fwff JN
12:
a resampling step
13:
For = I,.,,,, N:
14:
Sample zi ~ Tee (2 (2(1) (Eq. 7) # proposal
15:
Set wn!" according to Eq. 6
16:
(8) Update z = (z) ...... T
17:
Normalize {w, N to sum to one
Sample one chain z ~ Multi((z)) ( (wo (0) )
18:
19:
Return z
method also uses the approximation of DPS, but by apply- ing SMC sampling it can correct for it. FPS (Dou & Song, 2024), also a recent SMC-based method, uses auxiliary vari- ables. FPS generates a sequence of observations Yı.r based on a duplex diffusion process, one process at the x space and the other process at the y space. Since this method is de- signed for linear inverse problems only, it permits tractable Bayesian inference. Our method combines the ideas of both TDS and FPS to obtain the best of both. Namely, we use the posterior mean approximation and Vı. in our SMC sampling process. As we will show, this combination can be helpful in both understanding the general semantics of an image and capturing fine details. We note here that there other methods that applied label augmentations similar to FPS, one example is (Abu-Husseds et al., 2022) which is applicable for linear inverse problems only. Hence, this method as well cannot be trivially combined with latent space diffusion models.
4. Method
Given a corrupted image yo. the goal is to sample zo po yo) using a pre-trained latent diffusion model as prior. Then, we can transform this sample into an image by ap- plying a pre-trained decoder D, i.c. X == D(zo). Here, we stack all the parameters of the networks (diffusion and encoder-decoder under A. In what follows, we first define a generative model for the data and then describe how to perform Bayesian inference on all latent variables using blocked Gibbs sampling. Specifically, we use the diffusion process and the corruption operator to augment the model with additional auxiliary observations. Then, inference is applied over the diffusion variables using SMC. The corre- sponding graphical model can be seen in Figure 2.
4.1. The Generative Model
We now explicitly define the data generation model,
Several inverse sampling methods were specifically tailored for latent diffusion models. PSLD (Rout et al., 2023) extend DPS (Chung et al., 2023b) by incorporating an additional gradient update step to guide the diffusion process to sam- ple latent representations that maintain the integrity of the transformation, ensuring it remains non- lossy. STSL (Rout et al., 2024) uses second-order informa- tion for reconstruction based on efficient approximations. Comparative analysis with STSL, was not feasible due to
1. 27 ~ N((OI)).
2. 2.2, 2. 12z ~ p-pp(z- (2:) WE (I,.,,,,
3. y./z N (((((()))) 12) VI (O..., T).
Where. pe (y is defined by the corruption model, p(2,1 1 (2)) = Bp(z)l2.) (p( (2(2-11 |2,, 2o)] is a backwards gen- erative process that corresponds to a non- non-Markovian forward
4

---

Pe (zolyo)) Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
True
Observation
LD LD-SMC (Ours)
LDPS
LTDS
ReSample
PSLD
PFLD
LatentDAPS
Figure 3: Comparison between LD-SMC and baseline methods on inpainting of ImageNet images.
process with
(b) Sample, Zo:r ~ po (Zu: (Yorr) using SMC based pre-trained diffusion model (Sec. 4.2.2).
p(z-11 1/2,, 20)
W (2- + / ai - oo? Vorzo 0.1
Here, we use the dynamics of the forward process and the graphical model dependencies in steps 2 and 3(a). The two steps that are not straightforward are obtaining an initial guess for zo (step 1) and performing the sampling process in step 3(b). We discuss both in the following sections, but before that make few comments. First, clearly the run-time of the algorithm depends linearly on the number of Gibbs iterations. As result, the sampling time can be slow. However, with proper initialization (as discussed in Section 4.2.1), we empirically found that even one iteration of Step suffices to achieve good results. Hence, unless otherwise stated, in our empirical evaluations, we applied this step only once. Second, one may be concerned that evaluating the decoder noisy latent variables can generate non-natural images. While this may be true, empirically we observed a graceful degradation in image quality with time, as seen in Figure 7. Furthermore, the latent variables, even if noisy, carry information about yo- This information is then transferred to the auxiliary labels which help guide the sampling process.
(2)
set according DDIM (Song et al., 2021a). Introducing unob- served data a known technique in statistics for conducting effective Markov chain Monte Carlo (MCMC) sampling (Van Dyk & Meng. 2001; Dou & Song, 2024). In our case, use it while leveraging the dependencies between the variables in order to build an efficient SMC sampling procedure, as described in the next section.
4.2. Sampling Procedure
Given the generative model defined in Section 4.1, our aim is to apply Bayesian inference over the latent vari- ables. In broad strokes, to obtain sample we use blocked Gibbs sampling to sample in turns from po(y). Lorr Zor.yo) yo) using knowledge on the corruption model and po (Zuer Yo.r) using SMC. Specifically, we propose the following procedure:
1. Obtain an initial guess for zo (Sec. 4.2.1).
4.2.1. INITIAL GUESS FOR
2. Sample, 21.7 ~ p(zer Zo. yo) = p(z (20) accord- ing to the forward process of DDIM (Eq. 2).
The first challenge is to obtain an initial (step 1. in the sampling procedure). There are multiple sensible ways to perform this initialization. Importantly, 2g should carry in- formation about the measurement Yo. which can then be used to guide the sampling process using the auxiliary la-
3. Repeat for some fixed number of steps:
(a) Sample, YI.r Po(y1.r Zo:T. (oVErlZZT)) = ((ylL(((
5

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
True
Observation
LD SMC (Ours)
LDPS
LTDS
ReSample
PSLD
PFLD
LatentDAPS
Figure 4: Comparison between LD-SMC and baseline methods on Gaussian deblurring of ImageNet images.
bels. As result, the variance in the process will be reduced and the convergence will be accelerated. For example, one option is to leverage the pseudoinverse of A for linear op- erators similar to (Wang et al., 2023). But, that would not work for nonlinear corruption operators. Although there are mitigations for this problem (Song et al., 2023), to maintain generality, applied the following optimization procedure,
Approximate target distributions. First, notice that due to the structure of the model, the posterior density of the r.v. z at each step f depends only on Z4+1:7- Hence, the target distribution of interest at each timestep pa Yo.T). However, even computing the unnormalized quantity of that target distribution be costly. Therefore, make the following assumption pe (2::|/o:r) = pe Ye.r Yo). This assumption is reasonable since yo stores all the input information to begin with. In Appendix B we arrive at the following recursive formula which forms the approximate target distributions of the SMC procedure for t:
arg max log pe (yolx) = arg min ||yo 12. (3) where is initialized to D(zu) for 2n I). Then, after reaching convergence, an initial 2n is obtained by ap- plying the encoder on the outcome, namely 2o = {xx)). An alternative for this procedure to perform the optimiza- tion process directly the latent space. However, our experiments we found that the former option worked better and was substantially faster as did not involve expensive gradient propagation through the decoder.
x
Po (ZuT Yo) x
Pa Py ZT (a ZT (ZT)) Poly, z. (yolee))) po Ze+ wwwp:///wwwwwwwww.1..... 1
T,
0 0< t < T,
Po
Pelyol my po (Zo (002111000020)
t= 0,
4.2.2. POSTERIOR SAMPLING
(4)
We now move on to explain step 3(b) of the sampling pro- cedure. Due to the non-linearity of the decoder, even for linear corruption operators A, finding the exact posterior is intractable. One option to overcome this difficulty to use SMC sampling. In what follows, we first describe approxi- mate (tractable) target distributions for (po(ze:) (yor) ) then we present the induced sequence weights w, and novel proposal distributions for all timesteps t. These components are used as part of a general sequential im- portance sampling with resampling (SISR) algorithm as depicted in Algorithm 1.
where we define (yool)))) = po(yolZo(2.))
N (yyolA(D(Z0(z.) (1 āz) with the variance taken to be the variance of z, 22 according to the forward pro- cess. Importantly, as the final target distribution at matches the desired distribution po (Zo:T) Yo:r)) (see Eq. 10 in Appendix C), in the large compute limit samples from the correct target can be obtained despite all approximations.
Sequence weights and proposal distributions. To de- rive an SMC procedure using the proposed target distri- butions, common to use importance sampling. The
6

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Table 1: Quantitative results on 1024 examples of size 256 x 256 from FFHQ test set. All methods were evaluated under
the same experimental setup using LDM. Lower (4) is better in all metrics.
Inpainting (Box)
Inpainting (Free-Form)
Gaussian Deblurring
Super-Resolution (8x)
FID
NIQE
LPIPS
FID
NIQE
LPIPS
FID
NIQE
LPIPS
FID
NIQE
LPIPS
LDPS
39.81
7.592
0.236
40.17
7.609
0.212
29.30
6.538
0.237
29.64
6.412
0.282
LTDS
39.57
7.602
0.236
39.78
7.578
0.212
30.23
6.553
0.238
30.45
6.412
0.284
ReSample
86.79
7.142
0.230
37.01
6.622
0.151
39.80
7.441
0.275
59.23
7.307
0,356
PSLD
39.68
6,544
0.246
36.26
6.835
0.216
36,31
6.802
0.341
40,33
6,803
0.347
PFLD
39.06
6,509
0.245
36.43
6,817
0.215
37.16
6.751
0.343
38.11
6.832
0.345
LatentDAPS
60.24
9.999
0.257
54.40
8.766
0.223
54.28
9.496
0.283
70.24
10.17
0.344
LD-SMC (1 particle)
33.37
7.032
0.212
33.67
7.034
0.194
29.19
6.575
0.232
30.02
6.426
0.277
LD-SMC (5 particles)
33.87
7.066
0.211
33.60
7.021
0.194
29.47
6.528
0.233
30.62
6.455
0.278
key idea is to construct proposal distributions ny (z.T), one for each timestep, from which is easy to sample, and approximate p(zerllyorr) with the empirical distribution EN w 8 (Zur). Where, s is the Dirac measure z and the weight w/ (presented here for < T) is defined as,
In Eq. 7, we set the variance to be the variance of the prior diffusion model, namely Øz+1 = 0++1; however, other choices are also applicable. The idea behind the proposal mean is to correct the prior mean estimation by shifting it toward latents that agree more strongly with both y, and yo- S Specifically, the second term of the correction can be seen as making one gradient update step starting from the current prior mean, which serves as an estimate to ZІ-
Po (Z. we
Te (Z(TT)
(z, No. 1
The parameters 7t and X control the effect of the correction terms to the prior mean. In practice, it is challenging con- trol the trade-off between the two correction terms. Hence, propose the following simple approach. During the sam- pling process, the first term only is used, that is, A = 0, and then starting from some predefined timestep s, the second term is used as well. The intuition here is that during the initial sampling steps, the quality of the labels y, may not be good. Therefore, we rely on the first term through the posterior mean estimator capture the general semantics of the image. However, in later sampling stages the quality of the labels increases (see Figure 7 in the appendix), and the latter correction term can help capture fine details in the image. This design choice also relates to the three-stage phenomenon in the diffusion sampling process witnessed in the literature (Yu al., 2023). Setting A timesteps reduces the LD-SMC proposal update to that of TDS (Wu et al., 2023). Appendix D shows an instantiation of 7/ and d used in this study.
(5)
Te(Zo|Zz->
Te+ (Z)+1TT)
7o (2+ (t,
W++1-
Te Te(Zz|z(1)
Here, (-) is the unnormalized approximate target density. Plugging the unnormalized target densities from Eq. 4 in Eq. for all timesteps & results in,
Pio (yr bar Fo (yol ker Ipo (xxr)
1= 0 < <<t< T.
(R7)
pe (yr le (yoo (2e
W/ X
Po (Yo, Re4 (2. (2.-1)
Wo+1,
pe (yn (2o X1)
W1.
t= 0.
Fu (y
With these weights, we now turn to defining the proposal distributions n, for all timesteps t. The optimal choice in the sense of minimizing the variance of the weights is T (Zz (elzzer)= = Po (z, (o:4).Yo:) (Doucet et al., 2000; Särkkä, 2013). However, they cannot be obtained in closed form. Hence, we design alternative proposal distributions. The proposal distributions for timesteps i < T are defined to be Gaussian Tz Z = (mm,, S,) with parameters:
for all
Having defined the weights and the proposal distributions, we present LD-SMC concisely in Algorithm 1. The algo- rithm has two parts; the first part is the Gibbs sampling process and the second the SMC sampling using SISR. In the algorithm the abbreviation "Multi" refers to Multi- nomial distribution and resampling step is performed at each iteration, although it is not mandatory. Appendix C provides a proof that LD-SMC can achieve an arbitrarily accurate estimate of pe (z (oo under several conditions. Importantly, the estimate does not depend on any of the approximations made derive our model. The following
S. = I
m, 140 (o((2+1,44 / + 1) (7t Vac. log Do poooelze41) (7)
+ + CaVVi,(ee1ttte log ( (z)1)))
Where qo = N and 7t. d are finite scaling coefficients. In addition, Tr (zr). ). the distribution of the proposal at time T, is set to the diffusion prior distribution, (z() = =O,
(2++1, 71),

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Theorem summarizes that informally,
10.6
36
23.54
Theorem 4.1. (informal) Let PN(ZO:)
Einww 6 cn (Zo.T) be the discrete measure obtained soir by the function SISR in Algoritim 1. Under regularity conditions PN (Zo:T) converges setwise to po (Zorr Nor) as N + xc. xc. Furthermore, the stationary distribution of the Gibbs sampling process is po (Z..T.Yu.Tll and the marginal pa (zolYo) is the limiting distribution of the zo 's subchain.
g
2.7
PSNR
56-
19.3 g
50-
192
36
23.42
100
200
333
500
Num. Particles
(a) Effect of s
(b) Effect of N
Figure 5: FID and PSNR values when varying & (left) and the number of particles N (right) on ImageNet box (left) and free-form (right) inpainting tasks.
Connection to TDS. While the derivation is different, the SMC procedure in L.DSMM resembles somewhat that of TDS (Wu et al., 2023). The main difference between the two methods is the dependence on the auxiliary variables y1:7. which are only relevant for LD-SMC in both the proposal distributions and the weights. Empirically, we observed that this additional conditioning helped to better align the sampling with the corrupted image yo compared to using the posterior mean approximation as in (Chung et al., 2023b) and (Wu et al., 2023).
(3) ReSample (Song et al., 2024), which applies opti- mization procedure during the sampling process to match the approximate posterior mean to the label followed by a resampling step; (4) PSLD (Rout et al., 2023), which intro- duces a correction term to the DPS step to "glue" zo: (5) PFLD (Nazemi et al., 2024), a particle filtering approach for LDM based on PSLD update for the proposal distribu- tion; (6) LatentDAPS which decouples the latents in the sampling trajectory. In all experiments,
5. Experiments
evaluated LD- SMC with N = {1,5) 5} particles. Using one particle incurs a computational cost similar to that of LDPS (see Table 4 in the appendix). LTDS and PFI.D were evaluated with N = 5 particles, All methods were tested on the following inverse problem tasks: box inpainting, Gaussian deblurring, super- resolution (8x) and free-form inpainting. We followed the setup in (Chung et al., 2023b) for the first three tasks and the setup in (Saharia et al., 2022) for the last one with few changes to the default hyperparameters. In inpainting tasks, the masks were randomly sampled for each new sample.
5.1. Experimental Setting
We evaluated LD-SMC on ImageNet (Russakovsky et al., 2015) and FFHQ (Karras et al., 2019); both are common in the literature of inverse problems (Chung al., 2023b; Dou & Song, 2024). In ImageNet, samples were conditioned on the class label. The guidance scale was fixed to 1.0 in all our experiments. Results can be improved by adjusting it (Rom- bach et al., 2022). Images were resized to x 256 x 256 and normalized to the range [0. 1]. We used the latent dif- fusion models VQ-4 CIN256-V2 (Rombach et al., 2022) as the prior model in FFHQ / ImageNet, respectively, with the DDIM diffusion sampler (Song et al., 2021a), accord- ing to the data split in (Esser et al., 2021). We sampled 1024 random images from the validation set of each dataset which were used to evaluate all methods. We followed the protocol of (Song et al., 2024) and added Gaussian noise with zero mean and standard deviation r = 0.01 to the cor- rupted images. Full experimental details are provided in Appendix A.
Evaluation metrics. In the main text we report the fol- lowing metrics, FID (Heusel al.. 2017). NIQE (Mittal et al., 2012), and LPIPS (Zhang et al., 2018). Full results including PSNR and SSIM (Wang et al., 2004) are deferred to Appendix H. FID and NIQE are considered perceptual metrics; lower values in them indicate higher perceptual quality. The other metrics can be considered as distortion metrics, which quantify some discrepancy between the gen- erated images and the ground-truth values. Since perceptual metrics and distortion metrics can be in conflict with each other (Blau & Michaeli. 2018), we put more emphasis on perceptual quality. Hence, for all methods, we performed and chose the best con-
Compared methods. We compare LD-SMC with several recent SoTA baseline methods. All methods were evalu- ated in a similar experimental setup using the same latent diffusion model ensure fairness in the comparisons. (1) DPS (Chung et al., 2023b), which introduces correction to the sampling process of the diffusion through the posterior mean estimator; (2) TDS (Wu et al., 2023), which uses the twisting technique for approximate sequential Monte Carlo sampling; since DPS and TDS were designed for pixel-space diffusion models, we term the adapted versions of them to the latent space LDPS and LTDS, respectively.
grid search over figuration according to the FID.
5.2. Experimental Results
Quantitative results are shown in Tables I and 2. From the tables, LD-SMC is usually the best or second best among all the comparisons. Specifically, on inpainting where extrapo- lation is needed within the masked region and details should
8

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Table 2: Quantitative results on 1024 examples of size 256 x 256 from ImageNet test set. All methods were evaluated under the same experimental setup using LDM. Lower (4) is better in all metrics.
Inpainting (Box)
Inpainting (Free-Form)
Gaussian Deblurring
Super-Resolution (8x)
FID
NIQE
LPIPS
FID
NIQE
LPIPS
FID
NIQE
LPIPS
FID
NIQE
LPIPS
LDPS
65.04
7.935
0.379
53.47
7.867
0.334
52.48
6.855
0.383
61.02
6.514
0.439
LTDS
64.74
7.907
0.378
52.75
7.884
0.334
50.82
6.695
0.379
59.12
6,270
0.435
ReSample
90.32
8.464
0.318
44.15
7.104
0.248
46.45
7.411
0.353
87.65
8.290
0.491
PSL.
71.15
8.042
0.434
62.38
8.037
0.411
60.68
6,599
0.417
66,56
7.669
0.489
PFLD
72.83
7,933
0,436
63.34
8,026
0.414
60.94
6.733
0.421
64.72
7.685
0,492
LatentDAPS
98.24
11.36
0.394
68.65
9.625
0.330
77.09
10.55
0.417
104.6
12.68
0.489
LD-SMC (1 particle)
51.49
6.878
0.326
38.21
6.969
0.285
52.48
6.855
0.383
58.06
6.243
0.434
LD-SMC (5 particles)
50.67
6.891
0.325
36.18
6.671
0.278
S2.29
6.791
0.383
57.89
6.238
0.433
be preserved outside of it, LD-SMC can greatly improve over baseline methods. This property is also manifested in Figure 3 and Figure 8 in the appendix. LD-SMC man- ages to produce plausible reconstructions while maintaining fine details. The differences are especially highlighted on the ImageNet dataset which has more diversity in it. Also, as is clear from the figures, ReSample images suffer from significant artifacts. We speculate that it partly stems from the complete optimization process performed in every few sampling steps according to this method.
distribution which results in a significant reduction in per- ceptual quality. Figure 5(b) shows an improvement in FID while suffering from a small reduction in PSNR when in- creasing the number of particles. In our comparisons, we chose N = 5 to balance between performance and com- putational cost incurred by adding particles (see Section E for further details). Empirically, we found that sometimes only increasing the number of particles resulted in favorable metric values, as also clear from the tables. We attribute this to the relatively small number of particles with which we experimented (up 10). However, in general, one can expect that the number of particles increases, the sam- ples will become more accurate. Finally, in Appendix F we present an ablation study on the effect of using several Gibbs steps. We show that improvements in FID and PSNR can be obtained by taking more than one step, yet there is not always a clear trend of improvements over the steps.
Figure 4 shows qualitative results for Gaussian deblurring of images by all methods. From the figure, LD-SMC, LDPS, LTDS, and ReSample are able to generate plausible recon- structions on this task. PSLD and PFLD images have no- ticeable artifacts, while LaPs images are blurry.
Additional quantitative and qualitative results can be found in Appendix H and I. In terms of the distortion metries PSNR and SSIM, LD-SMC usually substantially outper- forms PSLD and PFLD and has a small advantage over LDPS and LTDS, but overall the three methods are com- parable. ReSample and LatentDAPS are usually the best. Nevertheless, as can be seen visually and as indicated by other metrics, it comes at the expense of perceptual quality. LatentDAPS images are overly smooth and details are not preserved in them, while ReSample images have noticeable artifacts, especially in inpainting tasks.
6. Conclusion
In this study, we presented LD-SMC, a novel method for solving inverse problems in the latent space of diffusion models using SMC. LD-SMC augment the model with aux- iliary observations, one per each timestep, and use these observations to guide the sampling process as part of the reverse diffusion process. This framework can be seen as applying one step of blocked Gibbs sampling. To perform SMC sampling, we suggested a novel weighing scheme and proposal distributions. Both are based on information from the auxiliary labels and the true label yo- Empirically, we validated LD-SMC against strong baseline methods on com- mon benchmarks. The results suggest that LD-SMC can improve the performance over baseline methods, especially in cases where extrapolation is needed (e.g., in inpainting tasks). one limitation of our approach is related to com- putational demand. The sampling time and the memory demand increase with the number of particles and Gibbs iterations. This effect can be partially mitigated by taking fewer particles and GPU parallelization.
5.3. Analysis
In Figure we evaluate I.DSMC in terms of FID and PSNR on inpainting tasks when varying s, the hyperparameter that controls the influence of y: in the proposal distribution, and N, the number of particles. From Figure 5(a) there is a clear trf in FID and PSNR when & changes. Higher s values result in a better FID at the expense of PSNR, while lower values show the opposite trend. For inpainting, s = 333 strikes a good balance between the two metrics while giving more emphasis to the perceptual quality. In addition, note that taking. s= = 0 is similar to using the DPS as proposal
9

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Impact Statement
Craig, I. and Brown, J. Inverse problems in astronomy. guide to inversion strategies for remotely sensed data. Bristol: Hilger, 1986.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Daras, G., Chung, H., Lai, C.-H., Mitsufuji, Y., Ye, J. C., Milanfar, P., Dimakis, A. G., and Delbracio, M. A survey on diffusion models for inverse problems. CoRR, 2024.
References
Del Moral, P., Doucet, A., and Jasra, A. An adaptive se- quential Monte Carlo method for approximate Bayesian computation. Statistics and computing, 22:1009-1020, 2012.
Abu-Hussein, S., Tirer, T., and Giryes, R. ADIR: Adap- tive diffusion for image reconstruction. arXiv preprint arXiv:2212.03221, 2022.
Blackledge, J. M. Digital image processing: mathematical and computational methods. Elsevier, 2005.
Dou, Z. and Song, Y. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. The Twelfth International Conference on Learning Representations, 2024.
Blau, Y. and Michaeli, T. off. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
The perception distortion trade-
2018.
Doucet, A., de Freitas, N., Murphy, K. P., and Russell, S. Blo-Blackwellised particle filtering for dynamic Bayesian networks. In Boutilier, C. and Goldszmidt, M. (eds.), Proceedings of the 16th Conference in Uncertainty in Artificial Intelligence (UAI), pp. 176-183, 2000.
Cardoso, G., Idrissi, E.,, Corff, S. L., and Moulines, E. Monte carlo guided diffusion for bayesian linear inverse problems. arXiv preprint arXiv:2308.07990 2023.
Chen, J., Richter, L.., Berner, J., Blessing, D., Neumann, G., and Anandkumar, A. Sequential controlled Langevin diffusions. In The Thirteenth International Conference on Learning Representations ICLR. OpenReview.net, 2025.
Doucet, A., De Freitas, N., and Gordon, N. An introduction to sequential Monte Carlo methods. Sequential Monte Carlo methods in practice, pp. 3-14, 2001a.
Chen, W., Zhang. B., Jin, S., Ai, B., and Zhong, Solving sparse linear inverse problems in communication systems: A deep learning approach with adaptive depth. IEEE Journal Selected Areas in Communicutions, 39(1): 4-17, 2021.
Doucet, A., De Freitas, N., Gordon, N. J., et al. Sequential Monte Carlo methods in practice, volume 1. Springer, 2001b.
Efron, B. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496):1602 1614, 2011.
Choi, J., Kim, S., Jeong, Y., Gwon, Y., and Yoon, S. Ilvr: Conditioning method for denoising diffusion probabilistic models. 2021 IEEE/CVF International Conference on
Computer Vision (ICCV), pp. 14347-14356 2021.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Chopin, N., Papaspiliopouloss O., et al. An introduction to sequential Monte Carlo, volume 4. Springer, 2020.
Chung, H., Kim, J., Kim, S., and Ye, J. C. Parallel diffusion models of operator and image for blind inverse problems. IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2023a.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D.. Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-firs International Conference on Machine Learning, 2024.
Chung. H., Kim, J., McCann, M. T., Klasky, M. L., and Ye, C. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Con- ference on Learning Representations ICLR. OpenRe- view.net, 2023b.
Garber, T. and Tirer, T. Image restoration by denoising diffu- sion models with iteratively preconditioned guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25245-25254 2024.
Chung, H., Ryu, D., McCann, M. T., Klasky, M. L., and Ye, J. C. Solving 3d inverse problems using pre-trained 2d diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 22542-22555 IEFE, 2023c.
Gordon, N. J., Salmond, D. J., and Smith, A. F. Novel ap- proach to mmnlinear/non-gaussinn bayesian state estima- tion. In IEE proceedings F (radar and signal processing). volume 140, pp. 107-113. IET, 1993.
10

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Robbins, H. An empirical bayes approach statisties, In Proceedings of the Third Berkeley Symposium Mathe- matical Statisties and Probability, Volume 1: Contribu- tions to the Theory of Statistics, volume 3, pp. 157-164. University of Californa Press, 1956.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in Neural Information Process- ing Systems,
Robert, C. P. and Casella, G. Monte Carlo statistical meth- ods, volume 2. Springer, 1999.
2020.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEFF/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Jalal, A., Arvinte, M., Daras, G., Price, E., Dimakis, A. G., and Tamir, Robust compressed sensing MRI with deep generative priors. Advances in Neural Information Processing Systems, 14918-149944,, 2021.
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 4401-4410. Computer Vision Foundation / IEEE, 2019.
Rout, L.,, Raoof, N., Daras, G., Caramanis, C., Dimakis, A., and Shakkottai, S. Solving linear inverse problems prov- ably via posterior sampling with latent diffusion models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Process- ing Systems (NeurIPS), 2023.
Kawar, B., Elad, M., Ermon, S., and Song, J. Denoising diffusion restoration models. Advances in Neural Infor- mation Processing Systems, 35:23593-23606. 2022.
Rout, L., Chen, Y., Kumar, A., Caramanis, C.,, Shakkottai, S. and Chu, W.-. Beyond first-order tweedie: Solving inverse problems using latent diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9472-9481, 2024.
Mardani, M., Song, J., Kautz, J., and Vahdat, A. A varia- tional perspective on solving inverse problems with diffu- sion models. In The Twelfth International Conference on Learning Representations, ICLR. OpenReview.net, 2024.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,, Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNett Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)2111-52,, 2015. doi:
Mittal, A., Soundararajan, R., and Bovik, A. C. Making "completely blind" image quality analyzer. IEEE Signal processing letters, 2003):209-11,, 2012.
2012.
Moliner, E., Lehtinen, J., and Välimäki, V. Solving audio inverse problems with a diffusion model. In IEEE In- ternational Conference on Acoustics, Speech and Signal Processing ICASSP, pp. 1-5, IEEE, 2023.
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., and Norouzi, M. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Moufad, B., Janati, Y., Bedin, L., Durmus, A., Douc, R., Moulines, E.. and Olsson, J. Variational diffusion pos- terior sampling with midpoint guidance. arXiv preprint arXiv:2410.09995. 2024.
Särkkä, S. Bayesian filtering and smoothing. Cambridge university press, 2013.
Sohl-Dickstedd J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Naesseth, C. A., Lindsten, F.. Schön, T. B., et al. Elements of sequential monte carlo. Foundations and Trends@ in Machine Learning, 12(3):307-39) 2019.
Song, B., Kwon, S. M., Zhang, Z., Hu, X., Qu, Q., and Shen, Solving inverse problems with latent diffusion models via hard data consistency. In The Twelfih Inter- national Conference Learning Representattins, ICLR. OpenReview.net, 2024.
Nazemi, A., Sepanj. M. H., Pellegrino. N., Czarnecki, C., and Fieguth, P. Particle-filterngg- sete-iieerinn-bseeee based latent diffusion for inverse problems. arXiv preprint 4rXiv:2408. 13868, 2024.
Raphaeli, R., Man, S., and Elad, M. SILO: Solving in- verse problems with latent operators. arXiv preprint arXiv:2501. 11746 2025.
Song, J., Meng, C., and Ermon, S. Denoising diffusion im- plicit models. In 9th International Conference on Learn ing Representations, ICLR, 2021a.

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Song, J. Vahdat, A., Mardani, M., and Kautz, Pseudoinverse- diffusion models for inverse prob- lems. In International Conference on Learning Represen- tations. 2023.
Wu, L.,, Trippe, B., Naesseth, C., Blei, D., and Cunning- ham, J. P. Practical and asymptotically exact conditional sampling in diffusion models. Advances in Neural Infor- mation Processing Systems, 36:3 16:2-337-33 -17223 2023.
Song, Y., Durkan, C., Murray, I., and Ermon, S. Maxi- mum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34: 1415-1428, 2021b.
Yu, J., Wang, Y., Zhao, C., Ghanem, B., and Zhang, J. Free- dom: Training-free meery--guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23174-23188 2023,
Song, Y.,
J., Kingma, D. P., Kumar, A.,, Er- mon, S., and Poole, B. Score-hased hased generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021c. URL http://ppneeeeeeeee////o//////tt
Zhang, B., Chu, W., Berner, Meng, C., Anandkumar, A., and Song, Y. Improving diffusion inverse problem solving with decoupled noise annealing. arXiv preprint 24Xi0:407.0152/, 2024.
Zhang, R., Isola, P., Efros, A. A.. Shechtman, E., and Wang. O. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
Song, Sohl-Dickstein, J., Kingma, D. P., Kumar, A.,, Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In 9th Inter- national Conference on Learning Representations ICLR. OvenReview.ne,, 2021d.
Sun, Y., Wu, Z., Chen, Y., Feng, B., and Bouman, K. Prov- able probabilistic imaging using score-based generative priors. IEEE Transactions on Computational Imaging, 00:1290-1305, 2024.
Tarantola, A. Inverse problem theory and methods for model parameter estimation. SIAM, 2005.
Tirer, T. and Giryes, R. Image restoration by iterative de- noising and backward projections. IEEE Transactions on Image Processing, 28(3): 1220-1244,, 2018.
Trippe, B. L., Yim, J.. Tischer, D., Broderick, T., Baker, D., Barzilay, R., and Jaakkola, T. Diffusion probabilistic mod- cling of protein backbones in 3d for the motif-scaffolding problem. In The Eleventh International Conference on Learning Representations, 2023.
Van Dyk, D. A. and Meng. X.-L. The art of data augmenta- tion. Journal of Computational and Graphical Statistics, 10(1):1-50) 2001.
Virieux, J. and Operto, S. An overview of full-waveform inversion in exploration geophysics. Geophysics, 74: WCC1-WCC26, 11 2009.
Wang, Y., Yu, J., and Zhang, J. Zero-shot image restora- tion using denoising diffusion null-space model. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from error visibility to struc- tural similarity. IEEE transactions on image processing, 130-)6600-62,, 2004.
12

---

Appendix of Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
A. Full Experimental Details
The experiments were carried out mainly using an NVIDIA A100 having 40GB and 80GB memory. In all experiments, used the DDIM formulation (Song al., 2021a), although LD- SMC can be applied with other sampling procedures. For all methods, we performed hyperparameter search 1j {0.05, 0.5, 1.0) and found that LD-SMC worked best with 1.0. For DPS and TDS we examined several scaling coefficient schemes for the prior mean update, including the ones proposed each of the corresponding papers, and found that our proposed update worked better for both. For all three methods (LD-SMC, DPS, and TDS) we searched for K1 E {0.3, 0.4, 0.5, 1.0, 1.5, 2.0}. For our method, we also performed grid search over 62 E {0.5, 1.5, 2.5). E (0, 100, 200, 333), and E (0.5, 0.75). In addition, when applying resampling, set the variance of Pe (ye (2,) to hyperparameters suggested in the paper and code did not yield good results. Hence, performed grid search over PSLD's hyperparameters "/ E {1e 4, le 3, le 2, 5e 2, 0.1, 0.2) and f) E {0.05, 0.1, 0.2, 0.5, 0.9). and used the best ones for both PSLD and PFLD. For ReSample, we found that using 0.0, the default value in the code, usually performed best. Also, we performed a grid search over 7, the scaling coefficient of the resampling step std in {4, 8, 16, 40, 80, 200, 400 For Late we used the default byperparameters suggested in the paper, except for super-resolution tasks where we made a grid search over 77 E {1e 4, 5e 5, le 5, 6, le 6, 5e 7, 7}. In addition, for comparability with other methods, we used observation noise of 0.01, instead of the default 0.05 in the code, and in ImageNet experiments used N 100 steps. To choose the best set of hyperparameters, we evaluated each method both visually and using the FID on sample of 64 images. Then, sampled 1024 images using the best configuration. Similarly ReSample, we found that applying an optimization process at the end of the sampling process in the latent space can sometimes improve visibility and metric values. We evaluated all models except LatentDAPS with and without this final optimization process and chose the best according to the FID. The optimization procedure was not applied to inpainting tasks since it created non-smooth changes at the boundaries of the box, making the images look non-natural.
ā to match the variance of (yo (e). For PSLD, in most cases, the default
For LD-SMC, present here the chosen hyperparameters:
Table 3: LD-SMC hyperparameters for tasks.
Dataset
Task
Ki
K2
8
Final Latent Optimization
Inpainting (Box)
1.0
2.5
333
0.75
No
Inpainting (Free-Form)
1.0
2.5
333
0.75
No
FFHQ
Gaussian deblurring
1.5
1.5
100
0.5
Yes
Super-resolution
1.0
2.5
100
0.5
Yes
Inpainting (Box)
2.0
2.5
333
0.75
No
Inpainting (Free-Form)
2.0
2.5
333
0.75
No
ImageNet
Gaussian deblurring
0.5
0
Yes
Super-resolution
0.3
2.5
333
0.5
No
B. Approximate Target Distributions
In Section 4.2.2 we presented approximate target distributions for the SMC sampling procedure. Here we present the derivation for each case, namely (1) / T, (2) 0 < < T, and (3) =
13

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
()) T=T:
po (o (Yr, & po po(yr(kr,,yo))o(2rr
o po (o ~ ps 2r Pa 2r p(rr).
Where, in the first transition we used Bayes rule, in the second transition we used the model dependencies and Bayes rule again, and in the last transition we make the additional approximation of conditioning on the posterior mean estimator.
(2) <<t<T:
(o Yer,, Yo) & po Y++1.T (o0e1rrroooo00(4//1100 Yo)
= po((y]/z))po 3 howp
po(ye|ze)po Ze+1. [1
Po (Yo z,)
po((ylle))
pa (Ze z2 (Z: M141.
pe yoo Ze-1
po (y Pa (yo Zt)
po (2
Po (yol Where, in the first transition used Bayes rule, in the third transition used the Markovian assumption, in the forth transition used Bayes rule again, and in the last transition we make an additional approximation and condition the posterior mean estimator for both time t and time t + 1.
0
Pe (eZooo//ViTT... = pe ZI.TT wor.)oo
po (oozo/z1.30 z1,
(y zo)
po (Zo 11)p)ppp0 (zu.r (2 Yorr)
(y z1
po yyo Zo)
pe Zu zn )pe (2irr .o.r)
(yo z
Where, in the third transition we used Bayes rule, and in the last transition we make an additional approximation and condition on the posterior mean estimator.
C. Asymptotic Accuracy of LD-SMC
Here we provide a proof that 2g samples can be made arbitrarily accurate according to our model design. Our proof composed of three parts, all of which have been previously established in the literature. We will restate them here and accommodate them our setting. The three parts are, (1) augmenting the model with auxiliary random variables, ergo performing a completion of the desired marginal density po(zo lyo) (Definition 10.3 in (Robert & Casella, 1999)), (2) Proving asymptotic accuracy of the SMC procedure following similar lines of Theorem 2 in (Wu et al., 2023), (3) Showing sufficient conditions that the Markov chain generated by the Gibbs sampling procedure is ergodic and hence pe (zolyo) is the limiting distribution from which Zn is are sampled (Theorem 10.6 in (Robert & Casella, 1999)).
First of all, here we provide a concise version of our Gibbs sampler presented in Section 4.2. We denote by the index of the Gibbs iterations:
1. Initialize 2
2. For j O, O,.... J 1:
(a) Sample, vit
pa pa yo). No). Yo) .o) using SMC.
(b) Sample, 2 ~ po 11
Definition C.1. (Robert Casella, 1999). Given a probability density Po(zolyo), a density that satisfies gzz. ZI.T, Ni.TyyI..Ve (oo is called a completion of pa(zolyo).
We denote by ((.T)) the Markov chain generated by our proposed Gibbs sampler. Similarly, denote by (z()) the corresponding subchain.
14

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Theorem C.2. (Robert & Casella, 1999). For the Gibbs sampler described in Section 4.2, if (a (ir) is ergodic, then the distribution g is a stationary distribution for it and
is the limiting distribution of the subchain (z))-
Proof. The support of g is Rd, with d dim (o::T and hence is connected. Also, since each conditional distribution in the Gibbs sampling process is Gaussian multiplication of Gaussian densities, they are all strictly positive. Following Lemma 10.11 in (Robert & Casella, 1999), ( vi.r) is irreducible and aperiodic, i.e., it is ergodic.
Specifically, Theorem C.2 shows that in order to obtain samples from the marginal distribution po (Zaa/yo)) we need to sample FI.7 and Zo:r from the conditional distribution according to the model at each iteration. Sampling yı:T variables is straightforward as it requires sampling from independent Gaussian distributions, that is, po (y (D(())))()))) I"I) for all / 0. On the other hand, sampling 20:7 requires sampling using SMC procedure. As show next in the large particle limit, samples from po Zo:r Yo.r) are accurate.
To prove the asymptotic accuracy of the SMC procedure in LD-SMC, we adopt the formulation of (Wu et al., 2023). In what follows, to prevent cluttered notation, we drop the index of the Gibbs iteration and note that all quantities are those of a specific iteration. Specifically, here we first reiterate the following 3 important quantities, the prior distribution po (zarr). the proposal distributions v. (Ze [z)+1), and the weighting functions We (Zz, Zz+1) and then proceed to the proof.
Prior distribution. Let pe (zo:T) denote the diffusion generative model defined according Eq. 1. Then, the Markovian structure of the prior diffusion model goes as follows,
ppo =N(Ho(ZotL
01+2 21-1 v ae e-s res (2)+t) + VI-āu ā. - -221 c6(2i-1,t (e)
=N(zolVan
1<t<T
vai+1
po(zr)=NN(0 = -O(0.))
t=
Proposal distributions. Denote the proposal distribution for timestep & < T as m. (Z(l = (M,,, with parameters
S. =
(8)
m;-- Mu(2++1,+11) (7e V 22 log po(yolze+1) + V (0(2111,12) log go(y, go(y.12x+1) (2++))).
Where (o (()g))((((7((24))))) (1 a.+ I)) and qo (y = (y, L(D(La (Ze+1, + 1))), 2)), and no d are finite scaling coefficients. The distribution of the proposal at time T, nr (ZT). from which the initial sample is taken, is set to the diffusion prior distribution po(2r) = (0. I).
Weighting functions. The unnormalized weighting functions for all timesteps are summarized as follows,
w Po |z (y (z)) we (Zz, 2(4 (Z 21) = po (yo Z (yoo /0 (/o (z)oo())))))))
(ol (re
where py(yllz) (V(y))A(D(()))) TI).
In SMC the proposal distributions and weighting functions define a sequence of intermediate target distributions (vr)T=. TT= defined as follows,
T
T
vs ((eT)
(T(ZT) Le
1
nт (ZT)
II r'=t
The (Zer mer+1))
ur (2r)
II t'=t
th (2e mx+11)
(9)
Importantly, if for given weighting functions and proposal distributions the final target distribution Po coincides with the desired posterior distribution p( (..or)lyo.r), then samples that are approximately distributed according to p(zo.T Yo.r) can be obtained (Doucet et al., 20 Naesseth et al., 2019). The approximation becomes accurate in the limit of large number of particles. For 0. plugging in Eq. 9 the proposal distributions and weighting functions and observing that all appearances
15

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
of the proposal distributions besides that of time T cancel out, we obtain:
TT-1 polyi ze (yoo ze+1) po (o(ouzzo)
1
(zo:r)
Po
Zr (o((r Zr (y)oo))
po(yolzi) po po(zolz))
Co
(10)
T
T-1
Lo
II
pa
(ye
II 1=0
pa (a(2) 2e+ (po(z))
= Lo
L. Po po (Zo.T) = pa (20
t=
Where in the second equality pa terms cancel out and in the last equality we applied Bayes rule.
Now that established necessary conditions to sample from po pozzorlly (yoor), the following theorem from (Chopin et al., 2020) characterizes the conditions under which SMC algorithms converge. We adopt the formulation of (Wu et al., 2023) and present it here, adapted to our case.
Theorem C.3. (CChopin et al., 2020) - Proposition 11.4). Let (zu.r: ) be the sequence of particles and final weights returned by the last iteration of the SMC algorithm with N particles and using multinomial resampling. If the weight functions of all timesteps ( are positive and bounded, then for every vo- vy-measurable function & of zu:r.
(i)
lim
wi o(Z
/ ((oo:T))o(zorrdddl
N-
with probability one.
Specifically, taking o ((aoT) = I E E] for any mn-measurable set E implies the convergence of Px = E.rwe 5 20 (E). Here, 82 is the Dirac measure defined for given point and meaauraaee set E. The following foir proposition characterizes the conditions under which Theorem C.3 applies in our case.
(8) 8 to (E) be a discrete distribution over particles with ((Z. w J returned Proposition C.4. Let PN)) = E wh An. by the SMC procedure in Section 4.2.2 with N particles. Assume for all t::
(a) The likelihood functions po(y)]zz) ((Y)))((((2))) I),, the ratios oo po(yollz)./oosollzaaa and po (y(ll)))), are all positive and bounded.
(b) log Po (yolzz) is continuous and has bounded gradients in Zą-
(c) log qo 1z is continuous and has bounded gradients in ph (o(2i1,, t 1).
(d) The proposal variance larger than the prior diffusion model variance, namely o? > of.
Then PN converges setwise to po (Zo:r (Y:r) with probability one, that is for every set F,, lim S Po (o(zzrl/y (Vor)ddo.rr
Note that here, unlike (Wu al., 2023), we do not need assume that (yoozz)) distribution. Here, however, we add assumption (c). Assumptions (b) and (c) the strongest assumptions they may not apply even for linear transformations. But, for sufficiently smooth decoder and diffusion model in the input (z, or Mo (Z + 1)) with uniformly bounded gradients, the assumptions will hold.
due to our revised posterior
Proof. To prove the statement, we need to show that (1) the target distribution of the SMC procedure at time / = is Po Po(Zo:z Morlyyor)) and (2) the weighting functions w; (z( 2:41) of all timesteps are all positive and bounded. The first condition was already established in Eq. 10, hence we proceed to the second condition.
Since all the weights are defined through multiplications of density functions, they are strictly positive. To show that they bounded, first note that, by assumption (a), wr (zr) ) is bounded. To show that the intermediate weights are bounded it is enough to consider the log transformation of the unnormalized weights.
log po (y: (x)) + log
(y(al)))) + log Pa (2)))) Po (..1) (. Pc:1) 0 0<t< 1 T,
log wi (1, 2 =
log pe (y (zo) + log mo (2))
Pa(2n
0.
16

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
66
22.0
W
23.62
52.5
64
21.8
PSNR
52.0
23.60
PSNR
FID
62
60
21.6
51.5
23.58
1
5
3
4
5
Gibbs Iterations
Gibbs Iterations
(a) Super-Resolution on ImageNet
(b) Gaussian Dehlurring on ImageNet
Figure 6: FID and PSNR values when increasing the number of Gibbs iterations (with 1 particle).
By assumption (a), log po(y) (z) and log ( z(41) are bounded for all t. We will show next that the log 12+1)/ (z, (x41) terms are all bounded based on assumptions (b) (d). We denote by He, Ho (Zc41, + 1),
po(z,|Ze11)
|2TO 2 1
0.5
11
log
log
| [
0.5 (xp(-(2o{++)
-| 1ze m [2]
-2 0 12 14l12 8. 10me-mm113 mx1
Expanding, || m, + M4l 1 || 11112 + 2(zz 14 m.) + 1114 m.
= -- 3,41 )| 12 11112 20, -2 - (2: ,1 14 m.) 0 11 m. 1
Notice, 1|1r - m, [/ |1 a., log po ( z(+1) 1) + VvW
n. log ( Ze+ IlB oc by applying Minkowski inequality and using assumptions (b) and (c)
c - o141)]l2, - m,11 - 25, 11,=,,,31
pc, Me -m))
Apply Cauchy-Schwand inequality.
S - - 11l12 + , 0 114
m2 - Mel12]
12
Apply the inequality
a - be v for =0 -2 - > 0 by assumption (d)
0+4
2
2a
< a.Hllum 11 m.
<0.
2(0 0.41)
Where the last inequality again due to assumptions (b) and (c). Here, C denotes an equality up to constant.
D. Proposal Distribution Scaling Coefficients
Recall that our proposal distribution (Eq. 7 in the main text) is made up of two elements. These elements are scaled by two coefficients, y and Ag. Here, we provide an explicit formula for these coefficients. We found that our proposed scaling works better than common procedures used the literature. For consistency with baseline methods, we also used our proposed scaling approach for DPS and TDS (which corresponds setting s = 0), since these methods apply similar update rule. We tried use it for other baselines but it did work well for them.
Let g! = Vz.llyo (z)(z((z)))) and gì = V (o(0mn1, - A(D(p, 1))) We set the scaling
coefficients 7/t and d according the following scheme: n [t2s] '61 maz(Iss:]..] +1ppal W2 (1 p) mart g 1)
and X 1 pss) K2 p mar(Iss]l..)) where p. K1, Ni are all hyper-parameters:
17

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
E. Computational Cost
We add here a comparison between the compared methods in average run-time (seconds) and memory (GB) over 10 trials for sampling single image on ImageNet box inpainting task. For TDS and PFLD we use particles. For LD-SMC, we inspect several variants that differ in the number of particles and Gibbs iterations. The results are shown in Table 4. From the table, the run-time is roughly linear in the number of particles and Gibbs iterations. Yet, importantly, it can be controlled by the practitioner to trade off performance (which can be good with one Gibbs iteration and one particle) and computational demand. In addition, we note that our code is not properly optimized and improvements can be made to it.
Table 4: Average run time and memory demand on ImageNet box inpainting task.
Method
Run Time (Sec.)
Memory (GB)
LDPS
105.5
8.123
LTDS
418.5
19.86
ReSample
333.4
5.769
PSLD
144.5
9.473
PFLD
469.8
26.51
LatenDDAPS
67.82
4.745
LD-SMC (1 particle)
136.3
9.213
LD-SMC (3 particles)
375.1
15.11
LD-SMC (5 particles)
537.2
21.16
LD-SMC (10 particles)
1013.
35.78
LD-SMC particle; 2 Gibbs iterations)
271.2
9.213
LD-SMC (1 particle; 4 Gibbs iterations)
541.0
9.213
F. Further Ablation Studies
Number of Gibbs iterations. Recall that in the main text we used only one Gibbs step throughout. Here we examine the effect of using more Gibbs steps in Figure 6. We do on two tasks, the first is super-resolution and the second in Gaussian deblurring, both on ImageNet. In super-resolution, we observe an improvement in the PSNR with the number of Gibbs steps while suffering from reduction in the FID. Interestingly, this experiment shows that we can obtain a comparable PSNR to ReSample while having a substantially better FID compared to it on this task. In Gaussian deblurring there is a less clear trend, nevertheless, the figure shows that the FID can be improved while maintaining roughly the same PSNR value. We speculate that the difference between these two cases stems from the different values used in these experiments. Taking 0 tends generate smoother images that are better aligned with higher values of distortion metrics such PSNR.
Proposal distributions. In Table 5 we compare different choices of proposal distributions. Specifically, we compare to (1) the naive choice of using the prior as a proposal distribution, i.e., a bootstrap filter (Gordon et al., 1993); and (2) taking the DPS step as a proposal distribution, which corresponds to setting = The table shows a clear advantage to the proposal distribution used in this study and presented in Section 4.2.2.
Table 5: FFHQ. Proposal distribution ablation. Box inpainting on 1024 test examples using N = 5 particles.
Perceptual Quality
Distortion
FID (4)
NIQE (L)
PSNR (+)
SSIM (t)
LPIPS (4)
Prior proposal
62.35
7.628
12.60
0.365
0.671
DPS proposal
39.94
7.558
24.20
0.814
0.235
LD-SMC
33.87
7.066
24.10
0.821
0.211
18

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
t=1000
t=800
t=600
t=400
t=300
t=200
t=100
t=1
t=0
Inpainting
(Box)
Gaussian
Deblurring
Super Resolution
Figure 7: Evolution of over time for different tasks according to forward process of DDIM.
G. Forward Process
In Figure 7 we present the evolution of the auxiliary labels ye time part of the forward process according to proposed sampling procedure in Section 4.2, steps (a) & (b). From the figure, we observe gradual cleaning of noise in the auxiliary labels when advancing from time / = 1000 to time = O.
H. Full Results
Table 6: ImageNet. Box inpainting 1024 test examples.
Perceptual Quality
Distortion
FID (L)
NIQE (1)
PSNR (1)
SSIM (t)
LPIPS (L)
LDPS
65.04
7.935
19.51
0.665
0.379
LTDS
64.74
7.907
19.49
0.665
0.378
ReSample
90.32
8.464
18.16
0.695
0.318
PSLD
71.15
8,042
18.10
0.565
0.434
PFLD
72.83
7.933
18.09
0.564
0.436
LatenDDAP
98.24
11.36
19.98
0.704
0,394
LD-SMC particle)
$1.49
6.878
19.38
0.672
0.326
LD-SMC (5 particles)
50.67
6.891
19.42
0.672
0.325
19

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Table 7: ImageNet. Free-form inpainting 1024 test examples.
Perceptual Quality
Distortion
FID (L)
NIQE (1)
PSNR (1)
SSIM (t)
LPIPS (L)
LDPS
53.47
7.867
23.76
0.722
0.334
LTDS
52.75
7.884
23.74
0.721
0.334
ReSample
44.15
7.104
22.78
0.778
0.248
PSLD
62.38
8.037
21.65
0.610
0.411
PFLD
63.34
8.026
21.59
0.609
0.414
LatentDAPS
68.65
9.625
24.75
0.766
0,330
LD-SMC particle)
38.21
6.969
23.54
0.727
0.285
LD-SMC (5 particles)
36.18
6.671
23.43
0.727
0.278
Table 8: ImageNet. Gaussian deblurring on 1024 test examples.
Perceptual Quality
Distortion
FID (L)
NIQE (4)
PSNR (t)
SSIM (t)
IPIPS (L)
LDPS
52.48
6.855
23.61
0.615
0.383
LTDS
50.82
6.695
23.57
0.614
0.379
ReSample
46.45
7.411
24.36
0.639
0.353
PSLD
60.68
6.599
21.94
0.506
0.417
PFLD
60.94
6.733
21.78
0.496
0,421
LatenIDAPS
77.09
10.55
24.44
0.659
0,417
LD-SMC particle)
52.48
6,855
23.61
0.615
0.383
LD-SM DMC (5 particles)
52.29
6.791
23.61
0.615
0.383
Table 9: ImageNet. Super-resolution (8x) on 1024 test examples.
Perceptual Quality
Distortion
FID (L)
NIQE (1)
PSNR (t)
SSIM (t)
LPIPS (L)
LDPS
61.02
6.514
21.65
0.523
0.439
LTDS
59.12
6.270
21.59
0.520
0.435
ReSample
87.65
8.290
22.05
0.532
0.491
PSLD
66.56
7.669
20.83
0,480
0.489
PFLD
64.72
7.685
20.83
0.479
0.492
LatenDDAPS
104.6
22.38
0,566
0.489
LD-SMC partiele)
58.06
6.243
21.51
0.524
0.434
LD-SMC (5 particles)
57.89
6,238
21.50
0.520
0.433
Table 10: FFHQ. Box inpainting on 1024 test examples.
Perceptual Quality
Distortion
FID (L)
NIQE (})
PSNR (t)
SSIM (t)
LPIPS (L)
LDPS
39.81
7.592
24.15
0.814
0.236
LTDS
39.57
7.602
24.24
0.814
0.236
ReSample
86.79
7.142
19.75
0.815
0.230
PSI.D
39.68
6.544
22.31
0.774
0.246
PFL.
39.06
6.509
22.40
0.774
0.245
LatenDDAPS
60.24
9,999
24.91
0.838
0.257
LD- partiele)
33.37
7.032
23.98
0.819
0.212
LD-SMC (5 particles)
33.87
7.066
24.10
0.821
0.211
20

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
Table 11: FFHQ. Free-form inpainting on 1024 test examples.
Perceptual Quality
Distortion
FID (4)
NIQE (4)
PSNR (t)
SSIM (t)
LPIPS (4)
LDPS
40.17
7.609
27.95
0.858
0.212
LTDS
39.78
7.578
27.95
0.858
0.212
ReSample
37.01
6.622
26.31
0.891
0.151
PSLD
36.26
6,835
26.19
0.823
0.216
PFLD
36.43
6,817
26.35
0.825
0,215
LatentDAPS
54.40
8.766
29.23
0.883
0.223
LD-SMC (1 particle)
33.67
7.034
27.35
0,859
0.194
LD-SMC particles)
33.60
7.021
27.35
0.859
0.194
Table 12: FFHQ. Gaussian deblurring on 1024 test examples.
Perceptual Quality
Distortion
FID (L)
NIQE (4)
PSNR (t)
SSIM (t)
LPIPS (1)
LDPS
29.30
6,538
28.03
0.775
0.237
LTDS
30.23
6,553
27.93
0.772
0.238
ReSample
39.80
7.441
28.45
0.763
0.275
PSLD
36.31
6,802
24.02
0.633
0.341
PFLD
37.16
6.751
23.96
0.628
0.343
LatenDAPS
54.28
9.496
29.70
0.831
0.283
DMC (1 particle)
29.19
6,575
28.37
0.789
0.232
LD-SMC (5 particles)
29.47
6.528
28.34
0.787
0.233
Table 13: FFHQ. Super-resolution (8x) on 1024 test examples.
Perceptual Quality
Distortion
FID (L)
NIQE (4)
PSNR (t)
SSIM (t)
LPIPS (L)
LDPS
29.64
6.412
25.48
0.701
0.282
LTDS
30.45
6.412
25.38
0.698
0.284
ReSample
59.23
7.307
25.55
0.661
0.356
PSLD
40.33
6,803
23.66
0.615
0.347
PFLD
38.11
6.832
23.69
0.617
0.345
LatentDAPS
70.24
10.17
26.87
0.760
0.344
LD-SMC particle)
30.02
6.426
25.52
0.706
0.277
LD-SMC (5 particles)
30.62
6.455
25.49
0.703
0.278
21

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
I. Additional Image Reconstructions
True
Observation
ID-MC (Ours)
LDPS
LTDS
ReSample
PSLD
PFLD
LatentDAPS
n0
Figure 8: Comparison between LD-SMC and baseline methods on inpainting of FFHQ images.
22

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
True
Observation
LD-SMC
True
Observation
LD-SMC
Figure 9: Gaussian deblurring. LD-SMC reconstruction of images from FFHQ (left) and ImageNet (right).
23

---

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo
True
Observation
LD-SMC
True
Observation
LD-SMC
Kaehler
Eadelier
Figure 10: Super-resolution. LD-SMC reconstruction of images from FFHQ (left) ImageNet (right).
24


Reviews:
[{"review_id": "qpA3Te5MBs", "reviewer": "Reviewer_sS3Y", "summary": "This paper studies inverse problems using sequential Monte Carlo sampling in a latent space. Generative models are great priors for the inverse problems. Although diffusion models achieve great performance, due to their computationally expensive reverse process and sequential nature, leveraging diffusion models in inverse problems raises technical challenges. To address the technical difficulty in sampling from the exact posterior distribution, the authors proposed sequential Monte Carlo-based sampling method. The authors proposed a new posterior approximation and proposal distribution. The proposed method achieves competitive performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort put into the review. We are encouraged that the reviewer regarded our approach as *Interesting exploration* and overall appreciated it. We address the reviewer’s comments in the following. All answers and suggestions will be incorporated in the paper.\n\n**(1)** \n> ”More baselines: RED-diff, DDRM, DDNM^+ \\PI GDM [...]” \n\nThank you for pointing us to these methods. We will make sure to reference them in the paper. These baselines were not included since the values reported in them are based on a different diffusion model and hence are not comparable. As such, due to computational limitations, we picked the set of recent works that were closest to our approach and could be applied with latent diffusion, and we ran all of the baselines ourselves under the exact same experimental setup. Please see comment (3) to reviewer yJi6 for additional baselines added here in the rebuttal.\n\n\n**(2)** \n>”Weak experimental results [...] poor performance in deblurring [...] especially in distortion metrics. Also, experiments in other inverse problems are missing [...]”\n\nThank you for the comment, but we believe otherwise regarding the results of LD-SMC. First, we would like to stress that there can be a trade-off between metrics that represent perceptual quality and those that represent distortion (Blau & Michaeli, 2018). We gave a stronger emphasis to the perceptual metrics, since the goal, as we see it, is to obtain high-quality images. Second, in image generation, the metrics can be biased, and visual inspection should be taken into account as well.\n\nWhile we did not achieve SoTA results on Gaussian deblurring, visually though most methods yield good and comparable reconstructions. This makes the distinction between methods on Gaussian deblurring very nuanced, and it is not obvious whether the metrics used are sufficient to capture the differences between all methods. We, therefore, focused our efforts on the more challenging inpainting task where LD-SMC significantly outperforms baseline methods, both visually and in terms of perceptual metrics.\n\nRegarding additional tasks, we present results for super-resolution in the paper. In addition, we add here results for free-form inpainting on ImageNet and FFHQ following the protocol suggested by Saharia et al., (2022). For numerical results please see Tables 3 & 4 in https://tinyurl.com/p6hh6bjy. Qualitative examples can be seen at https://tinyurl.com/2u6mmj2x\nFrom the tables, LD-SMC outperforms baseline methods in terms of perceptual metrics and is comparable to baseline methods in terms of distortion metrics. Visually, as in box inpainting tasks, LD-SMC reconstructions better preserve fine details compared to baseline methods. In addition, despite having good values in several metrics, ReSample presents artifacts that make the images look non-natural.\n \n\n**(3)**\n> “No analysis on the main contributions: approximation and proposal distribution.”\n\nThank you for this valuable feedback. Please see comment (7) to reviewer 326R where we torch upon this point. Regarding the proposal, the theory is very general and allows a large leeway to pick an appropriate distribution. Ideally, we would want a proposal that generates samples that are in agreement with $\\mathbf{y}_0$ and have a high likelihood. We experimented with numerous formulations and we picked the one that worked best. \n\n**(4)** \n> Difference from existing SMC methods\n\nThank you for the suggestion. Please see comments (3) to Reviewer yJi6 in which we discuss other SMC methods.\n\n\n**(5)** \n> ”Any quantitative results to separately evaluate the quality of restored images in large patterns and local details.”\n\nWe believe the common metrics, such as FID and NIQE reflect that. It can be witnessed visually as well.\n\n**(6)** \n> “Resample is overall comparable or often outperforms LD-SMC”\n\nIn terms of perceptual metrics, LD-SMC outperforms ReSample (13/16 comparisons in favor of LD-SMC). In terms of distortion metrics, indeed ReSample has an advantage (15/24 comparisons in favor of ReSample). Visually ReSample presents significant artifacts in all tasks and especially in in-painting. An additional visual comparison on ImageNet Gaussian deblurring is here: https://tinyurl.com/54pec8kn. Zooming in on ReSample images, noticeable artifacts can be seen in the generated images.\n\n**(7)** \n> “Compared to other simple proposal distributions, the performance gain is not fully analyzed”\n\nThank you for this suggestion. Table 5 in https://tinyurl.com/p6hh6bjy shows a comparison of LD-SMC with the proposed proposal distribution and two alternatives, DPS as a proposal, namely taking the hyper-parameter $s=0$, and the prior as a proposal distribution. \nFrom the table, LD-SMC proposal outperforms both alternatives in most metrics.\n\n**(8)**\n>Computational cost of LD-SMC\n\nThank you for this suggestion, please see comment (4) to Reviewer AxT3 who also raised this point.", "claims_and_evidence": "Claim 1. The authors claim that the proposed method is effective in capturing both large-scale patterns and local fine-grained details. \n\nIncorporating diffusion models into Inverse problems is technically challenging especially for the joint posterior of original image x at all time points, namely, p(x_0, … x_T | y). In the literature, recent works have attempted to address this challenge in various ways. One line of approaches based on P(Y|E[x_0|x_t]) captures large scale patterns and the other line of approaches with auxiliary y_{1:T} captures fine-grained local details. The authors claimed that the proposed method achieves the best of both. Figure 2 demonstrates the traits of two different approaches and the proposed method outperforms both lines of methods in the in-painting task. Figure 4 shows another results in deblurring. But the quantitatively is not analyzed.", "methods_and_evaluation_criteria": "Method. The proposed method is reasonable and is a variant of recent approaches. The sequential Monte Carlo has been proved effective in sampling with diffusion models. The approximation in Section 4.2.2. is introduced but no analysis of this approximation is provided. \n\nEvaluation. The evaluation was performed similarly to previous works. 256 x 256 ImageNet and FFHQ 256 x 256 with 1024 samples were used. The authors have reported the performance in various evaluation metrics: FID, NIQE, LIPIPS, SSIM, PSNR. The authors mainly presented  Perceptual Quality metrics and distortion metrics (PSNR, SSIM) are presented in the appendix.\n\nOverall, the proposed method and evaluation are reasonable.", "theoretical_claims": "No theoretical results are presented.", "experimental_designs_or_analyses": "The experimental setting is pretty standard. No discussion is needed. However, more baselines can be included.", "relation_to_broader_scientific_literature": "No relation to any findings in scientific literature is explicitly discussed by authors and I did not observed either.", "essential_references_not_discussed": "More baselines: RED-diff, DDRM, DDNM^+ \\PI GDM and so on. To make the comparison more comprehensive, the authors may want to include other diffusion model-based inverse problem methods even though they are not performed in the latent space.", "other_strengths_and_weaknesses": "Strengths\n\n1. Interesting exploration. This paper studies a new sampling scheme, inspired by the prior work on sequential Monte Carlo sampling, for inverse problems with a new proposal distribution and its update.\n\nWeaknesses\n\n1. Weak experimental results. Although the proposed method achieves competitive performance in inpainting, its efficacy in other inverse problems is not fully proven. The method exhibit poor performance in deblurring and the proposed method did not outperform baselines, especially in distortion metrics. Also, experiments in other inverse problems are missing such as super-resolution, motion deblurring, colorization and so on.\n2. No analysis on the main contributions: approximation and proposal distribution. The accuracy of approximation and efficiency of the proposal distribution (computational cost, distance to posterior distribution or gap induced by the proposal distribution, the number of rejections if rejection sampling schemes are utilized.) are not discussed.", "other_comments_or_suggestions": "In the main paper, if the performance in all metrics are provided in the main paper, it would be better for readers to understand the strengths and weaknesses of the proposed method.\n\nThe difference between the proposed method and existing SMC methods needs to be highlighted.", "questions_for_authors": "Q. Any quantitative results to separately evaluate the quality of restored images in large patterns and local details. Regarding distortion metric, PSNR, SSIM, LPIPS in Table 4 and other Tables in the supplement, Resample is overall comparable or often outperforms LD-SMC. Please provide more explanation.\n\nQ. Analysis of proposal distribution? If a proposal distribution is closer to the posterior distribution and easy to draw sample, then the proposal distribution is ideal. The effect of proposal distribution update is reported in Table 3. But this kind of thorough analysis is not presented. Also, compared to other simple proposal distributions, the performance gain is not fully analyzed.\n\nQ. Similar to other methods, computational costs (the number of parameters, latency) are not compared. Does the proposed method induce additional computational overhead?\n\nQ. Why does the proposed method exhibit weak performance in Gaussian Deblurring?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gSseAdpy8p", "reviewer": "Reviewer_AxT3", "summary": "The work proposed a Sequential Monte Carlo based sampling algorithm for solving imaging inverse problems with latent diffusion model. The writing is clear and easy to follow.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort put into the review. We are encouraged that the reviewer appreciated LD-SMC perceptual quality and the evaluation part. We address the reviewer’s comments in the following. All answers and suggestions will be incorporated in the paper.\n\n**(1)**\n> “This work suffers from a significant loss in distortion quality.” \n\nAcross 24 comparisons of distortion metrics (including free-form inpainting in comment (2) to Reviewer sS3Y), LD-SMC is first 3 times, second 10 times, third 10 times, and fourth one time. Hence, in our view, LD-SMC is comparable to baseline methods in distortion metrics. \nIn general, there can be a trade-off between metrics that represent perceptual quality and those that represent distortion (Blau & Michaeli, 2018). As mentioned in the paper, we gave a stronger emphasis to the perceptual metrics, since the goal, as we see it, is to obtain high-quality images. Conversely, we could have put more emphasis on distortion metrics, for example when performing a hyper-parameter search, but it would have come at the expense of the perceptual quality.\n\n**(2)** \n> Moving distortion metrics to the main text\n\nThank you for the suggestion. Due to lack of space, we report in the main text the FID and NIQE which are considered perceptual metrics, and LPIPS, a distortion metric. Note that it is common in the literature to report only the FID and LPIPS in the main tables, (e.g., Chung et al., 2023b; Dou & Song 2024). Nevertheless, we will include all the metrics in the next version of the paper. \n\n**(3)** \n> “The number of particles used in the proposed method is 5, is there any reason for choosing this number?”\n\nThank you for this suggestion. We chose 5 particles because of run-time and memory considerations. In SMC one can expect an improvement in the performance when increasing the number of particles, yet as the reviewer mentioned, in the latent space it can be costly due to encoder-decoder operations. Following the reviewer's suggestion, Table 6 in https://tinyurl.com/p6hh6bjy shows an ablation for the number of particles. The table shows a general trend of improvements in perceptual and distortion metrics when increasing the number of particles. Importantly, even taking only one particle results in favorable performance for LD-SMC compared to baseline methods at a similar computational cost. Per comment (7) to Reviewer 326R, please note that there are small changes from the results reported in the paper for LD-SMC.\n\nIn addition, we examine the performance of LD-SMC using one particle and multiple Gibbs iterations in Tables 7 & 8 at https://tinyurl.com/p6hh6bjy. From the tables, results can be improved even when using one particle (which reduces the computational demand of the method) by using multiple Gibbs iterations.\n \n\n**(4)** \n> “Please provide a computational cost comparison with baselines methods including memory and inference time per image.”\n\nThank you for this suggestion.\nWe add here a comparison between the methods in average run-time (seconds) and memory (GB) over 10 trials for sampling a single image. For LD-SMC we inspect several variants, including using only 1 particle and multiple Gibbs iterations. As can be seen in comment (3), using 1 particle allows to almost match the results of LD-SMC with 5 particles in inpainting tasks and when allowing to take multiple Gibbs iterations the performance can be further enhanced in Gaussian debluring and super-resolution. From the table, the run-time is roughly linear in the number of particles and Gibbs iterations. Yet, importantly, it can be controlled by the practitioner to trade off performance (which can be good with one Gibbs iteration and one particle) and computational demand. In addition, please note that our code is not properly optimized and we believe that large improvements can be made in the run time of LD-SMC.\n\n| | Run time (sec.) | Memory (GB)\n|----|---|---|\n| Latent DPS | 105.5 | 8.123 \n| Latent TDS | 418.5 | 19.86\n| ReSample | 333.4 | 5.769\n| PSLD | 129.8 | 9.590\n| LD-SMC (1 particle) | 136.3 | 9.213\n| LD-SMC (3 particles) | 375.1 | 15.11\n| LD-SMC (5 particles) | 537.2 | 21.16\n| LD-SMC (10 particles) | 1013  | 35.78\n| LD-SMC (1 particle; 2 Gibbs iterations) | 271.2 | 9.213\n| LD-SMC (1 particle; 4 Gibbs iterations) | 541.0 | 9.213\n\n**(5)** \n> “By using the weighting and resampling [...] an improvement in the worst-case performance, [...] I am wondering if authors can add an experiment or discussion to cover this part.”\n\nThank you for this valuable feedback. Indeed the weighting and resampling correct for the approximations in the proposal distribution and posterior (section 4.2.2). We formalize this claim in comment (7) to Reviewer 326R.", "claims_and_evidence": "The claim is solid. Despite achieving the highest perceptual quality, this work suffers from a significant loss in distortion quality. This critical trade-off needs to be explicitly addressed in the main paper (moving distortion quality into the tables in the main paper). Failure to do so would be misleading.", "methods_and_evaluation_criteria": "Yes, the evaluation is fair and well-designed. To help better understanding the computational cost, please provide a computational cost comparison with baselines methods including memory and inference time per image.", "theoretical_claims": "There is no theoretical claims.", "experimental_designs_or_analyses": "Yes, the comparison is fair and well-designed.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "It has discussed most of the important works.", "other_strengths_and_weaknesses": "Weakness:\n(1) This work presents a method that builds upon existing techniques, notably TDS, by extending the sampling process to the latent space, with a method introduced in Resample baseline. While this extension successfully introduces Sequential Monte Carlo to the latent space and demonstrates performance improvements, it comes at a significant computational cost. The reliance on multiple particle sampling and decoder operations during guidance presents a practical challenge.", "other_comments_or_suggestions": "No more.", "questions_for_authors": "(1) The number of particles used in the proposed method is 5, is there any reason for choose this number?  An ablation study and a corresponding discussion will be helpful to inform if scaling up the particles can further improve the performance or not.\n(2) By using the weighting and resampling method as introduced in this paper, I think there will be a improvement in the worst-case performance, which may be the main reason for the improvement in the table. I am wondering if authors can add a experiments or discussion to cover this part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rZkPx9fvXZ", "reviewer": "Reviewer_yJi6", "summary": "The authors propose a sampling method based on Sequential Monte Carlo (SMC) in the latent space of diffusion models. The proposed approach leverages the forward process of the diffusion model to introduce additional auxiliary variables (e.g. noisy measurements), followed by SMC sampling as part of the reverse process.\n\nThe method is evaluated on commonly used benchmarks: ImageNet and FFHQ, demonstrating improvements in Gaussian deblurring, inpainting, and super-resolution tasks. The results suggest that incorporating SMC in the latent space of diffusion models may enhance sampling efficiency and reconstruction quality at the cost of expensive gradient updates (see Eq. (4)) and hyper-parameter tuning (see Sec 4.2.3).\n\nWhile the approach is promising, further clarification on the computational trade-offs and runtime analysis would strengthen the contribution. Additionally, an ablation study on the role of auxiliary variables and each gradient update from Eq. (4) in improving sample quality could provide deeper insights into the method's effectiveness.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort put into the review. We are encouraged that the reviewer found our approach specifically and SMC in general promising. We address the reviewer’s comments in the following. All answers and suggestions will be incorporated in the paper.\n\n**(1)** \n> “Further clarification on the computational trade-offs and runtime analysis”\n\nThank you for this valuable suggestion. Please see comment (4) to Reviewer AxT3.\n\n**(2)** \n> “The role of auxiliary variables and each gradient update from Eq. (4)”\n\nThank you for raising this point. Please see the following figures on box inpainting that analyze that. ImageNet- https://tinyurl.com/yrht7az9; FFHQ - https://tinyurl.com/2snc4vdz. Similar to Table 3 in the paper the figures show the tradeoff between FID and PSNR when varying $s$, the hyper-parameter that balances between the terms. Taking $s=0$ is equivalent to using only the first gradient term which leads to a large degradation in the FID. Conversely taking $s$ to be too large results in a steep degradation in the PSNR between $s=333$ and $s=500$. Using $s=333$ strikes a good balance between the FID and PSNR in inpainting tasks.\n\n**(3)** \n> Contribution and relation to prior works\n\nThank you for referring us to ADIR, we will address it in the paper. Similar to FPS, ADIR noise the measurements based on the assumption of a linear corruption operator. As such, it is not readily clear how to use these methods with latent diffusion models due to the decoder's non-linearity. A main contribution of our paper is the ability to use auxiliary labels in latent diffusion models. The auxiliary labels are then used as an integral part of our SMC procedure.\n\nRegarding particle-based sampling approaches. As far as we know there are only five relevant studies, MCGDiff (Cardoso et al., 2023), FPS (Dou & Song, 2024), TDS (Wu et al., 2024),  SMCDiff (Tripper et al., 2023), and PFLD  (Nazemi et al., 2024). Specifically, MCGDiff and FPS both rely on the assumption of a linear corruption operator and cannot be used with latent diffusion models due to the non-linearity of the encoder-decoder. TDS is being extensively compared against. As for SMCDiff, it was designed mainly for motif-scaffolding, and not general inverse problems. Nonetheless, the proposal used in that study, namely the prior diffusion model, can be used in our case as well. PFLD is a concurrent study that uses PSLD update as a proposal distribution, which we did compare to in the paper.\n\nFollowing the suggestions of this reviewer and reviewer 326R, we add here a comparison to PFLD, LD-SMC with the prior as a proposal distribution, and LatentDAPS (Zhang et al., 2024). The comparison is presented in Tables 1 & 2 at https://tinyurl.com/p6hh6bjy. Per comment (7) to Reviewer 326R, please note that there are small changes from the results reported in the paper for LD-SMC. From the tables, LD-SMC significantly outperforms PFLD and LD-SMC with the prior proposal in all metrics. In comparison to LatentDAPS, LD-SMC has a clear advantage in FID, NIQE, and LPIPS, while LatentDAPS is better in PSNR and SSIM.\n\n**(4)**\n> Initial guess for $\\mathbf{\\hat{z}}_0$\n\nThank you for raising this interesting question. Indeed there are multiple valid ways to initialize $\\mathbf{\\hat{z}}_0$. We did not use the original image since in some tasks (such as super-resolution) it cannot be applied due to a mismatch in the image dimensions. The important part of the initialization is that it will carry information about the measurement $\\mathbf{y}_0$ which can then be used to guide the sampling process using the auxiliary labels. Fig. 5 in the paper shows that $\\mathbf{y}\\_{1:T}$ are sensible. Tables 9 & 10 at https://tinyurl.com/p6hh6bjy show an advantage for LD-SMC initialization compared to reviewer’s proposal, perhaps because it  doesn’t take into account the corruption operator.\n\n**(5)** \n>”Why is $\\mathbf{y}_t$ independent of $\\mathbf{y}_0$?”\n\nDue to the model structure when conditioning on $\\mathbf{z}_t$ these variables become conditionally independent.\n\n\n**(6)** \n> ”In Eq. (4) [...] what is the justification for using them both?”\n\nWe would appreciate further clarification in case we misunderstood the question. The terms use different labels and we do not see how these gradients or their scale are the same. Moreover, $\\gamma_t$ is a free scalar parameter and it doesn’t have to be connected to the Jacobean.\n\n**(7)** \n>”Why would the third term give any meaningful gradients [...]?”\n\nThank you for raising this question. As the decoder wasn’t trained on noisy images, we do not expect its reconstructions to be natural images, especially for large $t$ values. However, as a fixed function, its gradients are meaningful even on out-of-distribution data as they have information on how this function reacts to small changes. As such, the gradients help LD-SMC to move the latent encoding to one whose reconstruction is closer to the desired image as verified empirically.", "claims_and_evidence": "The claims are not supported with clear evidence. Please see the weaknesses below.", "methods_and_evaluation_criteria": "Proposed method and evaluation criteria make sense. However, the extent of the evaluation is limited and relevant baselines in particle filtering area are missing (please see weaknesses below).", "theoretical_claims": "This is an empirical paper.", "experimental_designs_or_analyses": "Yes, I have checked the soundness/validity of experimental designs or analyses presented in Section 5.", "relation_to_broader_scientific_literature": "The main idea of noising the measurements and performing a gradient step to minimize the measurement error in the noisy latent space has been previously explored (see for example ADIR: https://arxiv.org/pdf/2212.03221). Additionally, the multi-particle sampling approaches have also been extensively studied in prior works, as discussed in the related works of this paper (Sec 3). Given these prior works, the contribution of the proposed approach appears limited.", "essential_references_not_discussed": "Essential references have been discussed. However, the experimental results lack comparison with these methods.", "other_strengths_and_weaknesses": "### Strengths\n\nThe experimental results demonstrate improvements over prior works in Gaussian deblurring, super-resolution, and inpainting tasks. These findings suggest that the Sequential Monte-Carlo (SMC) approach effectively enhances posterior sample quality in diffusion models.\n\n### Weaknesses\n\n1. Given that the proposed method employs a multi-particle sampler, it would be valuable to compare its performance against existing particle filtering methods discussed in related works, such as PFLD and FPS. Although the compared baselines are relevant, a direct comparison with these particle-based methods could provide insights into the advantages and potential limitations of the proposed approach, particularly in terms of sampling efficiency, particle diversity, and robustness in diffusion-based inverse tasks.\n\n2. Why is the initial guess for $\\hat{z}_0$ (Sec 4.2.1) as the argmin of Eq (3) important? Can’t we just use the original image itself? Especially, because the VAE-encoder will produce a valid latent if the input is an image. The solution of the optimization problem in Eq (3) on the other hand may not produce an image.\n\n3. The main idea of noising the measurements and performing a gradient step to minimize the measurement error in the noisy latent space has been previously explored (see for example ADIR: https://arxiv.org/pdf/2212.03221). Additionally, the multi-particle sampling approaches have also been extensively studied in prior works, as discussed in the related works of this paper (Sec 3). Given these prior works, the contribution of the proposed approach appears limited. \n\n4. Line 243 (right col): Why is $y_t$ independent of $y_0$ in the first equality?\n\n5. In Eq (4), the gradient wrt $\\hat{z_t}$ can be decomposed by applying chain rule with variable $\\mu_{\\theta}(\\cdot,\\cdot)$. In this case, the multiplying factor $\\gamma_t$ seems to be the Jacobian of $\\mu$ wrt $\\hat{z}_t$. So the second and third terms are roughly on the same scale, except some scaling factors and additive constants. In that case, what is the justification for using them both? Besides, why would the third term give any meaningful gradients given the fact that $D$ is not trained to produce valid images given noisy latents?\n\n\n##############\nPost-rebuttal\n##############\n\nThe reviewer thanks the authors for responding to the raised concerns. Based on the newly added results, the reviewer is raising the score to borderline reject. Please see below for the remaining concerns.\n\nThe reviewer has the following concerns regarding the contributions of the paper post rebuttal. \n\nThe authors claim that a paper very similar to theirs, called PFLD, is a concurrent work (although it had first appeared on ArXiv in Aug 2024). The idea of noising measurements has also previously appeared in ADIR. The authors argue that this technique can not be extended to latent diffusion models due to the non-linearity of encoder-decoder. However, this is the case for most of the compared baselines including the proposed method. For examples, $y_t$ will not provide useful signal unless the decoder preserves some structure of the original image (as shown in Figure 1) through linear approximation. Therefore, the main contribution of this paper appears limited given the fact that it combines these two ideas without properly crediting these prior works. \n\nThe argument that $y_t$ is independent of $y_0$ is also not necessarily true in general. It is a mere implication of the model structure assumed by the authors. There is no guarantee that this is the true structure. The gradient in the second term of Eq. (4) can be decomposed by applying chain rule with $mu_\\theta$.\n\nMajority of the relevant baselines were omitted during the main paper submission and only appeared during the rebuttal. Given these concerns, the reviewer believes the paper needs a major revision before getting accepted.", "other_comments_or_suggestions": "Please see the weaknesses above.", "questions_for_authors": "Please see the weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CgDnBpEhku", "reviewer": "Reviewer_326R", "summary": "This paper is about solving inverse problems in a training-free manner using latent denoising diffusion priors and sequential Monte Carlo methods. It proposes a novel probabilistic models that enables inference of the hidden states with latent diffusion priors given the observation. \n\nSpecifically, this probabilistic models relies on combining two \"orthogonal\" previous ideas introduced in the literature. On one hand, a popular approximation for the likelihood of the observation, of which the gradient is required to be able to implement a diffusion posterior model given the noisy state is $p(y|x_t) \\approx p(y|E(x_0|x_t))$. Another approach consists instead of sampling drawing noisy observations according to the forward process and then conditioning, using various heuristics, the reverse process on these observations. \n\nThis paper essentially combines these two approaches and blends them in a SMC scheme. This is done by defining a specific probabilistic models in which there are hidden states of which we have observations. Given a hidden state $z_t$, the observation $y_t$ is assumed to be obtained by first decoding the hidden state $z_t$, applying the forward operator to it then adding Gaussian noise with std $\\tau$. The proposed algorithm is then a SMC applied to this probabilistic model.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort put into the review. We are encouraged that the reviewer found our approach interesting and the analysis sound. We address the reviewer’s comments in the following. All answers and suggestions will be incorporated in the paper.\n\n**(1)** \n> “The main claim in the paper [...] is not really studied qualitatively”\n\nThe main claim in this paper is that combining auxiliary variables with the DPS approximation can perform better sampling for inverse problems. This point is supported by our experiments. The “large scale” vs “fine detail” is our qualitative observation on LD-SMC performance, and not the main claim itself. An illustrative example of this can be seen at https://tinyurl.com/2u6mmj2x, specifically the generated writing in the third image.\n\n**(2)**\n> Behaviour of the decoder and stability issues\n\nThank you for raising this great question. We agree with the reviewer that evaluating the decoder on noisy latents, especially for large $t$ values, can generate non-natural images. However, it does not imply stability issues as long as the gradients of the decoder are well-behaved, which is verified empirically. Importantly, the latents, even if noisy, carry information about the label $\\mathbf{y}_0$ which guides the sampling process using the auxiliary labels closer to the desired image. Please see also comments (4) & (7) to reviewer yJi6 that touched this point.\n\n**(3)** \n> Evaluating the decoder at the denoised hidden state\n\nThank you for this suggestion. In our early experiments, we tried using DPS, for generating the labels and initialization of LD-SMC. Since it did not improve the results we used the proposed simple alternative which is also computationally cheaper.\n\n**(4)**\n>What do you mean by \"the variance term is taken to be the variance of the forward diffusion process?”\n\nThe variance of the conditional distribution $p(\\mathbf{z}_t|\\mathbf{z}_0)$ which is $1−\\bar{\\alpha}_t$. \n\n**(5)** \n> Large variance of the Gaussian likelihood\n\nThe variance aims to reflect the stochasticity in $\\mathbf{z}_0 | \\mathbf{z}_t$, it starts small and gets larger with $t$, but capped at 1. This choice is sensible although other options are also applicable. Either way, the weighting mechanism can fix for that approximation.\n\n**(6)** \n> “In the probabilistic model [...] the authors assume that it is Markovian”\n\nThe forward process is indeed not Markovian, but, in the backward process, $\\mathbf{z}_t$ depends only on $\\mathbf{z}\\_{t+1}$ as evident in Eq. 10 & 12 in the DDIM paper.\n\n**(7)** \n> Methodology - the SMC procedure and Gibbs sampling\n\nWe thank the reviewer for the valuable and important comments on LD-SMC methodology. Following the reviewer’s comments we show here that the empirical distribution over samples converges to the true target of interest $p_\\theta(\\mathbf{z}_0 | \\mathbf{y}_0)$ in the large compute limit. This addition leads to a few modifications to the algorithm presented in the paper. Importantly, since the Gibbs sampling process was applied only once, the modifications from the current presentation are minor. In addition, we stress that these modifications preserve the results witnessed in the paper (and even slightly improve them). Link to algorithm: https://tinyurl.com/2t9rpf3j. The main changes are, (1) Generate the auxiliary label $\\mathbf{y}_T$, and use it in the SMC procedure for correcting the initial sampling step at time $t=T$; (2) Sample the full chain $\\mathbf{z}\\_{0:T}$ using SMC, for the Gibbs sampling procedure. These modifications allow us to show the following result,\n\nTheorem (informal). Let $\\mathbb{P}_N(\\mathbf{z}\\_{0:T}) = \\sum\\_{i=1}^N w\\^{(i)}_0 \\delta\\_{\\mathbf{z}\\_{0:T}\\^{(i)}}(\\mathbf{z}\\_{0:T})$, be the discrete measure obtained by the function $\\mathbf{SISR}$ in Algorithm 1, where $\\delta$ is the Dirac measure. Under regularity conditions $\\mathbb{P}_N(\\mathbf{z}\\_{0:T})$ converges setwise to $p\\_\\theta(\\mathbf{z}\\_{0:T} | \\mathbf{y}\\_{0:T})$ as $N \\rightarrow \\infty$. Furthermore, the stationary distribution of the Gibbs sampling process is $p\\_\\theta(\\mathbf{z}\\_{0:T}, \\mathbf{y}\\_{1:T} | \\mathbf{y}_0)$, and $p\\_\\theta(\\mathbf{z}_0 | \\mathbf{y}_0)$ is the limiting distribution of the $\\mathbf{z}_0$'s subchain.\n\nLink to the full claim and proof: https://tinyurl.com/3rdvpnad. Importantly, this result does not depend on *any* of the approximations made to derive our model.\n\n**(8)** \n> Recent works from last year\n\nThank you for referencing us to these studies. We will discuss them in the paper. We added here a comparison to LatentDAPS (Zhang et al., 2024) under our experimental setup. Please see the results and discussion in comment (3) to reviewer yJi6.\n\n**(9)**\n>The images generated are blurry\n\nThe quality of images heavily depends on the prior diffusion model. We used LDM VQ-4 which is not as strong as recently released diffusion models. This point was also shown in Table 1 (and discussed in the appendix) of Zhang et al. (2024).", "claims_and_evidence": "The main claim in the paper is that adding auxiliary observations should in combination with the DPS approximation should help in capturing the \"large scale semantics\" as well as the \"finer details\". While this is demonstrated through quantitatively through superior performance with various metrics, it is not really studied qualitatively. Can the authors exhibit an example in which this can be understood precisely?", "methods_and_evaluation_criteria": "Besides some issues with the methodology that can be fixed and which I discuss below, there are some design choices that are rather odd and are not explained in the paper. For example, in the probabilistic model introduced, the authors assume that the observation is obtained by applying the forward operator to the decoding of the current noisy state. Doesn't this introduce some stability issues that could yield to blurry images? The decoder has been trained on clean data only and evaluating it at noisy samples should result in odd behavior. Can the authors further discuss this? I would have assumed that a more reasonable approach would be to evaluate the decoder at the denoised hidden state. In this case one could set the variance to be $1 - \\bar\\alpha_t$ instead of letting it be a hyperparameter.  Similarly, the decoder is applied on highly noisy samples in the proposal transition. \n\nAlso, the Gaussian likelihood approximation of $p(y_0|x_t)$ has variance $1 - \\bar\\alpha_t$. This is quite odd since $y_0$ is the clean data, why would one assume such a large variance? In Wu et al. 2024 the variance is that of the p(y_0 | x_0), could you explain what you mean by \"the variance term is taken to be the variance of the forward diffusion process?", "theoretical_claims": "While the idea is interesting and the methodology sounds reasonable, I believe that there are several issues with it that needs to be clarified/addressed. \n- First, in the probabilistic model considered (defined in 4.1) the joint distribution of the hidden states is **not** necessarily Markovian due to the fact that the authors use DDIM. Unless $\\eta = 1$, the hidden process is not Markovian. Still, the authors assume that it is Markovian as evidenced the computation that starts at line 240 in the second column. Unless I am missing something, this derivation does not hold unless $\\eta=1$. \n- Regarding the same derivation, in SMC one only needs to propagate the particle approximation of $p(z_{t:T} | y_{t:T})$ but here the authors instead perform what is known as *online smoothing*; they derive particle approximations of the hidden states conditioned on all the remaining observations. This is arguably harder than filtering and it seems to me that this is not needed. Actually, the authors assume that \n$$ \np(z_{t:T} | y_{0:T-1}) \\approx p(z_{t:T} | y_{t:T-1}, y_0)\n$$ \nwhich significantly simplifies the computations. Still this requires computing $p(y_0|z_t)$ but this is approximated using the DPS approximation. It seems to me that smoothing is considered here because the authors want to further condition on the initial observation.  Now I would like to emphasize that the probabilistic model considered in the paper is **arbitrary**, meaning that the observation likelihood could have been different. Hence, since the authors want to condition on both $y_0$ and $y_t$ at each step of the diffusion process they could have changed their probabilistic model so that there are **two** observations $(y^1 _t, y^2 _t)$ for the same hidden state $z_t$. If their likelihood is given by \n$$ \np(y^1 _t, y^2 _t | z_t) = \\mathcal{N}(y^1 _t | \\mathcal{A}\\big(\\mathcal{D}(\\overline{z}_0(z_t)), (1 - \\overline\\alpha_t) \\big)  \\mathcal{N}(y_t | \\mathcal{A}\\big(\\mathcal{D}(z_t)), \\tau^2 \\big) \n$$ \nA classical particle filter with this model, with observations $y^1 _t = y_0$ for all $t$, and observations $y^2 _t$ generated as in the paper would yield, I believe, the exact same algorithm if all the steps are performed conditionally on $y^1 _{0:T}$, which is not restrictive at all. The catch however is that the observation model for $y^1 _t$ is misspecified and the targeted marginal distribution is no longer the posterior distribution of interest. \n\n- The authors mention that the proposed algorithm is a blocked Gibbs sampling procedure but I am unsure that this is the case. A blocked Gibbs sampler in this case would proceed by first sampling the hidden states given the observations then the observations given the hidden states. The proposed algorithm however has one more step where the hidden states $z_{1:T}$ are sampled conditionally on $\\hat{z}_0$ only and not the observation. I do not see how this can be a Gibbs sampler. Shouldn't one draw a whole trajectory from the particle smoothing approximation then use it to draw the observations?", "experimental_designs_or_analyses": "- The method is tested on the standard imaging benchmark that is used in most diffusion posterior sampling papers. The method is also compared to standard latent diffusion posterior sampling methods. The analysis of the result is sound. \n\n- The images generated with the various algorithms are much blurier than one would expect. For example, posterior sampling with pixel-space diffusion yields images with much better quality. I believe that a discussion of this matter is warranted.", "relation_to_broader_scientific_literature": "The paper borrows various ideas from existing diffusion posterior sampling methods such as [1,2,3]. The methodology presented in the paper is still interesting and is not merely a simple combination of these papers. \n\n[1] Dou, Z. and Song, Y. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective.\n[2] Trippe, B. L., Yim, J., Tischer, D., Broderick, T., Baker, D., Barzilay, R., and Jaakkola, T. Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem. \n[3] Song, B., Kwon, S. M., Zhang, Z., Hu, X., Qu, Q., and Shen, L. Solving inverse problems with latent diffusion models via hard data consistency.", "essential_references_not_discussed": "While the algorithms to which the method is compared to are still relevant, there are nonentheless recent works from last year that achieve much better performance on posterior sampling with latent diffusion. \n\n[1] Zhang, B., Chu, W., Berner, J., Meng, C., Anandkumar, A. and Song, Y., 2024. Improving diffusion inverse problem solving with decoupled noise annealing.  \n[2] Moufad, B., Janati, Y., Bedin, L., Durmus, A., Douc, R., Moulines, E. and Olsson, J., 2024. Variational Diffusion Posterior Sampling with Midpoint Guidance.", "other_strengths_and_weaknesses": "no further strengths and weaknesses.", "other_comments_or_suggestions": "see above.", "questions_for_authors": "no further questions, see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors introduced a new sequential Monte Carlo-based  algorithm to sample from the latentspace of diffusion models.    The main contribution is a new procedure to solve inverse problems with latent diffusion models and the use of SMC methods to solve posterior sampling problems. They initially validated their approach using two datasets (ImageNet and FFHQ). Some reviewers were concerned by the fact that the ideas of the paper were already explored in other contributions and that the contribution is incremental while the experiments do not match state-of-the-art results in some settings. 

Although the paper shares some similarities with  diffusion posterior sampling methods, the methodology is still new and after the discussion period the authors provided an updated version of the algorithm which improved the empirical results. The authors also provided extensive numerical simulations, adding computation costs and ablation studies (on the number of particles for instance) during the rebuttal period. They also clarified the contributions with respect to previous works combining SMC algorithms and diffusion models.
The discussion should lead to a deep revision of the paper but I believe this would provide an interesting contribution. 

For all these reasons, I lean towards accepting the paper. 

Best regards.
Decision: Accept (poster)