Diffusion Posterior Sampling for Linear Inverse Problem Solving: A Filtering Perspective
Authors: Zehao Dou
Yang Song
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion Models
linear Inverse problem
Bayesian posterior sampling
Bayesian filtering
importance sampling
Primary area: generative models
TL;DR: We reveal a link between Bayesian posterior sampling and Bayesian filtering in diffusion models, and propose a consistent Filtering Posterior Sampling (FPS) model which has state-of-the-art performance.

Abstract: Diffusion models have achieved tremendous success in generating high-dimensional data like images, videos and audio. These models provide powerful data priors that can solve linear inverse problems in zero shot through Bayesian posterior sampling.
However, exact posterior sampling for diffusion models is intractable. Current solutions often hinge on approximations that are either computationally expensive or lack strong theoretical guarantees. In this work, we introduce an efficient diffusion sampling algorithm for linear inverse problems that is guaranteed to be asymptotically accurate. We reveal a link between Bayesian posterior sampling and Bayesian filtering in diffusion models, proving the former as a specific instance of the latter. Our method, termed filtering posterior sampling, leverages sequential Monte Carlo methods to solve the corresponding filtering problem. It seamlessly integrates with all Markovian diffusion samplers, requires no model re-training, and guarantees accurate samples from the Bayesian posterior as particle counts rise. Empirical tests demonstrate that our method generates better or comparable results than leading zero-shot diffusion posterior samplers on tasks like image inpainting, super-resolution, and deblurring.

Published as a conference paper at ICLR 2024
DIFFUSION POSTERIOR SAMPLING FOR LINEAR INVERSE PROBLEM SOLVING - A FILTERING PERSPECTIVE
Zehao Dou
Yang Song
Yale University zehao. eeuuyale.edu
OpenAl songyanggooenaii.comm com
ABSTRACT
Diffusion models have achieved tremendous success in generating high- dimensional data like images, videos and audio. These models provide powerful data priors that can solve linear inverse problems in zero shot through Bayesian posterior sampling. However, exact posterior sampling for diffusion models is intractable. Current solutions often hinge on approximations that are either com- putationally expensive lack strong theoretical guarantees. In this work, we introduce an efficient diffusion sampling algorithm for linear inverse problems that is guaranteed be asymptotically accurate. We reveal link between Bayesian posterior sampling and Bayesian filtering in diffusion models, proving the former as a specific instance of the latter. Our method, termed filtering posterior sampling, leverages sequential Monte Carlo methods to solve the corresponding filtering problem. seamlessly integrates with all Markovian diffusion samplers, requires no model re-training. and guarantees accurate samples from the Bayesian posterior as particle counts rise. Empirical tests demonstrate that our method generates better comparable results than leading zero-shot diffusion posterior samplers on tasks like image inpainting, super-resolution, and motion deblur.
INTRODUCTION
Score-based diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; 2020; Ho et al., 2020; Song et al., 2020b) have made remarkable strides in data synthesis over the past few years, revolutionizing fields like image synthesis (Dhariwal & Nichol, 2021; Nichol et al., 2021; Ramesh al., 2022; Saharia et al., 2022; Rombach al., 2022; Zhang et al., 2023), video generation (Ho et al., 2022b;a), audio synthesis (Kong al., 2020; Chen et al., 2020) and molecular conformation generation (Xu et al., 2022: Shi et al., 2021: Luo et al., 2021). ). As evidenced by their unparalleled sample quality, diffusion models provide powerful data priors that capture the intricacies of high dimensional data distributions. Using such priors, we can deduce data from lossy measurements through Bayesian posterior sampling. This enables many diffusion-based methods for solving linear inverse problems without task-specific training (Song et al., 2020b; Chung et al., 2022c; Chung & Ye, 2022: Choi et al., 2021; Kawar et al., 2022; Chung et al., 2022a; Kawar al., 2021; Bardsley et al., 2014; Song et al., 2023a; Jalal et al., 2021). Examples of their applications include image inpainting, colorization, super-resolution, motion deblur, and medical image reconstruction (Song al., 2021).
Despite the empirical success of diffusion models in solving inverse problems, obtaining exact Bayesian posterior samples for these models remains intractable, necessitating the use of approxima- tions. Current methods for this approximate sampling can be broadly grouped into three categories. The first approach modifies standard diffusion sampling process by enforcing data consistency at every time step, ensuring all intermediate samples align with observed, lossy data measurements (Song et al., 2020b; Chung et al., 2022c; Chung Ye, 2022; Choi et al., 2021; Kawar et al., 2022). The second approach estimates the score function (i.e., the gradient of the log probability density) of the Bayesian posterior (Song et al., 2020b), leveraging it to guide each diffusion sampling iteration (Chung et al., 2022a; Kawar et al., 2021: Bardsley et al., 2014; Song al., 2023a; Jalal et al., 2021; Song et al., 2023b). Lastly, some methods train a neural network to minimize a statistical divergence between its sample distribution and the true Bayesian posterior of the diffusion model (Graikos

---

Published as a conference paper at ICLR 2024
Inpainting (box)
Inpainting (random)
Motion Deblur
Gaussian Deblur
Super Resolution
Label
Measure- ment
FPS
Sample
Figure 1: Several widely-used noisy measurements for images, and the recovered images FPS.
al., 2022; Feng al.. 2023; Feng Bouman, 2023; Mardani et al., 2023). For the first two strategies, while each individual sampling iteration is grounded in theoretical insights, assessing the accumulated errors introduced over successive iterations poses challenging theoretical question. The third strategy requires training additional neural networks for each novel task, which causes substantial computational overhead especially when applying the same diffusion prior to diverse inverse problems.
To address these challenges, we introduce a novel family of methods that allow diffusion models to tackle linear inverse problems in shot. We present unique interpretation of Bayesian posterior sampling in diffusion models and establish its equivalence to Bayesian filtering. This insight lets us approach the Bayesian posterior sampling problem from filtering perspective. Our proposed framework, Filtering Posterior Sampling (FPS), transforms any filtering algorithm into a diffusion posterior sampler. In particular, we devise an FPS algorithm based on particle filtering and sequential Monte Carlo (Doucet et al., 2001). To minimize the variance of particle filters, we tailor the proposal distribution at each sampling iteration. Our FPS algorithm sidesteps the need for costly optimization or training new neural networks. It works in harmony with any stochastic Markovian sampler of diffusion models and is easily adaptable diverse linear inverse problems. As particle count increases, our algorithm yields better posterior samples, offering a flexible trade-off between compute and sample quality.
Theoretically, prove that our FPS algorithm correctly samples from the Bayesian posterior distri- bution as the number of particles approaches infinity when both the score estimator and SDE solver perfect. To our knowledge, this is of the first diffusion posterior sampling algorithms with global consistence guarantee for general Bayesian linear inverse problems, along with concurrent works like Wu et al. (2023); Trippe al. (2022). Furthermore, when using just one particle, prove that our algorithm provides a specific approximation of the Bayesian posterior's score function at each sampling iteration, elucidating the link between FPS and existing methodologies. Empirically, we benchmark our FPS against current methods across various linear inverse problems in computer vision, such as inpainting (Ych et al., 2017), super-resolution (Ledig al., 2017: Haris al., 2018) and motion deblur (Kupyn et al., 2019; Suin et al., 2020). In evaluations on both the FFHQ (Karras al., 2019) and ImageNet (Deng al., 2009) datasets, our FPS surpasses most competing methods.
BACKGROUND
Linear Inverse Problems Linear inverse problems are pervasive various scientific domains, with examples like image inpainting, colorization, super resolution, deblurring, Computed Tomography, and Magnetic Resonance Imaging. Given a datapoint x, its lossy measurement denoted as RR y gd A E Rdx and ~ (V d represents the
y Ax + In this equation, measurement noise with a known noise level a > 0. The challenge of a linear inverse problem is to recover from the incomplete measurement y. As often D > this results in ill ill-posed and under-
2

---

Published as a conference paper at ICLR 2024
determined problem. To narrow down the solution space, usually require additional information about x. A common assumption is the prior knowledge of the distribution over x, represented as p(x). Given this, we can traverse the solution space of the linear inverse problem by sampling from the posterior distribution p(x y), formulated using Bayes' rule as p(x y) x p(x)p(y x), with p(y x) (yy o2).
Diffusion Models Diffusion models comprise a forward noising process and backward denoising process. In the discrete formulation (Sohl-Dee Sohh--Dcksteen et al., 2015; Ho et al., 2020), the forward process manifests as a Markov chain described by
((xEN xp)
(x(; xk-1), with ((x: xx-1) N (a.xt-1, bgT) k=1
II
(I)
The coefficients {ax 1V and {bx JN k=1 are manually designed and may differ across various diffusion formulations (Song et al., 2020b). Given that each Markov step (Xt. x2-1) 1) is linear Gaussian model, the resultant marginal distribution g(xx Xo) (o d. assumes a Gaussian form, where {ck N and {de ) can be derived from {a
k== and [br 1= For sample generation, we train neural network, denoted by se (Xx, tx). to estimate the score function Vx, log g(x; Xo). The backward process, which we assume to be Markov chain, typically represented
px(xx-1 xx) - (ux (x(x)) + sx w)).
(2)
where (xx) = (Xe + (x lx))/cx is the predicted xo derived by X obtained from the Tweedie's 's formula. Here {ux ) (uk k== and {urk {wa) can be computed from the forward process coefficients {ax ) and (bk \N in Eq. (I). {tx h is the set of discrete timestamps where to = 0, łN = T. The formulation in Eq. (2) encompasses many stochastic samplers of diffusion models, including the ancestral sampler DDPM (Sohl-Dickstein et al., 2015; Ho al., 2020), the predictor-corrector sampler in Song et al. (2020b), and the DDIM sampler in (Song et al., 2020a).
For variance preserving diffusion models (Ho al., 2020), we have
a₁ Vaki by
V Ck vāk, dN vI Oije
where ag 3r, a2 represent DDPM sampling, we have:
k - aj. and ap. By follow the notations in Ho et al. (2020). To
UA = Vak-i, Uk = (o - T)- wk V Bx 1 Ch
ax
and for DDIM sampling (Song et al., 2020a), we have:
Ug V Us
T of vI Tkk
Wkk 0k
where the conditional variance sequence fax N k= can be arbitrary. Note that DDPM sampling is special case of DDIM when setting of = B/ (1 x4-1)/(1 (r.).. In this paper, we parameterize
1 ak-1
ak c. 3r
Okk
with being a tunable parameparameter.
BAYESIAN POSTERIOR SAMPLING AS BAYESIAN FILTERING
In the following part, introduce Bayesian filtering and explain how it is related to Bayesian posterior sampling. Bayesian filtering aims to infer latent variables from observations based on a sequence of conditional probability distributions. For each time step k: = 1,2,..., ..,...N.. N, we define the latent state and its corresponding measurement as:
p(X;
XR-1), Yk ~ p(ya xx).
The dynamic model of this system determines X3 and yı depends on the distribution of measurements given the state. As we can see, the sequence {xL ) k= forms a Markov chain. The Bayesian filtering problem seeks to sample from the distribution p(X; Y1:)).
3

---

Published as a conference paper at ICLR 2024
To derive the conditional distribution, well-known iterative approach is utilized. Starting with the known prior distribution p(xo), we employ the Chapman Chapman-Kolnogon Kolmogorov equation to obtain p(X;; Y1:k) from p(x y1:k-1) 1) as the "prediction" step
f p(x;: xx-1) p(x6-1 V1:k 1
p(x V1sk-1)
(3)
and then use Bayes' rule to compute p(x* Y1:k) as the "update" step:
ply& xx) p(xk V1:k-1)
xx Y1.kk
J(yx Xx)
p(Xk
Yi:k-1
By iterating these two steps, we can solve the Bayesian filtering problem. For those system where dynamics and measurements adhere to linear Gaussian distributions, Kalman filter (Kalman, 1960) provides a closed form solution to the Bayesian filtering problem.
Back to our difffusion posterior sampling problem, the dynamic system's backward process is given by X ~ (xxx X)-1), where pe (X& x defined in Eq. (2). The forward process described by
X = ag X + bg
(5)
where 2₁ (0, I) is a standard Gaussian noise independent of x;-1- Meanwhile, we construct the [ya] sequence as:
yo Yk 0x Yk-1 + by Azz for 1: >> 1.
The noise sharing technique is also applied Song et al. (2021), ) but with different framework. Given y [Ax,, I².). it is easy to show that yı (Axx, o2 for e2 a1 ... On through mathematical induction. This formulation establishes Bayesian posterior sampling as a reverse-time Bayesian filtering problem, well-studied topic with many existing solutions and algorithms. By following the prediction and update steps in Bayesian filtering, we successively compute the marginal posterior distribution po (X& Yk:N) until we obtain (xo Yo:N). This allows to sample from the Bayesian posterior po X Yo) due to the following important observation:
We can generate sample x. from the Bayesian posterior distribution pe yo) by first sampling from pe (YI:N yo) = q(YINN yo) before sampling from Po(Xo Yo.v). This is because Po(xo, yo) = po (Xo Yo.N) Pol(y:N yo)dy1.N-
Here, q and Pe denote the probability distribution of the forward process and the backward process respectively. That is, we can leverage any algorithm for Bayesian filtering to solve the Bayesian posterior sampling problem whenever the prior is captured by diffusion model.
4 FILTERING POSTERIOR SAMPLING (FPS)
In this section, we propose a specific algorithm for solving the Bayesian filtering problem associated with a diffusion model. Due to the aforementioned connection between Bayesian filtering and Bayesian posterior sampling, this algorithm can be directly employed to generate posterior samples from a diffusion model for inverse problem solving. Our algorithm is based on particle filtering, but has tailored proposal distribution for variance reduction.
4.1 BACKWARD DIFFUSION PROCESS WITH LINEAR FILTERING
In the backward diffusion process, we first generate the sequence of [yx) Then, we recursively sample x based on X and y by solving Bayesian linear filtering.
Step 1: Generating Sequence [ There are two different ways to generate the sequence of N. {y& k= One leverages the forward process and the other depends on the backward process. For the forward process, we apply Eq. (6) to obtain
Yk OLyK-1 bb Aze
4

---

Published as a conference paper at ICLR 2024
where 2₁ ~ (0. I) is independent Gaussian noise, and the initial observation yo y is given. When generating [y& from the backward process, we apply DDIM sampling. Since we have full access to yo, there no need predict (yx) because Vo Yo- Therefore, sample {y via the following recurrence relation:
Yk-1 Y + uRYK + Wi.
Azu-
where Ugs 2k. W/ can be computed from ag. by (see Section 2), and the initial observation y approximately follows N N0,, AAT) As result, the sequence {yx )==) is fully determined through either direction, given by
N
N
dYEEN yo) = II
<yy (y-11,8)) = N) II 2(yx-1 Yo). k=
(8)
A=1
Remark 4.1. The generation of sequence {y i o yo is possible only in linear inverse problems. Otherwise, for nonlinear inverse problems, Eq. (6) not longer works, and neither forward or backward processes of {y&Jk=n tractable.
Step Backward Sequence of {xx In the second step, we generate the backward sequence of {x based on the measurement sequence {y generated in Step 1, which leads to Po (yiN (Yo = g(V1.N yo). Given that the last state XN is approximately standard Gaussian,
we have
XN px YN) x px(xx) polyN xx) with px(xN) - (0,
Since (y xN) qyyN (Axx o² I) where ey a102 ... aN. the posterior distribution po Xx YN) is Gaussian and can be expressed in closed form.
Next, we progressively sample x conditioned X₃ and yk-1- We know that po (y xx-1) q(vx-1 x3-1) = N (A 2=10² I) with Ck-1 = a102......... - In addition, po (Xa-1 Xx) determined once have the score function:
p xx) (ugx(((xs) + sx tw), w
where Xy (xx) := (x2 + ( Xx., tg.)//c ię is the conditional expectation of Xo given X which can be computed via Tweedie's s formula. Here, the choices of parameters <k; d&, Ujs Uk; Wy are determined from the process of unconditional diffusion sampling. For the DDPM and DDIM framework, we list these parameters in Scction 2. Now, we can compute the posterior distribution
pe ( Xk.Yk-1 X ( xx) Po ( Xx-1).
Here, we rely the conditional independence of x5 and yx-1 given X Since both po (Xx=1 - Xx) and pe (yk-1 x 1) are Gaussian, it is straightforward to conclude that pe Xt. Yk-1) 1) is also Gaussian distribution with a tractable form. Given {y we can now sample from po (x(xx-1 Xk; Yk-1) recursively for k N, 1,...,1 1 provides an approximate solution to the Bayesian filtering problem, and we call this algorithm Filtering Posterior Sampling (FPS).
obtain J This procedure
4.2 FURTHER ANALYSIS ON FPS
In this section, analyze how FPS provides approximate solution the Bayesian filtering problem. As discussed in Section 3, we recursively compute a prediction step and an update step to calculate p(xx Yk:N). For FPS, the following approximation is made:
po Xt. Xt+1
Yl:N) 2s Po X
Po X
YkoN)po(Xx
Xk+1. pe (x Yk: )po Xx+11 Ve)
Yk+1:N) (x( xx+1,yk).
(10)
Here, we leverage the conditional independence between Yk41 and Y442:N when X is given. After taking integral on both sides of Eq. (10) with regard X we have
(x YE.N) ~ / YE41.N) px( Xk+1, Be41.y)dde yx)dx
(1I)
5

---

Published as a conference paper at ICLR 2024
(x6
xy-1
Particle Filtering with Variance Reduction
End
M Particles Each Step
Yx-1
Yi
Yo
Start
Progressive Denoising
Figure 2: Illustration of FPS-SMC algorithm.
which means we skip the update step in FPS, and we make the approximation po (x Yk:N) ) E Po(X<+1 Vk+1.N). Here, we sample a single particle Xx++1 from (x YR+1:N) and then another particle ~ (x(x YR:N) by following the prediction step. Notice that Kalman filtering is not applicable here because po Xx X1++1) 1) not a linear Gaussian process, so Monte Carlo methods necessary for solving this Bayesian filtering problem.
One can derive additional theoretical insights by generalizing FPS to continuous-time time Markov chains. When we make the time step At O and consider the continuous limit of FPS, our backward diffusion process converges to the following SDE:
B(t) 8(1) x, log vi)) di + V
dx;-
For the detailed proof, see Appendix D.1. According to this equation, FPS approximates the unknown conditional score function Vx, log ( y) with Vx, logpa, yr). Unlike the original score-based SDE, y: sampled from a noisy version of in FPS; instead, they sampled from a backward diffusion process.
4.3 FPS WITH SEQUENTIAL MONTE CARLO
To improve the approximation made by FPS, we propose variant based on particle filtering and sequential Monte Carlo. We call this improved method Filtering Posterior Sampling with Sequential Monte Carlo (FPS-SMC).
To begin with, notice that
Po (X Yk:N) xx Po(yx
Xg po (X YK+1.N)
Po(yv
xx) XI+1IN YK+1:N) ) (x(xxi YE+IN)
pe(yx
Xx (x)x
xx+1) po (x YK+1IN)
(12)
(y
Xk (x
xx-1) po(xx X
(Y+HN)).
po(xx,
Xx:
Xk+
In particle filtering, we generate M i..d particles x pxN yx). After obtaining fx f hie ()) x for j E [M.] Afterwards,
we move them backward for one step to obtain x pe(xx
we randomly sample M particles with replacement, which we denoted as {x }je[ur]- The particles re-sampled according to the following probability distribution
Po
X
pe
x
2221)/po
x 3k)
P
X
13
Σ
(V
X
Ioo
x
(1)
x x 1/po (x
(1)
x
(
(13)
One important observation about the resampling weights
Pe (Y*
Xx 1x))pxX (xK
xx+1).
Po(yK x Po
xb+1)
= po(yx xx+1).
Po (xk
X1++1 yk)
PolyK
XR. XE+1)

---

Published as a conference paper at ICLR 2024
Based on this, propose the FPS-SMC method with pseudo-code provided in Algorithm 1 and visual illustration given in Fig. 2. Clearly, FPS can be viewed as a special case of FPS-SMC with particle size . 1. Unlike FPS, FPS-SMC is consistent, meaning that the approximation error converges to zero as the particle size M approaches infinity. Below we provide a rigorous statement about the convergence of FPS-SMC.
Assumption 4.1. Assume the following two facts:
(1) Our score estimation is perfect, i.e: so(X,,t) = log po., (x,))
(2) The time step O, which means the backward diffusion ODE/SDE solver can provide accurate solution without discretization error.
These two
twofact facts make sure that both our diffusion model and sampling procedure are perfect during the backward Bayesian filtering.
Proposition 4.1. Denote po (o(x Yo:N) as the result of FPS-SMC algorithm with particle size M, and p" (Xo:N Yo:N) as the solution of Bayesian filtering problem (which follows the forward diffusion model). Under Assumption 4.1, we have po (xxx (x+1)) = =p" (xx X++1) for Vk- = 0, I,...,N Then, we can conclude that:
px(xo yo) 5h p0x(x |oo) = 9(xo yo) when M + 00,
where ". implies weak convergence.
Proof. See Appendix D.2.
This result proves the asymptotic consistency of FPS-SMC when M oo, N We can also derive finite sample bounds for W, (pe yo). p" (Xo yo)) with regard to the estimation error in score models well the time step At T//.. more comprehensive theoretical exploration of these finite sample bounds remains a topic for our future research.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
We evaluate our algorithms on the FFHQ-Ik- validation set (Karras et al., 2019) and the set (Deng et al., 2009),
Algorithm 1 The Framework of FPS-SMC
1: Given: N, particle size M and the score estimator so(:,))
both frequently used in inverse problem research involving diffusion models. Each dataset con- tains images of dimensions 256 256 nor- malized to the range [0, 1]. These images serve for testing various linear inverse problems. We follow the experimental settings in Chung al. (2022a) to ensure a fair comparison. We assume all the observations contain Gaussian noise of mean zero and standard deviation a = 0.05. In our experiments for the FPS-SMC, we choose a particle size of M = 20. Another hyper- parameter find important for FPS and FPS- SMC is c, utilized during DDIM sampling. Our empirical observations highlight the significant influence of on the smoothness of generated images, so we tune e separately for different tasks and datasets. For more detailed analy- sis, see Appendix for an ablation study, and Appendix E for thorough description of all the five tasks test in experiments.
2: Sample the {yx} sequence from g(yIN yo).
3: Sample M i.id samples x ~ po(xN IYN) for j E [M].
4: for = N, - 1,..., do
5: Generate M i.i.d samples (
X
(x x yk-1) for E [M].
())
6: Randomly pick M samples with replace-
following the probability distribution (13). Denote the
ment from (x(2)
(1)
JE[M]'
M new samples as (x (x
7: end for
8: The output x. is uniformly sampled from (x(²) }E(M)
During the backward process of FPS, we set the number of time steps as = 1000 and use the pretrained score model from Chung et al. (2022a) for the FFHQ dataset, and the score model from Dhariwal & Nichol (2021) for the ImageNet dataset. We use multiple metries to evaluate the quality of our generated images, including peak signal-to-noise ratio (PSNR), Fréchet Inception Distance
7

---

Published as a conference paper at ICLR 2024
Label
Measure- ment
FPS-SMC
SPF
Figure 3: Comparison between FPS-SMC and Standard Particle Filtering (SPF) when par- ticle size M 20 Inpainting (box) task over FFHQ dataset.
Figure The relation between the particle size M and the PSNR value, with 68% (xl std) confidence interval added
(FID), structural similarity index (SSIM), and Learned Perceptual Patch Similarity (LPIPS) (Zhang et al., 2018). These metrics enable comprehensive assessment of image quality. All our experiments are carried out on single A100 GPU.
5.2 EXPERIMENTAL RESULTS
In Tables and 2, we compare FPS and FPS-SMC against various baselines terms of FID and LPIPS. For additional quantitative results and comparisons in PSNR and SSIM, we relegate them to Tables and 4 in Appendix E. The results demonstrate that FPS outperforms baselines significantly in tasks such super resolution, inpainting (box). Gaussian deblurring, and motion deblurring. It is remarkable that in the challenging impainting (random) task, where substantial 92% of all pixels are masked out, FPS achieves the second-best performance. When assessing performance across the four metrics, FPS emerges as the front-runner in three of them, with superior results compared to the other baseline models.
5.3 ABLATION STUDY: INFLUENCE OF THE PARTICLE SIZE
In theory, we can solve the filtering problem with increases. By default, we apply M 20 to all our experiments to make it computationally efficient. Comparing FPS (where M = 1) and FPS-SMC (where M 20), we observe that FPS-SMC has an edge only with respect the PSNR metric, but the difference in other three metrics SSIM, FID and LPIPS is small. To rigorously study the variation of image quality as a function of particle size M, we focus inpainting (box) over FFHQ 256 x 256-1k Ik validation set. As can see Fig, 4, PSNR metric increases as the particle size M grows, but has diminishing returns once reaches 100. For the increasing running time of FPS-SMC for larger partiele size M, we demonstrate in Appendix F.2 that it approximately holds that &( (M) VM. We also compare it with other popular methods, which shows that the running time of FPS-SMC at the average level.
more accurately when the particle size M
5.4 ABLATION STUDY: FPS-SMC VERSUS STANDARD PARTICLE FILTERING
In Eq. (12), we observe that
Po(XE:N yk:N) x Po xx) (x X1 (x-1))0(x6411) Y1+1:N),
which leads to the standard way for implementing particle filtering. After obtaining (x(1) Je[M] from previous steps, we generate M particles independently by simulating the unconditional backward
process: x(3)
pe (x
x 2
(i) for j [M]. Afterwards, we randomly sample M particles with
replacement according to the distribution P X = x x Po x
We name as Standard
Particle Filtering (SPF). In theory, both FPS-SMC and SPF consistent solvers for the Bayesian
8

---

Published as a conference paper at ICLR 2024
filtering problem. However, in SPF, the {x+ jk= sequence only get the measurement information from the sampling step, which provides weak signals for x; to follow the measurement y. As a result, we need a much larger partiele size M to obtain reasonable performance for SPE, especially in high dimensional spaces. In Fig. 3, we compare the performance between FPS-SMC and SPF when see, the particles generated by SPF are almost completely independent of the
= 20. As measurement. In contrast, we tailor the proposal distribution in FPS-SMC to significantly reduce the variance, resulting in an effective algorithm with smaller particle size.
Table 1:
Quantitative results (FID, LPIPS) of our model and existing models on a various of linear inverse problems on FFHQ 256 x 256-1k validation dataset.
Super Resolution
Inpainting (box)
Gaussian Deblur
Inpainting (random)
Motion Deblur
FFHQ
Methods
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
FPS
26.66
0.212
26.13
0.141
30.03
0.248
35.21
0.265
26.18
0.221
FPS-SMC
26.62
0.210
26.51
0.150
29.97
0.253
33.10
0.275
26.12
0.227
DPS
39.35
0,214
33.12
0.168
44.05
0.257
21.19
0.212
39.92
0.242
DDRM
62.15
0.294
42.93
0.204
74.92
0.332
69.71
0.587
MCG
87.64
0.520
40.11
0,309
101.2
0.340
29.26
0.286
PrP-ADMM
66.52
0.353
151.9
0.406
90.42
0,441
123.6
0.692
Score-SDE
96.72
0.563
60.06
0.331
109.0
0.403
76.54
0.612
ADMM-TV
110.6
0.428
68.94
0.322
186.7
0.507
181.5
0.463
Table 2: Quantitative results (FID, LPIPS) of our model and existing models on a various of linear inverse problems on ImageNet 256 256 1k validation dataset.
Super Resolution
Inpainting (box)
Gaussian Deblur
Inpainting (random)
Motion Deblur
ImageNet
Methods
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
FPS
47.32
0.329
33.19
0.204
54.41
0.396
42.68
0.325
52.22
0.370
FPS-SMC
47.30
0.316
33.24
0.212
54.21
0.403
42.77
0.328
52.16
0.365
DPS
50.66
0.337
38.82
0.262
62.72
0.444
35.87
0.303
56.08
0.389
DDRM
59.57
0.339
45.95
0.245
63.02
0.427
114.9
0.665
MCG
144.5
0.637
39.74
0.330
95.04
0,550
39.19
0.414
PmP-ADMM
97.27
0.433
78.24
0.367
100.6
0.519
114.7
0.677
Score-SDE
170.7
0.701
54.07
0,354
120.3
0.667
127.1
0.659
ADMM-TV
130.9
0.523
87.69
0.319
155.7
0.588
189.3
0.510
CONCLUSION
We introduce Filtering Posterior Sampling (FPS), novel approach addressing the challenges inherent in diffusion models when solving linear inverse problems. FPS leverages a unique perspective on Bayesian posterior sampling, establishing enabling a transition from a sampling problem to a filtering one. Notably, our FPS algorithm employs particle filtering and sequential Monte Carlo techniques without necessitating optimizations or the training of supplementary neural networks. It offers compatibility with any Markovian sampler and showcases versatility across diverse linear inverse problems. The flexibility in compute-resource allocation in our method ensures an optimal balance between computational effort and sampling quality. On a theoretical front, we demonstrated the global convergence guarantee of our FPS method, marking pioneering achievement in diffusion posterior sampling. Empirical evaluations across multiple computer vision tasks, including inpainting, super-resolution, and motion deblur further demonstrate the superiority of FPS over prevailing methods benchmarks like FFHQ and ImageNet. This research paves the way for enhanced efficiency and precision in leveraging diffusion models for intricate linear inverse problems.
equivalence with Bayesian filtering and thereby
-istecceinteesivve

---

Published as a conference paper at ICLR 2024
REFERENCES
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3)313-326, 1982.
Riccardo Barbano, Alexander Denker, Hyungjin Chung, Tae Hoon Roh, Simon Arrdige, Peter Maass, Bangti Jin, and Jong Chul Ye. Steerable conditional diffusion for out-of-distribution adaptation in imaging inverse problems. arXiv preprint arXiv:2308. 14409, 2023.
Johnathan M Bardsley, Antti Solonen, Heikki Haario, and Marko Laine. Randomize-then-opiimizz A method for sampling from posterior distributions nonlinear inverse problems. SIAM Journal on Scientific Computing, (194)))99,99- 2014.
Benjamin Boys, Mark Girolami, Jakiw Pidstrigach, Sebastian Reich, Alan Mosca, and O Deniz Aky- ildiz. Tweedie moment projected diffusions for inverse problems. arXiv preprint arXiv:2310.06211, 2023.
Gabriel Cardoso, Yazid Janati El Idrissi, Sylvain Le Corff, and Eric Moulines. Monte carlo guided diffusion for bayesian linear inverse problems. arXiv preprint arXiv:2308. 2308807988, 2023.
Stanley H Chan, Xiran Wang, and Omar A Elgendy. Plug-and -lay admm for image restoration: Fixed-point convergence and applications. IEEE Transactions on Computational Imaging, 3(1): 84-98, 2016.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00713 2020.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Condi- tioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.0299 2021.
Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical Image Analysis, 80: 80:102479, 2022.
Hyungjin Chung. Jeongsol Kim, Michael T Mccann, Mare L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209. 14687, 2022a.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 0-25583--26996 2022b.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-dffuus--aatte Accelerating conditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12413-12422, 2022c.
Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning latent diffusion models for inverse problems. arXiv preprint arXiv:2310.01110, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 255. Icee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Arnaud Doucet, Nando De Freitas, and Neil Gordon. An introduction to sequential monte carlo methods. Sequential Monte Carlo methods in practice, pp. 3-14, 2001.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, (06(496):1602-166 2011.
Berthy T Feng and Katherine Bouman. Efficient bayesian computational imaging with a surrogate score-based prior, arXiv preprint arXiv:2309.0/9.0 09999.9999999........... 2023.
10

---

Published as a conference paper at ICLR 2024
Berthy T Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine L Bouman, and William Freeman. Score- based diffusion models as principled priors for inverse imaging. arXiv preprint arXiv:2304.1175,, 2023.
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug- and-play priors. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: htttpa://ppenreeee/ i id= (atehyh12222222
Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Deep back-projection networks for super-resolution. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1664-667,, 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 15:6840-6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint
2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint 2:XY::2204.03458, 2022b.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 34:1444 14:18938-- 14954,, 2021.
Rudolph Emil Kalman. A new approach linear filtering and prediction problems. 1960.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4411 2019.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochas- tically. Advances in Neural Information Processing Systems, 34:21757-21769, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. arXiv preprint arXiv:2201.1 11793, 2022.
Kwanyoung Kim and Jong Chul Ye. Noise2score: tweedie's approach to self-supervised image denoising without clean images. Advances Neural Information Processing Systems, 34:864 874, 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro, Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09760 00761, 2020.
Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang- Deblurgan-v2: Deblurring faster and better. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8878-8888 8887, 2019.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681 4681-4690, 2017.
Shitong Luo, Chence Shi, Minkai Xu, and Jian Tang. Predicting molecular conformation via dynamic graph score matching. Advances in Neural Information Processing Systems, 34:19784-19795, 2021.
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. variational perspective on solving inverse problems with diffusion models. arXiv preprint arXiv:2305. 23350:9.00 2023.
Alex Nichol, Prafulla Dhariwal. Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.1074. 2021.

---

Published as a conference paper at ICLR 2024
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. In International Conference on Machine Learning, pp. 9558-9568 PMLR, 2021.
Jascha Sihl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverss--uudeed diffusion models for inverse problems. In International Conference Learning Representations, 2023a.
Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 32483-32222 32498. PMLR, 23-29 Jul 2023b. URL httph:tt//// http:////oooidooeooccooc ing. mm press / v202 song23k html.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 13:12438-12448, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. basre-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13356 2020b.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score- based generative models. arXiv preprint arXiv:2//1.08005 2021.
Maitreya Suin, Kuldeep Purohit, and AN Rajagopalan. Spatially-attentive patch- aich-hierarchical network for adaptive motion deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3606-3615 2020.
Brian I. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. arXiv preprint arXiv:2206. 0411441111 2022.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa-
tion, 23(7):1661-1674 2011.
Luhuan Wu, Brian Trippe, Christian A Naesseth, David M Blei, and John P Cunningham Practical and asymptotically exact conditional sampling in diffusion models. arXiv preprint arXiv:2306. /7775, 2023.
Minkai Xu, Lantao Yu, Yang Song. Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint 22iv:2203300225 2022.
12

---

Published as a conference paper at ICLR 2024
Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson and Minh Do. Semantic image inpainting with deep generative models. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5485-5493, 2017.
Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-lo-image diffusion model in generative ai: A survey, arXiv preprint arXiv:2303.07900 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
13

---

Published as a conference paper at ICLR 2024
A MORE BACKGROUND ON CONTINUOUS DIFFUSION MODELS
Continuous Diffusion Models In the continuous-time diffusion models (Song et al., 2020b), Gaussian noise is sequentially injected to data in the forward process and then we create samples by progressive denoising from noise in the backward process, Generally, the forward process follows stochastic differential equation (SDE) with the following form:
dx, = pxx t)dt + a())dW for E [0,
(14)
x(0)
A(x..jat al .wW
Ao. y
dx (x...t)
00 logo oija 1 w/me) aww,
Memeroment y
-509 Agg (80
(a) Unconditional Generative Model
(b) Conditional Generative Model
Figure 5: Compared with the unconditional case, we replace the score function with the posterior score function in the backward process.
Here, W-(-) is the standard Brownian motion. (x,, f) and (t) are the drift term and the diffusion term respectively. The diffusion process starts from x(0) ~ Pdata (x), the latent distribution data, and x(T) (x), which is almost a pure Gaussian noise. As is stated in Anderson (1982), the reverse of the diffusion process (Eq. (14)) is also diffusion process, with the formulation:
dx; [(xx,t) ((t)2Vx,
Vx. log dt (t)
(15)
Here, W, is a standard Brownian motion with time flows backwards from to O. Vx, (x;)) is the score function of the noisy data distribution time t, which is learned via denoising score matching (Vincent, 2011) 1). is worth mentioned that Eq. (15) is not the only backward diffusion process whose solution trajectories sampled at follow the distribution Pe (x.). Practically, widely used framework is the variance preserving SDE (VP-SDE) in Dhariwal & Nichol (2021):
B(t) 2
dx,
x,dt + V (()
(16)
Here B(t) > is the noise schedule, which is usually chosen as a linear function over t. standard Gaussian distribution can be achieved for = T. i.e. xT NoO, I). Its corresponding backward diffusion is:
B(t) 2
dx; =
x, - B(t) log (xx) dt + /B(t)dW,
(17)
Conditional Diffusion Models following SDE
For most existing conditional diffusion models, they analyze the
dx; = B(t) Xx - Vx) Ix x) dt + VB(j)wW.
(18)
where the score function Vx, Jog (x;:) is replaced by the conditional score function Vx, log Pe y). The key problem falls onto the estimation of conditional score function without re-training. According to the Bayes rule, it holds that:
Vx, log (x:l)((x)))) = Vx, log Jog.e(xx + Vx, loyl())
The first term is the unconditional score function, which can be estimated by our pretrained score estimator. The second term does not have explicit form even in the linear inverse problems. In the
14

---

Published as a conference paper at ICLR 2024
projection-type approaches such Song et al. (2020b); Chung et al. (2022c); Chung & Ye (2022); ?, the authors consider the noiseless case where n = 0. After an unconditional update, they ignore the posterior score function Vx, log p(y ). Instead, they conduct projection step keep the model consistency y Ax still holds. SVD-type approaches Kawar et al. (2022) use SVD to simplify the linear inverse problem to a component-wise unconditional diffusion model. Besides, another line of works such as Chung et al. (2022a); Kawar et al. (2021): Bardsley et al. (2014); Song et al. (2023a); Jalal et al. (2021) try different ways provide close approximation of Vx, log p(y[xe). Their main idea is to use Xx, log (y x.) = Xx, log (y & (xx))) to approximate the conditional score with another explicit score. Jalal et al. (2021) utilize the approximation Vx, log (y x.) ~ A Ax- where (1% } are based on which Song et al. (2023a) propose Pr X2) ~ (A(xo(x))) r? AA T + o) Here,
o2t2l
Chung et al. (2022a) explicitly compute Xx, log log (xx)).)
* (x+) E(xo
X = + V x; log)((x;)
is the score-based estimated initial data point X given its noisy version X₃- Besides the mean E[xo xr], its variance additionally considered in Boys et al. (2023). Another way approximate the conditional score is directly tuning the score network (i.e. U-Net) over y, which is recently proposed and implemented by Barbano et al. (2023); Chung et al. (2023).
Instead of upproximating the conditional score function at each time step. we use a completely different framework and apply Bayesian filtering model to approximate the posterior sampling directly. This method is computationally efficient since we do not need to take derivative on estimated score function.
Monte Carlo based Diffusion Models and Asymptotic Consistency A number of results are proposed recently on Monte Carlo based conditional diffusion models, including several concurrent works. Trippe et al. (2022) developed an efficient method SMCDiff to sample scaffolds from distribution conditioned on given motif, which is popular and pivotal task in protein design. This prior work only considered the inpainting-type task, which is just special type of linear inverse problem. Besides, the proposed asymptotic consistency requires the perfect match between forward and backward unconditional diffusion process, just like our Proposition 4.1. Based on this work, Wu et al. (2023) introduced the Twisted Diffusion Sampler (TDS), a sequential Monte Carlo (SMC) algorithm conditional diffusion models as well. By using a constant {yx JN sequence, there is no explicit formulation on p(Xx X Yk) for y yo. A tractable twisting function and Monte Carlo based technique are needed to approximate the probability. For the asymptotic consistency, several regularity conditions are required on the twisting function sequence. As simultancous work, we use different methods to solve the same problem. By introducing an appropriately designed {ys) sequence, we transform the posterior sampling problem into Bayesian filtering problem where all transition probabilities are explicit, which can be elegantly solved by variance reduced Sequential Monte Carlo with asymptotic consisteney holds. Another concurrent work proposed by Cardoso et al. (2023) introduces Monte Carlo guided diffusion (MCGdiff) to solve Bayesian linear inverse problems. Compared to Wu et al. (2023), different proposal kernel is used on SMC.
Formulations of DDPM and DDIM In this part, we still use the notation { {ap N and {
IN introduced in Section 2. For the Denoising Diffusion Probabilistic Model (DDPM), the forward process is Markovian with
q(xI:N xo) II {x*x xx-,), where q(*a xI=1) N -N (Vanxi-1, H. k=1
A direct corollary for the forward process is that q(xx xo) = -N -N (1-ax)() a)) which leads to the following expression for X5:
X v 0g + VI Cg. where I).
With Xg added to the transition d(xx X;-1, Xo), we can keep the marginals unchanged but the forward process not Markovian any more. For the backward process proposed in Denoising Diffusion Implicit Model (DDIM) (Song et al., 2020a), we fix the prior px())) = =N(0,) /) and we can generate Xk-1 from X via:
X5 VI (xE)( + v σ - -o x5(()) + +
X4-1 = 0
Vap
15

---

Published as a conference paper at ICLR 2024
Here, E& ~ ~V(0., I) is standard Gaussian noise independent of x5 and
(xx)) ~ Vxx Xx log pk (xx) 1t/.vV-irr
approximated sample-dependent noise learned by score matching. Noise that different choices of variance de leads to different backward processes all them keep the trajectory with the same marginal distributions (x+). When we choose o 3x
1-og_ 1 for all k. the process degrades to T-
the case of DDPM.
B EXPLICIT FORMULATION OF FPS WITH DDIM PLUGGED IN
As a compliment our framework described in the main text, we make explicit computation in this section for FPS with DDIM plugged in. First, we introduce the duplex forward diffusion and quantitatively show the connection between posterior sampling and the Bayesian filtering problem.
B.1 DUPLEX FORWARD DIFFUSION PROCESS
For the discrete case of diffusion models such DDPM and DDIM, their forward diffusion can be expressed as the following recursive linear Gaussian process:
X = v 3x X + BB
(19)
where 3ς I), is standard Gaussian noise independent with X;-1- Mcanwhile, we make:
y& /1 Bx Yk-1 AB
(20)
as the forward process for measurements where Yo y the initial measurement. Notice that these two { represent the exactly same noise vector. So, we combine data and measurement as pair, the forward SDE can be defined the following form:
(t) 2
ID
0
d y,
=
y1
dt
B(t)
A
dw
Here, W, is the standard 2 -dddiennii Wiener process. After simple calculations over Equations (19), (20), we can directly build connection between Xo. Yo and Xx, Proposition B.1. the case of duplex forward diffussin procexs, we can simply represent by:
Xk: = ook- Xo + vT ork 2, and yk = -Vok yo + vT -ap . Az, 2 2~ N0,
We can also conclude that:
g(yk x4) N(xx,, d² Ia);
which means yk also follows normal distribution with mean Ax and decreasing variance o² Ozk. For the initial step l: = 0, we have oig = 1 and it degrades to the model consistency function yy Axp where ~ (00,
Unlike X which we can only estimate, we do know yo. the initial measurement without any extra noise, which makes the score function for y: tractable.
Proposition B.2. For the case of duplex forward diffusion process, the score function y is:
Vy, log qu (y())
1
(AAT)-1 (yx Vowyo).
a
According to Equation (20), the distribution of sequence (yx) given the measurement y is tractable since the diffusion sampling a Markov chain:
g VNN yo)
1(ye yy--))
For conditional sampling, do not know the distribution of { but know that of {yx N Therefore, calculating the hidden Markov chain from the ] sequence is equivalent to solving a reverse-time Bayesian filtering problem since we have access to po xx)) = g(yx. xx)) from Proposition B.1 and po (xx-1 Xx) from score-based backward diffusion process.
k=
16

---

Published as a conference paper at ICLR 2024
B.2 BACKWARD PROCESS OF FILTERING POSTERIOR SAMPLING
Step 1: Generating Sequence {yk
NN
For the DDIM backward process with regard to {vk k==
we have:
c) ak-1) (yx agSo) + vol1 Tk-1)) As.
Yk-1 = aV-1V0
Ck
where S NO, 1) and y0 Elyo yı] is a posterior mean calculated by Tweedie's formula (Efron, 2011; Kim & Ye, 2021). Here, Yo since the ground truth measurement yo is given and no estimation or approximation is needed.
c) ai--1)
Y = V 1Yo
-
(yx agyo) + vo(1 ak-1)) AE.
(21)
0k
For yn. we have already proved that qyYN = =N)==NV(aNoo (1 -xx A A)) in Proposition B.1. Since v,y converges to 0 for large enough N, therefore directly make the following sampling:
YN AEN for 1).
(22)
Remark B.1. Since Yo is given in the case of linear inverse problems, also reasonable to directly use the forward process of sequence {yk) where yk VI 3x Yk-1 + VBa Az for by NO, I) since both the forward and backward sequences have exactly the same marginal distributions qu (y)). In this paper. aligned.
the backward process make our model structurally
Step 2: Generating Backward Sequence of {xx Jizo Given the {y sequence given, we continue generate the {x backward sequence by using the filtering method. In this section, provide the explicit Gaussian formulation of each po (x Vk-1) by using Eq. (9), which depends on the unconditional update model
x Xx) = N' Nuu (HuR(X,, 0), 2x)
we choose to plug in. For example, for DDPM framework, we have:
BE
Bx - 0i-1))
[4 (xx, 0)
X
+
(o(k,, (x) and Σg
Oak
Vak
ak
where so (Xk: fk) is the pretrained approximation for the score function Vxa (og.Px(X44) For the DDIM framework, we have more general formulation with:
Pm (xk, 0) (xe + Txx 80 (x))xx,t))) - V(1 -1 - - (1-)())))) s0((xx; (x).
a&
and
Σ = o3 I.
We have claimed that when of we apply the more general DDIM framework and make of = =c(1= Tz-1) with E [0. 1] left as tunable syper-parameter which makes:
1 the DDIM framework becomes a DDPM. Therefore,
1-0%
DDIM Xx, p
8)
(x+ (1-0a )s0 So
c) σ 1) (1 (K So (oxx,,t (23)
)
a
and the covariance matrix
Σ
c(1 -
(24)
After combining Po (Yk-1 xx-1) and Po px xx). the following conclusion can be obtained by linear filtering.
17

---

Published as a conference paper at ICLR 2024
Proposition B.3. Under the case that
po py Xk-1) (Ax d I) and po (x X(s) N ( (ADDIM(X (xi, O), ERDIM)
with pk
ADDM Q) and 2
EDDIM defined in Equations (23), (24). We can conclude that:
PPS po(xx-1 Xx. Yk-1) = (u (Kss Yk-1,
2)x
where
DPP
(MRDIM) -1
A A
0². 1 Tk--
and
PPS 14 (x 0) =>
PPPS
(SDDIM) -1
14
DDIM
(X,, 0) + r² O1--
A T yy-1).
Till now, we have figured out all the details of FPS as well as its variant FPS-SMC, We rewrite the aoouudo-lgrrihmmss of FPS-SMC with DDIM plugged in the following Algorithm 2. When the particle size M = 1, it degrades to FPS.
Algorithm 2 The Framework of FPS SMC with DDIM Plugged-in
1: Given: N,y, { hE[N) c, particle size and the score estimator sg(:,))
2: yN AEN, X = EN EN ~ N(O, I) for j E [M].
3: for k N, N 1, 1..1.d. do
(1-c) .9h Ve(l ak-1)
4: Pk
T-Th
5: Yk-1 Vāk-1V + Pb( vaxy) + Ag
6: end for
7: for k
==
N 1,,,..., do
(i) 8: Sample x(61 (u ( 14 Vk. 0). MPPS) for j 1,2,...,
FPS where μ 1kt1 (x 0), k+1 are defined in Proposition B.3. ()) with probability distribution (13). We
9: Pick M samples with replacement from xx jE(ar]
denote the sample set as
(
JE[M]
end for
11: The output Xg is uniformly sampled from (x) hje[M]-
C PROOF OF PROPOSITIONS IN APPENDIX B
In this section, we provide detailed proofs for the three propositions stated in Section 3.
C. PROOF PROPOSITION B.
According to the duplex forward diffusion framework.
X Van Xn /1 C. Yk VoL yo + v1 ag Az.
We have:
Yk Ax vā. (yo Axo).
Since yo- Axo ~ N(0, Ia). we can easily conclude that:
Y& Ax ~ NN(0, Ia),
which is equivalent to:
Yk N(Axx, Par Ia)-
18

---

Published as a conference paper at ICLR 2024
C.2 PROOF OF PROPOSITION B.2
Given the fact that
Y& (V (1- AAT):
ax)
the density function as well as the score function can be explicitly written as:
(yk) & exp (-2(x² -oayo)
(AAT)-
(y: - auvoo)
1- ak
and
yy () 02K
(AAT)- (yx -agyo).
which comes to our conclusion.
C.3 PROOF OF PROPOSITION B.3
For simplicity, denote
po (y x1- = NN(1, Σ1) and px 1 x1.) = -N(T2, Ez).
where 71 Ax /1 DDIM (xx: 0) and Σ1 02 I,E2 2-k
5DDIM
Then,
po Xx, Yk-1) x pe (x X Yk-1) = pe (x xx) pe py xx-1) pe (x*).
Here, we use the fact that Yk-1 and X₁ are independent given x;-1- Therefore:
Pe px Xx. Vk-1) x (x xx) -po x(-1);
which leads to:
Xs. Yk-1) x exp (-
1 1 (x(x--1 T2) -2 (xx-1 -21) T2) - - -yyn-1 - -Ti)
po(x
eexp
xx exp
X k--
Dz1 Xk-1 + T32 Lnz Xk-1
2 T
x ETTT AAx=1 + -b- ET- Axx-1
= exp
2 x (2ig)
Σj A) X4-1 + (2ig T2 + A - yk-1) Xk-1
Therefore, we have pe (xx-1
X y yk-1) = N( PPS (Xs. (XH, Yk- 0), LPPS) where
-1 1 A A
EPPS (Enz)
A Σー A) A)
(5IDDIM)-
0²
P Yx- 0) PPS ( T2 + A y:-1)
PPS
PPS Σ
(> (DIDIM))
-1 DDIM
(xks 0) + 02 . -ax-
A ATyy y2-1)
which comes conclusion.
D THEORETICAL UNDERSTANDING FPS AND FPS-SMC
D. THE CONTINUOUS LIMIT OF FPS
In this section, we are going to reveal the theoretical properties of Filtering Posterior Sampling (FPS) in the continuous setting where the time step At - 0. In the Diffusion Posterior Sampling (DPS), the backward diffusion process has the following formulation:
B(t) x; - 3() log (x y) dt + v 3(tddW.
dx;
2
Actually, when using diffusion models to solve inverse problem posterior sampling, the SDE above is the only framework to follow. The only difference how to approximate Vx, log p. (x, y). Here, are going to theoretically answer the following question:
19

---

Published as a conference paper at ICLR 2024
For the continuous FPS, what is our actual approximation for Xx, log pi(e y)?
Notice that when At - 0, ied DDIM and EDDIM have the following form:
[(t) x, B(t)Vx, x, log pi At,
DDIM B(t) I, f4 (NDDMM(XE (xx) +
which match the Euler- Malle--aruyama form of the backward diffusion SDE. Then, for ue! PPS and Σ, EFPS have:
2
>.PS
(>:DDIM)
A A
3(t)at+I. I,
0² at
and
PPS S.PPs.
(DDDIM) -1 DDDM (x₂)
1
AT yt
f4
0² at
(EPDIM (2, ( mi (NDDIM(XX (xx)
1
YPPS (ATyt)
02 a
B(t) A ATA) A) 1r. (X.DM(()))
B(t)
A'y Yt
g².0z aht
0² Cx.
B(t) X1 + ((1)xx. lx) AL
3(t)
Xx +
(A yv A Ax.).
We have already known (y xx) [[Ax,, aa I), it holds that:
1
V logply: x1) r2 . (A A Ax.).
Therefore, we can finally conclude that
B(t)
H1 = X4 +
X2 B(t
log (x))(())) At + B(t) Vx, logply: xx))
2
B(t)
Xt +
Xt + Vx, lo yr) At.
2
Therefore, when time step At + 0, the continuous limit of FPS should be
3() 2 X - Vx, (x y1) dt + /3(tJdW.
dx;
D.2 THE CONSISTENCY OF FPS-SMC
In this section, we are going to use the method of induction to prove the Proposition 4.1. Before the official proof, I would like to introduce some useful properties weak convergence over distributions. For simplicity of notations, we denote p po in this proof.
Lemma D.1. Assume sequence of distributions over support set Ω satisfies pn. T p" when c: + R is a continuous and bounded function over Q. Then: denote
c(x)Px(xx
c(x)p=(xx
qm qx((x) (x)x))x(x)xxx and (x(x) = [((x)pp (x)dx
It holds that qu. q" when n - oc.
Lemma D.2. Assume sequence of distributions over support set Q satisfies p.. 3 p" when
q(x' x) is conditional distribution. Denote
ra(xx)) = -f (x)xxx and rx(x) = f g(xe) xx)p (x)dx.
Then holdy that vn. y- when -> XX.
20

---

Published as a conference paper at ICLR 2024
Lemma D.3. Assume sequence of distributions over support set Z x S2 satisfies pro (xx,xx)) 28 p" xxxxxx') when n - oc. Denote
qn(x)= = - Pro Xx(xxx and q"(x) / p x'(xxx)dxx.
Then it holds that On 2h qp when xc.
These three lemmas can be easily proved by the definition of weak convergence. According to Assumption 4,1, we know that p(xx Xx+1)== = =p" (x& Xx+1) holds for all Since p(y: xx) =
p" p"(v xx) is fixed and known in both forward and backward processes, we have:
p(X;
1s) oc p(Xs
X6++ (y(x
xx) = p (xe Xx+1 )p" xx) x p" Xx+1,YE),
which leads to p(x: Xx+1.ya) = p" (xx Mk.. Now we start proving:
p yk:N) 2 (xx YEEN) when M 00
(25)
by using the method of induction over k. We have known that (xx) } are i.idd samples from p" (xN yN), which means Eq. (25) holds for l N. If the equation holds for ls + 1, then (x are iid samples from p(x++1 Yk+1:N). For each j E [M], generate
~ p(*
xx)
For each j. (x
(5) xx+1) follows the distribution
(R,, X yE:N) x Xx+1, X4) YK+1:N) Ki p' Xk+1, xx)+1,8) (x++1 yk+1:))-
The next step is the resampling, whose weights are c(x(²) xx+1) = p(ys xx()) Since p(ys
Xk+1) a Gaussian distribution, the weight function c(:,)) .) continuous and bounded. For each continuous bounded function f and each E [M]. by the Law of Large Numbers:
Ex(x,, x6+1) ΣØe c(x()) xx-1) 1(x]) x
Σ/ c(xx) (1) (5)
.x++1)
ag I [((x,, xx+1) (xk: xk+1) q(X,, X6+1 YKEN dxx
I c(xx, xx+1) ((x,, X VR. dxx
= / ((xx, X++1 YEEN dxxgdxx41
which implies p(X,, X 1 Yk:N) I r(xs, X YK:N) where r r(Xk; Xx+1 yK.N) & o(xe, xx+1) g(xx, Xx+1 yk:N).
By using Lemma D.I,
r(X,, Xk+1 Yk:N) "1 r" (xx, Xb+1 YkiN) oc c(xk, xxxx) xx+1,y%) p" (xL+1 (E+11N) =yyy xx+1)p(X; (641,y%) p= (xk+1 YK+1.N) p" (Xk, X6+1 Yk:N),
and it finally leads to By Lemma D.3, it holds that:
p(x,, X641 VE.N) 2 p" (Xt. X141 Ye.N).
p(x: yk:N) 3 ppp(xx (xx ykaN)-
Now finish the induction. Let 0, we have:
p(xo yo.n) " p" (xa No.N).
After taking expectation over p(y1.N Yo) and using Lemma D.2:
p(xo yo) Th p" (xo Yo),
which comes conclusion.
21

---

Published as a conference paper at ICLR 2024
E ADDITIONAL EXPERIMENTAL RESULTS
We first outline the configurations for the five tasks we cast as linear inverse problems in image processing. (a) Inpainting task with random-type: we mask each pixel in the image 92% probability, affecting all three RGB channels. (b) Inpainting task with box-type: we select a random 128 x 128 box-shaped region within the 256 256 image and mask out all the pixels within this box, across all three RGB channels. (c) Super resolution: we downscale our 256 x 256 images to 64 x 64 by employing bicubic downsampling with a factor of 4. (d) Gaussian deblurring and motion deblurring: The shape of the Gaussian kernel is 61 x 61 with its intensity to be 3.0, while for motion deblurring, we apply kernel of the same size, but with an intensity of 0.5. As we know, the kernel of Gaussian deblurring is separable, which faciliates memory mappings in the form of (YI + A)- Following Kawar et al. (2022), we replace traditional motion deblurring with anisotropic Gaussian deblurring in experiments, mirroring the approach used in DDRM.
computation of linear
Next, we present additional experimental outcomes the FFHQ 256 256- Ik validation dataset and ImageNet 256 x 256-1k Ik validation dataset on the other two metrics, PSNR and SSIM, in Tables and 4. The results show that our proposed methods achieve higher PSNR values compared to baseline approaches. However, they do not surpass the best existing methods in terms of the SSIM metric. Overall, our FPS and FPS-SMC methods exceed the performance of all current models on the majority of relevant tasks when evaluated using PSNR, LPIPS and FID metrics. Moreover, we illustrate in 6 9 the generated figures by our two methods as well the corresponding inputs and measurements for inpainting (box), super resolution, Gaussian deblurring and inpainting (random) tasks on both FFHQ and ImageNet datasets.
Observing the performance in the inpainting (random) task where 92% of all the pixels are masked out, our methods successfully reconstruct most of the labels, which demonstrates their robustness in handling high undersampling factors. Meanwhile, in the inpainting (box) task, while the generated samples do not perfectly align with the labels, which is a common issue in this task due to the absence of information in the masked image area, they remain structurally consistent and semantically cohesive with the unmasked portions. Moreover, with the FFHQ dataset, our model's ' capability to generate human faces with varied facial expressions further validates its strength in preventing mode collapse and maintaining diversity.
Table 3: Quantitative results (PSNR, SSIM) of our model and existing models on a various of linear inverse problems on FFHQ 256 x 256-1k validation dataset.
Super
Inpainting
Gaussian Deblur
Inpainting (random)
Motion Deblur
FFHQ
Resolution
(box)
Methods
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
FPS
27.48
0.807
24.17
0.865
26.45
0.773
26.79
0.820
26.70
0.828
FPS-SMC
28.10
0.807
24.70
0.862
26.54
0.773
27.33
0.819
27.39
0.826
DPS
25.67
0.852
22.47
0.873
24.25
0.811
25.23
0.851
24.92
0.859
DDRM
25.36
0.835
22.24
0.869
23.36
0.767
9.19
0.319
MCG
20.05
0.559
19.97
0.703
6.72
0.051
21.57
0.751
PDM-
26.55
0.865
11.65
0.642
24.93
0.812
8.41
0.325
Score-SDE
17.62
0.617
18.51
0.678
7.12
0.109
13.52
0.437
ADMM-TV
23.86
0.803
17.81
0.814
22.37
0.801
22.03
0.784
F ABLATION STUDIES
F.1 FORWARD AND BACKWARD GENERATION OF y- D-SEEUENCE
In this section, we focus on the {ye) IN propose. To synchronize the k=o sequence with the seqqueee, we construct our
o sequence employed in the filtering-based methods we -eequeeee
22

---

Published as a conference paper at ICLR 2024
Table 4: Quantitative results (PSNR, SSIM) of our model and existing models on a various of linear inverse problems on ImageNet 256 256-1 1k validation dataset.
Super
Inpainting
Gaussian Deblur
Inpainting (random)
Motion Deblur
Resolution
(box)
Methods
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
FPS
24.32
0.724
20.16
0.752
23.58
0.581
23.39
0.688
22.71
0.598
FPS-SMC
24.78
0.731
22.03
0.748
23.81
0.599
24.12
0.685
23.27
0.614
DPS
23.87
0.781
18.90
0.794
21.97
0.706
22.20
0.739
20.55
0.634
DDRM
24.96
0.790
18.66
0.814
22.73
0.705
14.29
0,403
MCG
13.39
0.227
17.36
0.633
16.32
0.441
19.03
0.546
PnP-ADMM
23.75
0.761
12.70
0.657
21.81
0.669
8.39
0.300
Score-SDE
12.25
0.256
16.48
0.612
15.97
0.436
18.62
0.517
ADMM-TV
22.17
0.679
17.96
0.785
19.99
0.634
20.96
0.676
Measurement
Lahel
FPS
FPS-SMC
Measurement
Label
FPS
FPS-SMC
Figure 6: Examples for inpainting (box) and super resolution over FFHQ dataset
h sequence by using DDIM backward process framework, with an explicit unconditional score function. Unlike existing posterior sampling methods such DPS (Chung al., 2022a) where y1 = y2 = yN = or conditional score-based generative models (Song et al., 2020b) that sample y& N(y, of I). the noise introduced to y at each step our methods is correlated as shown in Eq. (26)
2Ą ~ I) independent with each other, (26) ensures that our generated sequence (yx) continuous, with incremental noise added at each step.
Y& / B& Yk- B Azk.
Here, we conduct separate experiments demonstrate the contrast between these two different ways of generating the [ sequence. In the FPS method proposed by us, we use DDIM backward process make Eq. (26) holds. Conversely, in the control group, the FPS with Independent Noise (FPS-IN) model generates the (yx N sequence in simpler manner by:
Y* JRy vI T4
Azz, Z₁ NNO I) are independent with each other.
Meanwhile, to ensure the validity of our ablation study, we maintain all other experimental conditions identical. From the following Table 5,
that in all the five tasks and both of the two datasets,
23

---

Published as a conference paper at ICLR 2024
Measarcment
Label
FPS
FPS-SMIC
Measurement
Label
FPS
FPS-8MC
Figure 7: Examples for Gaussian deblurring and inpainting (random) over FFHQ dataset
Measurement
FPS
FPS-SMC
Measurement
FPS-SMC
Figure 8: Examples for inpainting (box) and super resolution ImageNet dataset
FPS consistently outperforms the control group, FPS-IN. This outcome affirms the significance of the additive noise correlation in the [y sequence for the model's performance.
Furthermore, generate two sets of images by using FPS and the control group. FPS-IN, for the inpainting (box) task, with the outcomes presented in Fig. 10. Both methods of generating {yx N sequence enable model to perform effectively, creating well-composed images. Comparatively. the images generated by FPS tend to be more accurate and closely match the unmasked portions of the labels.
24

---

Published as a conference paper at ICLR 2024
Measurement
Lahel
FPS
FPS-SMC
Label
Figure 9: Examples for Gaussian deblurring and inpainting (random) over ImageNet dataset
Table 5: Quantitative results (FID, LPIPS) of FPS and our control group. FPS-IN on FFHQ 256 256-
1k validation dataset and ImageNet 256 x 256- Ik validation dataset.
Super
Inpainting (box)
Gaussian Deblur
Inpainting (random)
Motion Deblur
FFHQ
Resolution
Methods
FID
I.PISS
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
L.IIS
FPS
26.66
0.212
26.13
0.141
30.03
0.248
35.21
0.265
26.18
0.221
FPS-IN
27.18
0.247
27.84
0.169
32.79
0.251
39.90
0.316
26.86
0.239
Super
Inpainting (box)
Gaussian Deblur
Inpainting (random)
Motion Deblur
ImageNet
Resolution
Methods
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
FPS
47.32
0.329
33.19
0.204
54.41
0.396
42.68
0.325
52.22
0.370
FPS-IN
50.06
0.541
34.21
0.255
56.21
0.409
44.40
0.337
53.10
0.389
F.2 RUNNING TIME OF FPS-SMC WITH INCREASING PARTICLE SIZE
In this section, we compare the wall-clock running time of FPS and FPS-SMC with other popular posterior sampling models. By using single A A100 GPU, the running time for each algorithm is listed in Table 6. As we can see, although not the fastest, FPS provides running speed faster than average for the posterior sampling in the inpainting (box) task of FFHQ dataset. Furthermore, we study the running time of FPS-SMC with increasing particle size. Despite the fact that the computational cost should be proportional to the particle size M, we can reduce the running time by batch-wise operators while coding. According Table 7, the running time t(M) approximately holds t(M) x VM.
F.3 INFLUENCE THE NOISE LEVEL IN DDIM FRAMEWORK
Within the DDIM unconditional generative framework, the noise level [0, 1] serves pre- determined eyper-parameter. According to our experiments, we notice that varying e leads to distinct generated images, with each task presenting optimal choice of Therefore, in this section, explore how the noise level e affects the quality of generated images. As evident in Figs. 11 and 12, excessively high value results in over-smoothed images, while very low causes insufficient smoothing, degrading image quality. We thus select task-specific values, with our chosen parameters detailed in Table 8.
25

---

Published as a conference paper at ICLR 2024
Label
Label
Figure 10: Comparison of FPS and the control group, FPS-IN under the inpainting (box) task on FFHQ and ImageNet dataset
Table 6: Wall-clock running time for single poste- rior sampling on the inpainting (box) task of FFHQ-Ik- validation dataset, by using a single A100 GPU
Table The connection between the running time of FPS-SMC and the partcle size M, by using a single A100 GPU
Running time (in seconds)
Model
Running time (in seconds)
Particle Size
FPS (our paper)
33.07
Score-SDE (Song et al., 2020b)
32.93
M (FPS)
33.07
DPS (Chung et al., 2022a)
70.42
M 2
39.15
DDRM (Kawar et al., 2022)
2.034
M 5
57.88
IIGDM (Song et al., 2023a)
33.18
M 10
82.12
MCG (Chung et al., 2022b)
73.16
M 20
116.90
PnP-ADMM (Chan al., 2016)
3.595
100
283.53
Table 8: Different values we choose in various tasks over datasets FFHQ and ImageNet
Super Resolution
Inpainting (box)
Gaussian Deblur
Inpainting (random)
Motion Deblur
Dataset
FFHQ
0.3
0.95
0.3
0.95
0.3
ImageNet
0.15
0.25
0.3
0.25
0.3
26

---

Published as a conference paper at ICLR 2024
measurement
label
c=
0.25
0.5
c= 0.75
c= 1.0
Figure 11: Generated FFHQ images by FPS with different noise level in inpainting (box) task
measurement
label
0.25
0.5
0.75
1.0
Figure 12: Generated ImageNet images by FPS with different noise level in super resolution task
27


Reviews:
[{"review_id": "riOqUBGekQ", "reviewer": "Reviewer_b9JP", "summary": "This paper presents an SMC algorithm for sampling from the posterior of an inverse problem where the prior distribution distribution is a diffusion model, which is a natural way of sampling from the posterior of the diffusion model, due to the sequential structure of the sampling procedure. The resulting procedure does not require any additional training and, as claimed by the authors, samples from the posterior in the limit of infinite particles.", "questions": "i have no further questions", "limitations": "", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "- The proposed method provides impressive results on inpainting tasks. Interestingly, other methods for inpainting with a diffusion prior often suffer from inconsistencies at the the borders of the patch. This is not the case of this method.", "weaknesses": "- I believe that method proposed in the paper is technically flawed. In the section 3 of the main paper, the authors give the state space model that they consider where notably, the use the same noise at step $k$ for both the observation and the state. They then proceed to claim that it holds that $p(x_0 | y_0) = \\int p(x_0 | y_{0:N}) p(y_{1:N} | y_0) \\mathrm{d}y_{1:N}$ and thus, to sample from $p(x_0 | y_0)$ it is enough to sample $p(y_{1:N} | y_0)$ and then sample from $p(x_0 | y_{0:n})$ using an SMC algorithm. However, this is not correct since $p(x_0 | y_{0:N}) = p(x_0 | y_0)$ for the state model that they consider. To show that this is the case, consider the case $N=1$. Then, the joint distribution of the SSM they consider is the following, since they share the noise: \n$$\np(y_{0:1}, x_{0:1}) = p(y_0 | x_0) p(x_0) \\int p(\\mathrm{d} z) \\delta_{a_1 x_0 + b_1 z} (x_1)   \\delta_{a_1 y_0 + b_1 A z} (y_1)\n$$\nand thus, \n$$\np(x_0 | y_{0:1}) = \\frac{\\int p(y_{0:1}, x_{0:1}) \\mathrm{d} x_{1}}{\\int p(y_{0:1}, x_{0:1}) \\mathrm{d} x_{1} \\mathrm{d} x_0} = \\frac{p(y_0 | x_0) p(x_0) p(y_1 | y_0)}{\\int p(y_0 | x_0) p(x_0) p(y_1 | y_0) \\mathrm{d} x_0} = p(x_0 | y_0)\n$$\nBesides this fact, the methodology developed in the rest of the paper does not in fact sample from the correct posterior asymptotically. To see why this is the case, note that the state space model on which the authors apply SMC is the following: \n$$ \np_\\theta(x_{0:N}, y_{0:N}) = p_N(y_N | x_N) p_N(x_N) \\prod_{s = 0}^{N-1} p_\\theta(x_s | x_{s+1}) p(y_s | x_s) \n$$ \nIt is straightforward to see that the p_\\theta(x_0 | y_0) resulting from this model is **not** the target posterior $p^*(x_0 | y_0) \\propto p(y_0 | x_0) p_\\theta(x_0)$.  \n\n- The idea of using the same noise for the forward process and the observations is not new and is used in [1], which the authors cite but they fail to mention that they use the same idea. The authors also fail to mention author work on SMC applied to diffusion posterior sampling [2]. In fact in this paper the authors use a specific decomposition of the posterior, similar to what is claimed in the text box at the end of the section 3 of this paper. In contrast, their decomposition is theoretically justified but holds under stringent assumptions on the backward process, further confirming that what is claimed in this paper is not true. \n\n- Finally, this paper is not the first one to develop a consistent diffusion posterior sampling algorithm, see [3] and [4] which develop **principled** SMC algorithms for asympotitcally exact posterior sampling. This is not a criticism, as these papers have been released 3/4 months ago and I understand that the authors may not have had knowledge of them. \n\n\n[1] Song, Yang, et al. \"Solving inverse problems in medical imaging with score-based generative models.\" arXiv preprint arXiv:2111.08005 (2021).\n\n[2] Trippe, Brian L., et al. \"Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem.\" arXiv preprint arXiv:2206.04119 (2022).\n\n[3] Wu, Luhuan, et al. \"Practical and asymptotically exact conditional sampling in diffusion models.\" arXiv preprint arXiv:2306.17775 (2023).\n\n[4] Cardoso, Gabriel, et al. \"Monte Carlo guided Diffusion for Bayesian linear inverse problems.\" arXiv preprint arXiv:2308.07983 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EYjtwh3asq", "reviewer": "Reviewer_x48M", "summary": "This work introduces an efficient, asymptotically accurate diffusion sampling algorithm for linear inverse problems, linking Bayesian posterior sampling to Bayesian filtering in diffusion models, using sequential Monte Carlo methods for filtering, requiring no model re-training, and outperforming or matching existing methods in tasks like image inpainting, super-resolution, and motion deblur.", "questions": "Is there a running time or NFE evaluation for the algorithm? \n\nAlthough the algorithm is designed to solve linear inverse problems, does it work on non-linear ones, which may happen in real practice? For DPS, they have shown the possibilities to solve nonlinear problems.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Solving inverse problems is a fundamental problem in machine learning and other fields. Diffusion models have a good potential to solve those problems, and it is a good attempts to provide a solution based on the filtering perspective.", "weaknesses": "The literature review for diffusion-based inverse problems are quite limited. It would be very important to compare with previous work to demonstrate the advantage and necessity of the new algorithm.\n\nThe scale of empirical experiments are relatively small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4qbWupP65C", "reviewer": "Reviewer_pD39", "summary": "This paper proposed a filtering perspective on the linear inverse problem with diffusion models, which yields a novel FPS algorithm. Moreover, this paper gave a theoretical proof that that the FPS algorithm correctly samples from the Bayesian posterior\ndistribution as the number of particles approaches infinity.", "questions": "1. How did you add noise with standard deviation \\sigma= 0.05? Did you account for the scaling of [-1, 1] so that the actual standard deviation added is 2 * \\sigma=  0.05*2 = 1, as did in DDRM? The code is not provided so that I cannot check this point.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A novel filtering perspective on solving linear inverse problems using diffusion models, which is interesting and inspiring.\n2. A theoretical proof is given that the FPS algorithm can correctly sample from the Bayesian posterior\ndistribution as the number of particles approaches infinity.", "weaknesses": "1. Compared to previous methods like DPS (in fact, currently, DPS is no longer the SOTA), the improvement seems marginal in some cases, and even worse in some cases like inpainting. From my understanding, inpainting is among the most challenging tasks for linear image restoration tasks and the inferiority of FPS might suggest some underlying unaddressed problem for this proposed method.\n\n2. There is a lack of complexity or running time analysis. How fast is FPS compared with other methods? How does the time increase with a different number of particles M?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lCXVGHsIZH", "reviewer": "Reviewer_cbzu", "summary": "This paper provides a link between Bayesian posterior sampling and Bayesian filtering in diffusion models for linear inverse problems. In general, exact posterior sampling is intractable. To overcome this difficulty, this paper introduces a diffusion process to the measurement vector and incorporates the information of the measurement by Bayesian filtering methods in the backward process.  In the ideal case, the exactness of the proposed method is theoretically guaranteed, which is realized by the sequential Monte Carlo method using infinitely many particles. Numerical experiments show that the performance of the method is as good as those of the SOTA methods.", "questions": "It is possible to extend the proposed method for nonlinear measurement cases?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This article provides an important perspective that has been missing until now for the linear inverse problem using diffusion models. The proposed method is reasonable in the Bayesian framework, and achieves as good performance as the current SOTA methods.", "weaknesses": "Extendability to nonlinear measurement cases is not clear. FPS-SMC, which uses multiple particles, requires considerably heavy computational cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)