Monte Carlo guided Denoising Diffusion models for Bayesian linear inverse problems.
Authors: Gabriel Cardoso
Yazid Janati el idrissi
Sylvain Le Corff
Eric Moulines
Venue: ICLR 2024 oral (2024)
Keywords: Monte Carlo
Denoising Diffusion model
score-based generative models
Sequential Monte Carlo
Bayesian Inverse Problems
Generative Models.
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Ill-posed linear inverse problems arise frequently in various applications, from computational photography to medical imaging.
A recent line of research exploits Bayesian inference with informative priors to handle the ill-posedness of such problems.
Amongst such priors, score-based generative models (SGM) have recently been successfully applied to several different inverse problems.
In this study, we exploit the particular structure of the prior defined by the SGM to define a sequence of intermediate linear inverse problems. As the noise level decreases, the posteriors of these inverse problems get closer to the target posterior of the original inverse problem. 
To sample from this sequence of posteriors, we propose the use of Sequential Monte Carlo (SMC) methods.
The proposed algorithm, \algo, is shown to be theoretically grounded and we provide numerical simulations showing that it outperforms competing baselines when dealing with ill-posed inverse problems in a Bayesian setting.

Published as a conference paper at ICLR 2024
MONTE CARLO GUIDED DENOISING DIFFUSION MODELS FOR BAYESIAN LINEAR INVERSE PROBLEMS
Gabriel Cardoso Ecole polytechnique IHU Lirye
Yazid Janati Ecole polytechnique
Sylvain Le Corff Sorbonne Université
Eric Moulines Ecole polytechnique
ABSTRACT
Ill-posed linear inverse problems arise frequently various applications, from computational photography to medical imaging. A recent line of research exploits Bayesian inference with informative priors to handle the ill-posedness based generative models (SGM) have recently been successfully applied to several different inverse problems. In this paper, we exploit the particular structure of the prior defined by the SGM define sequence of intermediate linear inverse problems. As the noise level decreases, the posterior distributions of these inverse problems get closer to the target posterior of the original inverse problem. To sample from these distributions, propose the use of Sequential Monte Carlo (SMC) methods. The proposed algorithm, MCGdi ff, is shown to be theoretically grounded and we provide numerical simulations showing that it outperforms competing baselines when dealing with ill-posed inverse problems in a Bayesian setting.
such problems. Amongst such priors, score-
LNTRODUCTION
This paper is concerned with linear inverse problems = Ax + TyE, where y E Rd is a vector of indirect observations, E Rd is the vector of unknowns, A E Rd, is the linear forward operator and & E IR is an unknown noise vector. This general model is used throughout computational imaging, including various tomographic imaging applications such as common types of magnetic resonance imaging Vlaardingerbroek & Boer (2013), X-ray computed tomography Elbakri & Fessler (2002), radar imaging Cheney & Borden (2009), and basic image restoration tasks such deblurring, super-resolution, and image inpainting González al. (2009). The classical approach to solving linear inverse problems relies on prior knowledge about æ, such as its smoothness, sparseness in dictionary, or its geometric properties. These approaches attempt to estimate & by minimizing a regularized inverse problem, 4 = argmin, {lly Ax||2 + Reg(z)), J. where Reg is a regularization term that balances data fidelity and noise while enabling efficient computations. However, common difficulty in the regularized inverse problem is the selection of an appropriate regularizer, which has a decisive influence on the quality the reconstruction.
Whereas regularized inverse problems continue to dominate the field, many alternative statistical formulations have been proposed; see Besag et al. (1991); Idier (2013); Marnissi et al. (2017) and the references therein see Stuart (2010) for a mathematical perspective. A main advantage of statistical approaches is that they allow for uncertainty quantification the reconstructed solution; see Dashti & Stuart (2017). The Bayes' formulation of the regularized inverse problem is based considering the indirect measurement Y. the state X and the noise e as random variables, and to specify p(y]æ) the likelihood (the conditional distribution of Y at X) and Bayes' theorem obtain the posterior distribution where "x" means that the two sides are equal to each other to a multiplicative constant
the prior p(z:) (the distribution of the state). One
p(æ y) x that does not depend on x. Moreover, the use of appropriate method for Bayesian inference allows the
"Both authors contributed equally.
Correspondence: I gabriel. victoorino centttoriooccrddsoooooo cardoso, caadeaa,,ddddddd, janati @@polytechniquee edu

---

Published as a conference paper at ICLR 2024
quantification of the uncertainty in the reconstructed solution æ. A variety of priors available, including but not limited to Laplace Figueiredo et al. (2007), total variation (TV) Kaipio et al. (2000) and Fergus et al. (2006). In the last decade, variety of techniques have been proposed to design and train generative models capable of producing perceptually realistic samples from the original data, even in challenging high- dimensional data such as images or audio Kingma et al. (2019); Kobyzev al. (2020); Gui et al. (2021). Denoising diffusion models have been shown to be particularly effective generative models in this context Socl-Dickstein al. (2015); Song et al. (;;2];;;;); Benton et al. (2022). These models convert noise into the original data domain through a series of denoising steps. A popular approach is to use a generic diffusion model that has been pre-trained, eliminating the need for re-training and making the process more efficient and versatile Trippe et al. (2023); Zhang et al. (2023). Although this was not the main motivation for developing these models, they can of course be used as prior distributions Bayesian inverse problems. This simple observation has led to a new, fast-growing line of research on how linear inverse problems can benefit from the flexibility and expressive power of the recently introduced deep generative models; see Arjomand Bigdeli et al. (2017); Wei et al. (2022); Su et al. (2022); Kaltenbach et al. (2023); Shin & Choi (2023); Zhihang et al. (2023); Sahlström & Tarvainen (2023).
-fausssinns
CONTRIBUTIONS
We propose MCGdiff, a novel algorithm for sampling from the Bayesian posterior of Gaussian linear inverse problems with denoising diffusion model priors. MCGdif ff specifically exploits the structure of both the linear inverse problem and the denoising diffusion generative model to design an efficient SMC sampler.
We establish under sensible assumptions that the empirical distribution of the samples produced by MCGdi ff converges to the target posterior when the number of particles goes to infinity. To the best of our knowledge, MCGdi ff is the first provably consistent algorithm for conditional sampling from the denoising diffusion posteriors.
To evaluate the performance of MCGdi ff, we perform numerical simulations on several examples for which the target posterior distribution is known. Simulation results support our thcoretical results, i.c. the empirical distribution of samples from MCGdiff converges to the target posterior distribution. This is not the case for the competing methods (using the same denoising diffusion generative priors) which are shown, when run with random initialization of the denoising diffusion, to generate a significant number of samples outside the support of the target posterior. We also illustrate samples from MCGdi ff in imaging inverse problems.
Background and notations. This section provides a concise overview of the diffusion model framework and notations used in this paper. We cover the elements that are important for understanding our approach, and we recommend that readers refer to the original papers for complete details and derivations Sohl-Dickstein et al. (2015); Ho et al. (2020); Song et al. (202)c;a). A denoising diffusion model is a generative model consisting of a forward and backward process. The forward process involves sampling Xo Qdata from the data distribution, which is then converted to a sequence X1m of recursively corrupted versions of Xo- The backward process involves sampling X, according to casy-to-sample reference distribution gdd and generating Xn I& by a sequence of denoising steps. Following Sohl-Dickstein et al. (2015); Song et al. (2021a), the forward process can be chosen Markov chain with joint distribution
q0.n (zo.n) = Qdata II. ge (
(<<2:/111) Na (1 - B,) x1-1, Br la.,
(1.1)
where Ia, is the identity matrix of size du {.ee JeeN (0.1) is non non- increasing sequence and .N(x; μ., E) is the p.d.f. of the Gaussian distribution with mean μ and covariance matrix Σ (assumed to be non-singular) evaluated at x. For all t > O, set oe = IT (1 - (3)) with the convention Og = 1. We have for all 0 < 8 < S n,
4n ( (...) == J de ( )dr, -d=,+11-11 1:-1
-xi (ar/a.) 1/2,, (1 āz. ) )
(1.2)
For the standard choices of ãt, the sequence of distributions (qr) converges weakly to the standard normal distribution as l. + oo, which we chose as the reference distribution. For the reverse process, Song et al. (202 la:b) introduce an inference distribution Li (#1. (2).. depending on a sequence (othen of hyperparameters satisfy-
2

---

Published as a conference paper at ICLR 2024
ing of E [0, - -0 for all t N", and defined as gimp (2)) L2 (z (20) I2 8- 121, 200. where G. (mo) N [zn / zu, (1 kx) Ia.) and 14-1 (x N (z 1: pul (Tu, 21), of Id.) with je (2o. 2x)) = 0.-1 + (1 - by backward induction the sequence (Z) (20) = - (40 1 (20)d is shown in (Song et al., 2021a, Lemma 1) that for all t E [1 nl, the distributions of the forward and inference process condi- tioned on the initial state coincide, i.c that Kij (2,|00) = q: (2, (o)). The backward process is derived from the inference distribution by replacing, for each & E [2: n²), zo in the definition ([-1)co (4 (2,, mo) with prediction where (1)(((t) == 0, -1 1/2 (ar - (1 -a) (x,, ()) where e (x,, (1x,)) is typically neural network parameterized
- 0 1/2 50) a,) For L E []
1], we define
by 0. More formally, the backward distribution is defined as (20m)) (r, II= p8 (10a (14), where Pa (=,) = (at, Od.: la.) and for all / E [1 : - 11,
(12( == 1, 1x X01++ (m(+1) = ((a (
(1.3)
where (141(1141) = 1 1xx0 (Te+1), 1,, 1e+1) and Oa is the null vector of size dz. At step O, we set (ouul()))) == (aro x611 (ai), T. ). The parameter is obtained (Song et al., 2021a, Theorem 1) by solving the following
optimization problem:
0_ E arg ming E (2_1(dd - + V1 21-ard acc, t) 1) Od,, Ja (dx)) (1.4)
Thus, e.. (X,,t) t) might be seen as the predictor of the noise added to Xo, to obtain X, (in the forward pass) and justifies the "prediction" terminology. The time 0 marginal Po (.x) I Poz. (foon) )dæz.y which we will refer to as the prior is used as an approximation of qdata and the time marginal is p (xx) = [ Po:n (Tom ( - In the rest of the paper, we drop the dependence on the parameter 0.. We detine for all v E R w R& the concatenation operator wnw = [o w " E BR For i E [1: E). we let e/i]] the i-th coordinate of v.
Related works. The subject of Bayesian problems is very vast, and it is impossible to discuss here all the results obtained in this very rich literature. One of such domains is image restoration problems, such as deblurring, denoising inpainting, which are challenging problems in computer vision that involves restoring a partially observed degraded image. Deep learning techniques are widely used for this task Arjomand Bigdeli al. (2017); Yeh et al. (2018) 8); Xiang et al. (2023); Wei et al. (2022) with many of them relying on auto-encoders, VAES Ivanov al. (2018); Peng al. (2021); Zheng et al. (2019), GANS Yeh et al. (2018); Zeng et al. (2022), or autoregressive transformers Yu et al. (2018); Wan et al. (2021). In what follows, we focus on methods based on denoising diffusion that has recently emerged as a way to produce high-quality realistic samples from the original data distribution on par with the best GANs in terms of image and audio generation, without the intricacies of adversarial training: see Sohl-Dickstein et al. (2015); Song et al. (2021c; 2022). Diffusion-based approaches do not require specific training for degradation types, making them much more versatile and computationally efficient. In Song et al. (2022), noisy linear inverse problems are proposed to be solved by diffusing the degraded observation forward, leading intermediate observations {va and then running modified backward process that promotes consistency with ys at each step s. The Denoising-Diffusion cnffuuiionnRestraatiinn model (DDRM) Kawar et al. also modifies the backward process so that the unobserved part of the state follows the backward process while the observed part is obtained as a noisy weighted sum between the noisy observation and the prediction of the state. As observed by Lugmayr et al. (2022), DDRM is very efficient, but the simple blending used occasionally causes inconsistency in the restoration process. DPS Chung et al. (2023) considers a backward process targeting the posterior. DPS approximates the score of the posterior using the Tweedie formula, which incorporates the learned score of the prior. The approximation error is quantified and shown to decrease when the noise level is large, i.c., when the posterior is close to the prior distribution. As shown in Section with a very simple example, neither DDRM nor DPS can be used to sample the target posterior and therefore do not solve the Bayesian recovery problem (even if we run DDRM and DPS several time with independent initializations). Indeed, we show that DDRM and DPS produce samples under the "prior" distribution (which is generally captured very well by the denoising diffusion model), but which are not consistent with the observations (many samples land in areas with very low likelihood).
3

---

Published as a conference paper at ICLR 2024
In Trippe et al. (2023), the authors introduce SMCdiff, Sequential Monte Carlo-based demoising diffusion model that aims at solving specifically the inpainting problem. SMCdi ff produces a particle approximation of the conditional distribution of the observed part of the state conditionally of the observation. The resulting particle approximation is shown to converge to the true posterior of the SGM under the assumption that the joint laws of the forward and backward processes coincide, which fails be true in realistic setting. In comparison with SMCdiff, MCGdifE ff is a versatile approach that solves any Bayesian linear inverse problem while being consistent under mild assumptions. In parallel to our work, Wu al. (2023) also developed a similar SMC based methodology but with a different proposal kernel.
forward-diffusee trajectory
2 THE MCGD IFF ALGORITHM
In this section, we present our methodology for the inpainting problem equation 2.1, both with noise and without noise. The more general case treated in Section 2.1. Let dy [] d. 1]. In what follows we denote the dy top coordinates of vector x Rdx by and the remaining coordinates by g. that TT.. The inpainting problem is defined as
== X + E g~ =~,(0,,1,)) la,)
02
(2.1)
where X are the first dy coordinates of random variable X Po- The goal is then recover the law of the complete state X given a realisation of the incomplete observation Y and the model equation 2.1.
Noiseless case. We begin by the case = 0. As the first dy coordinates are observed exactly, we aim at infering the remaining coordinates of X,which correspond to X. As such, given an observation y, we aim sampling from the posterior og (Z)) x Po(""" Σn) with integral form
H x (Tn) [IT==] P. (x( Po Zo Zolr1 )ddid
(2.2)
To solve this problem, we propose to use SMC algorithms Doucet et al. (2001); Cappé et al. (2005); Chopin et al. (2020), where a set of N random samples, referred to as particles, is iteratively updated to approximate the posterior distribution. The updates involve, at iteration selecting promising particles from the pool of particles <1:Y- = (62 (f.+1...... eN 1 based on a weight function Wg. and then apply a Markov transition pd is designed to follow the backward process while
to obtain the samples El:N The transition (x guiding the dy top coordinates of the pool of particles El:N towards the measurement y. Note that under the backward dynamics equation 1.3, X, and X, are independent conditionally on X1+1 with transition kernels respectively P. (x(e((2441) == Nxxx;; me+1 (1++1) (,,t), (a,) and g, (2)11))1) == ( ( o2+1 Td,-,)) where ( E IRd and m441 (8+41) E Rdd--d are such that m (xx-1) = m i) m4+1 (#+++) and the above kernels satisfy p. (xa)(eee+)) = p, (1241 )p, (x.(ae))) We consider the following proposal kernels for E [1: n 1),
p. (1a1)144) & (2 Mejo where (Z) af (1 -0))),)))))
(2.3)
For the final step, we define (1)) Po (x)) Using standard Gaussian conjugation formulas, we obtain
1 (x = 2, 2, (1+1)) - (are Kro 2 (.) (1-ā))k, -Ta))
+1 where K, = 0241 +1 For this procedure to target the posterior dg. the weight function W, is chosen
follows; we set "ن (mm)) = Pn- (2n x ) -1o---1oooo (1)dn_nn__1__-111 = = 1- mn (xn)) 0 + an-1)
and for t [1 : - 2],
N( 1/2 y: m (w (o + - ac)lla.)
(x(41)) = I (z I Deo (Tll)dd
(2.4)
1 (Tt+1l1)
N (a2f19: F441, (1 -0141)la,)
4

---

Published as a conference paper at ICLR 2024
For the final step, set (x) := (o(o(z))/71j((3 The overall SMC algorithm targeting o% using the
instrumental kernel equation 2.3 and weight function equation 2.4 is summarized in Algorithm 1. We now provide
Algorithm 1: MCGdifE (a = 0)
Input: Number of particles N
Output: cI:N
21N sid.
~ _ J:
Operations involving index repeated for each ie []
for 1:00 do
ifs n then
1 N N(aa y; (C.), a.):
else
W, (Ei+1) = (a. // y;
a.) /N 1/2 (551th - aut1):
1i Cat( (at(( (G.
Σ - 0 ((
-,~ No(,,, 2t ~N(s, -d, Id.-
k y K. k.)m (3TT k/2 & = =m 0st122:
C E c:
a justification Algorithm 1. Let {g]3==1 be sequence of positive functions with gy Consider the sequence of distributions {ol >>> defined as follows; 6% (xn) x (h (I,)),( and for & E [1 -0-]
of (x)) x gh (#441) 9
244 )o (dx+11)
(2.5)
By construction, the time t marginal equation 2.5 is of (z+) x (z, (z,) for all t E [1 n].. Then, using of and equation 2.2, we have that (Ix) x (x1) Po (v) T Dr, (col in (d()))
The recursion equation 2.5 suggests way of obtaining particle approximation of oh: by sequentially approxi- mating each of we can effectively derive a particle approximation of the posterior: construct the intermediate particle approximations
use the framework of auxiliary particle filters (APF) (Pitt & Shephard, 1999). We focus on the case 9f (xz) = (T which corresponds to Algorithm The initial particle approximation Φ is obtained by drawing N i.i.d. samples g!:N from P.. and setting Φi = N-1 N dg. where de is the Dirac mass E. Assume that the empirical approximation of 6/41 is &N = N ビ EN dei_, wwere E!:Y are N random variables. Substituting di into the recursion equation 2.5 and introducing the instrumental kernel equation 2.3, we obtain the mixture
N (a:) = Σ. (E( (xr ΣSn (E(+))
(2.6)
Then, a particle approximation of equation 2.6 is obtained by sampling N conditionally i.i.d. ancestor indices 1:N Ltd. (11 ~ (Et " 3 1)) and then propagating each ancestor particle / according to
li+1
the instrumental kernel equation 2.3. The final particle approximation is given by Φb - ōg: where S P, (-15f),, I ~ Cat( Cat((f0 (Et)) Ex= ) The sequence of distributions (px ) approximating the marginals of the forward process initialized at Po defines path that bridges between p.. and the prior Po such that the discrepancy between P: and P++1 is small. SMC samplers based on this path are robust to multi-modality and offer an interesting alternative to the geometric and tempering paths traditionally used in the SMC literature, see Dai et al. (2022). Our proposals of (xx.) P=(x:) (T.ly) inherit the behavior of (Pe)ren and bridge the initial distribution dy and posterior dy Indeed, y is noiseless observation of Xo ~ Po. we may consider az y + (1 a,) with e, ~N(a,, Ia,)). as a noisy observation of X Ps and thus, of is the associated posterior. We illustrate this intuition by considering the following Gaussian mixture (GM) example. We assume that Po(xo) = ΣM wi Na Mi, Ia.) where M > and {wx) 1 are drawn uniformly on the simplex. The marginals of the forward process are available in closed form and are given by
5

---

Published as a conference paper at ICLR 2024
450
t
100
70
50
t
20
Display of samples from (24) x (o for the GM prior. Samples from of (yellow), those from the prior (purple) and those from the posterior @ (light blue) with n 500.
Figure 1:
Ia,), which shows that the discrepancy between Pa and P-+1 is small as long as a, 1/22 - is close to O. The posteriors {o] Jeeee:o [0:]]] are also available in closed form and displayed in Figure I, which illustrates that our choice of potentials ensures that the discrepancy between consecutive posteriors is small. The idea of using the forward diffused observation to guide the observed part of the state, as we do here through 7. (Tely), has been exploited in prior works but in different way. For instance, in Song al. (2021c; 2022) the observed part of the state is directly replaced by the forward noisy observation and, as it has been noted Trippe al. (2023), this introduces irreducible bias. Instead, MCGdi f weights the backward process by the density of the forward one conditioned on y, resulting in natural and consistent algorithm.
Pr(2)) - Ei= -
We now establish the convergence of MCGdi ff with a general sequence of potentials {ox 1- We consider the following assumption the sequence of potentials {o,)²-
/ 9 (= 1 < co for all t E [1 :1:n-1] 1].
(A1) sup ( < and sup rER*=
nERd>
The following exponential deviation inequality is standard and is direct application of (Douc et al., 2014, Theorem 10.17). In particular, it implies a O(1/V bound on the mean squared error 1| (h) - (h)1l2
Proposition 2.1. Assume (Al). There exist constants CL. 22n E os) such that, for all N E NN 0 and bounded function h Rd + R. PP | (h) (h) 2 E| < C1,n exp(-o2,n // where 1 := supaeRd [x())
We also furnish our estimator with an explicit non-asymptotic bound on its bias. Define T = E E[ where of = -N-1 E 6gg is the particle approximation produced by Algorithm and the expectation is with respect to the law of (EJiN II:N). Define for all t [1 n], 07 (x1) pa(e)) (zo (1 dd), where (oo(2))) (1I:-)02 Ps (x (es+))++)) dr
Proposition 2.2. It holds that
KL ( S CN - 1) N
(2.7)
where Dim. O. Cy.,. := Em= S all t E [1 n) and Zu := (dFo) (z (2o) fffurthermore (AI) holds then both CY... and DY... are finite.
2/20 { (d (20 2d) } (dzz) and Z, := == [ (z.)p( for of (.))
The proof of Proposition 2.2 is postponed to Appendix B.1. (A1) is an assumption on the equivalent of the weights { with a general sequence of potentials [9] ) )= and is not restrictive
can be satisfied by setting for example gf((x,) = 7.jo (T, + 8 where 5 > 0. The resulting algorithm is then only a slight modification of the one described above, see Appendix B. for more details. is also worth noting that Proposition 2.2 combined with Pinsker's inequality implies that the bias of MCGdiff ff goes to 0 with the number of particle samples N for fixed n. We have chosen to present a bound et al. (2018 8); Huggins& & Roy (2019), as it allows an explicit dependence on the modeling choice { > see Lemma B.2. Finally, unlike the theoretical guarantees established for SMCdi Ef in Trippe et al. (2023), proving
Kullback-Leibler (KL) divergence, inspired by Andrieu

---

Published as a conference paper at ICLR 2024
the asymptotic exactmess of our methodology w.r.t. to the generative model posterior does not require having (2+1(2s1)),, = (e for all s E [0:n n 1], which does not hold in practice.
Noisy case. We consider the case oy > 0. The posterior density is given by (z)) (T))) where (zo) == (yy To. of In what follows, assume that there exists We denote ÿ- = ot 1/2 y. We can then write that
[1 n] such that o² (1 ãi //a)
(z)z) a. (irra.//ao (1- ā,) Ia,)= a./2
1- - (To)
(2.8)
which hints that the likelihood function de is closely related to the forward process equation 1.1. We may then write the posterior o (zo) as (x)) x 7r (To) (xo) x I ōg. (1=10 (zo))d Next, assume that the forward process equation 1.1 is the reverse of the backward one equation 1.3, i.c. that
( ( = (2)
Vt E [0 n
(2.9)
This is similar to the assumption made in SMCd1 ff Trippe et al. (2023). Then, is easily seen that it implies po(20)9:00(0=)0 (x=//=))===-)))0)))))) (x))))) (xo( and thus
(00))) = - Po| (~oo )6, (dr, )pr (z+))d,, / f 6g. (pr,) (
- =
Pol- (2o Poj- 20 (dæ,), (2.10)
where oFr (x,) x Pr (5, 2, equation 2.10 highlights that solving the inverse problem equation 2.1 with 0y > 0 is equivalent to solving an inverse problem on the intermediate state X, ~ P, with noiseless observation Vr of the dy top coordinates and then propagating the resulting posterior back to time 0 with the backward kernel Poj.- The assumption equation 2.9 does not always holds in realistic settings. Thenefsr, while equation 2.10 also holds only approximately in practice, we can still use it as inspiration for designing potentials when the assumption is not valid. Consider then {g Jo and sequence of probability measures {0/)n Ji= defined for all 1/22 y, (1 - (1 x) )a,)) 2 0. In the case of œ = 0, we have g (zz) = Tejo (+)) for t E [r+11 : n] and 6% = - The recursion equation 2.5 (2oo)=+)))))) (((()))'PP))(( +0+)==))))=))))))))))))))))))))) (dx+), which resembles the recursion equation 2010. In practice we take x to small in order to mimick the Dirae delta mass at 7 in equation 2.. Having a particle approximation ob = N E deg of Φy by adapting Algorithm 1, we estimate Øf with oi " where E (-KEt) and wi x g (((")) ). In the next section we extend this methodology to general linear Gaussian observation models. Finally, equation 2.10 allows us to extend SMCdiff ff to handle noisy inverse problems in principled manner which is detailed in Appendix A.
I E [r: : n) as oi (xx) x ge (are), where of (xt)) = NM
holds for t E [T n] and assuming >> O, we find that (5a)) x (0())) 9 Pooj
2.1 EXTENSION TO GENERAL LINEAR INVERSE PROBLEMS
Consider Y = AX + TyE where A E Rd,xd=, E N (Od,, Ia,)) and o, 2 0 and the singular value decomposition (SVD) A = USVT, where V E Rdy, U E Rd,xdy are two orthonormal matrices, and S E RR xdy is diagonal. For simplicity, it is assumed that the singular values satisfy 81 > ... > sd, > 0. Set b = d,- - dy. Let V E Rd xb be an orthonormal matrix of which the columns complete those of V into an orthonormal basis of Rd, i.e. V'y = Ib and V = Oz.dy- We define V = [ E Rd= In what follows, for a given x E Rdd we write x E Id for its top dy coordinates and x E IRbb for the remaining coordinates. Setting X == V x and Y = UY and multiplying the measurement equation by S-'UT yields
Y X + +o, E~ (~,(0,,T,))
In this section, we focus on solving this linear inverse problem in the orthonormal basis defined by V using the methodology developed in the previous sections. This prompts us to define the diffusion based generative model this basis. As V is an orthonormal matrix, the law of Xo = VT Xo is (x(o)) := Po(Vx)). By definition of Po
7

---

Published as a conference paper at ICLR 2024
and the fact that ||xx|2 = ||x||2 for all x E IR we have that
(xo) - Po V (r)) [II--] Pa (drs|(=at)) Pm (n(d)))= - S (xooxx) (II--] > (cxx/ 1)} pa (ddx))
where for all E [1: n]. - xx) == (xx (x,)),,
where m, == V m. (Vx,). The tran-
define a diffusion based model in the basis V. We write (x,) for the first dy coordinates of m, (x,) and m. (x,) the last b coordinates and denote by ps the time s marginal of the backward process.
sition kernels (A. 12-1 {²_)==)
Noiscless. In this case the target posterior is o% (xo) oc Po(y"Xn)- The extension of algorithm is straight
forward; it is enough to replace y with y (=S-!UT") y) and the backward kernels {p, (pr 1- with {A }=
Noisy. The posterior density is then oy (xo) x gó (xo (o))P((x)). where gă (xo) = -1.--1 x.
As in Section 2, assume that there exists {T, T== C [1: n] such that ā, = (1 a., )s² and define for all i E [1 dy]. y; := a., 1/2 [i]. Then we can write the potential øe in similar fashion to equation 2.8 as the product of forward processes from time O to each time step T;, i.e. 5ō (xo) = I a (N a7/2 xx x (1 - ax,)). Writ- ing the potential this way allows us to generalize equation 2.10 as follows. Denote for E [1 dz). x\ RRd--1 the vector x with its l-th coordinate removed. Define
dy
2n (dx+ x {IT A (x,, (X44. )oy, (dx,, [xl))x
) Pray (X Dōga (dxxa, (dy [xx
which corresponds to the posterior of a noiseless inverse problem on the joint states X ~ Prien with noiseless observations Yr, of Xx, [i] for all i E dy].
Proposition 2.3, Assume that Pa+1 (x (x, holds that o% (xx) x xJ >0 (Xu(x,, Joy (d
(x,1xx41) (x++1) = P.
- 1]. Then it for all s E [ n-
The proof of Proposition 2.3 is given in Appendix B.2. We have shown that sampling from oy is equiv- alent to sampling from ox.n then propagating the final state X,, to time 0 according to > There- fore, in equation 2.8, we define {g")a )m and (o) for all E [TI n] by of (xx) of (xx) and ge (x²) := II N (xx; 1/2 y., (1)1(1)),aa,, k)ā,/a,)) K 0. We obtain a particle approximation of dy using particle filter with proposal kernel and weight function X (x, xx/xx++) W,(x = oy (x+ )r, (dx: ( which are both available in closed form.
(xx
3 NUMERICS
A prerequisite for quantitative evaluation in ill- ill-posed inverse problems in Bayesian setting is to have access to samples of the posterior distribution. This generally requires having at least an unnormalized proxy of the posterior density, so that one run MCMC samplers such the No U-turn sampler (NUTS) Hoffman & Gelman (2011) 1). Therefore, this section focus on mixture models of two types of basis distribution, the Gaussian and the Funnel distributions. We then present brief illustration of MCGdi E E image data. However, in this setting, the actual posterior distribution is unknown and the main goal is to explore the potentially multimodal posterior distribution, which makes a comparison with a "real image" meaningless. Therefore, metrics such as Fréchet Inception Distance (FID) and LPIPS score, which require comparison to a ground truth, are not useful for evaluating Bayesian reconstruction methods in such settings.
Mixture Models. We refer to the Funnel mixture prior as FM prior (see Appendix B.3). For GM prior, we consider mixture of 25 components with known means and variances. For FM prior, we consider a mixture of 20 components consisting of rotated and translated funnel distributions. For given pair (dz, dy), we sample prior distribution by randomly sampling the weights of the mixture and for the FM case the translation and rotation of each component. We then randomly sample measurement models (g, A,)y) Bdy x Rd xd, x [0, 1]. For each pair of prior distribution and measurement model, we generate 10 samples from MCGdi ff, DPS, DDRM, RNVP. and from the posterior either analytically (GM) or using NUTS (FM). We calculate for each algorithm the sliced Wasserstein (SW) distance between the resulting samples and the posterior samples. Table I shows the CLT
'The code for the experiments is available at ht attps: //github. com/ www//abbreeeeeeeeeeooooeeeeaeeee gawriaavae/naaaaeeoeeea diff.
8

---

Published as a conference paper at ICLR 2024
MCGdiff
DDRM
DPS
RNVP
MCGdiff
DDRM
DPS
RNVP
PCA
PCA1
Figure 2: The first and last four columns correspond respectively to GM with (dz, dy) = (800, 1) and FM with (dz, dy) (10, 1). The blue and red dots represent respectively samples from the exact posterior and those generated by each of the algorithms used (names on top).
BUGOLEE
DUKM
DPS
KNI22
d
dg
MOGGLE1
DDRN
DPS
RNV 4
4.20
5.43
80
0.67 = 0.24
7.07 1.35
5.10 1.23
7.79 +
1.50
5
0.73 0.33
0.67
3.47 0.78
2.07
800
2.40 # 1.00
7.44
1.15
6.49 + 1.16
7.74
1.34
10
2.45
0.42
3.82 + 0.64
4.30
6.04 +
1.12
4.94
800
0.47 a 0.19
8.39 1.48
5.51
1.18
7.81
1.63
10
0.71 0.12
2.32 0.74
3.74 0.77
5.11
Table 1: Sliced Wasserstein for the GM (left) and FM (right) case.
95% confidence intervals obtained over 20 seeds. Figure illustrates the samples for the different algorithms for a given seed. We see that MCGdiff outperforms all the other algorithms in each setting tested. The complete details of the numerical experiments are available in Appendix B.3 well as additional visualisations.
Image datasets. Figure shows samples of MCGdiff in different datasets (Celeb, Churches, Bedroom and Flowers) for different inverse problems, namely Inpaiting (Inp), super resolution (SR). Gaussian 2D deblur (G2Deb) and Colorization (Col). Visual comparison with competing algorithms and different datasets are shown Appendix B.3 as well as numerical details concerning Figure 3.
CONCLUSION
this paper, we present MCGdiff a novel method for solving Bayesian linear Gaussian inverse problems with SGM priors. We show that MCGdiff ff is theoretically grounded and provided numerical experiments that reflect the adequacy of MCGdiff in Bayesian framework, as opposed to recent works. This difference is of the uttermost importance when the relevance of the generated samples is hard to verify, in safety critical applications. MCGdi ff is first step towards robust approaches for addressing the challenges of Bayesian linear inverse problems with SGM priors.
Y
MCGdiff
MCGdi
MCGdiEf
MCGdiff
Y
SR Bedroom
By Caleb
dy Flowers
ay = 0.1 Church
Figure 3: Illustration of the samples of MCGdiff ff for different datasets and different inverse problems.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Christophe Andrieu, Anthony Lee, and Matti Vihola. Uniform ergodicity of the iterated conditional SMC and geometric ergodicity of particle Gibbs samplers. Bernoulli, 24(2):842 872, 2018. doi: 10.3150/15-BEJ75) URL https:
0 311001155
Siavash Arjomand Bigdeli, Matthias Zwicker, Paolo Favaro, and Meiguang Jin. Deep mean-shift priors for image restoration. Advances Neural Information Processing Systems, 30, 2017.
Joe Benton, Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. From denoising diffusions to denoising markov models. arXiv preprint arXiv:2211.03595. 2022.
Julian Besag, Jeremy York, and Annie Mollié. Bayesian image restoration, with two applications in spatial statistics. Annals of the institute of statistical mathematics, 43:1-20, 1991.
Olivier Cappé, Eric Moulines, and Tobias Ryden. Inference in Hidden Markov Models (Springer Series in Statistics). Springer- Veringe--Verlag, Berlin, Heidelberg, 2005. ISBN 0387402640.
Margaret Cheney and Brett Borden. Fundamentals of radar imaging. SIAM, 2009.
Nicolas Chopin, Omiros Papaspiliopoulos, et al. An introduction to sequential Monte Carlo, volume 4. Springer, 2020.
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2023. URL https:://////ct /ttpg:ss///peereviee.. net iaaerrummi-OODSS GAGTOK.
Chenguang Dai, Jeremy Heng, Pierre E Jacob, and Nick Whiteley, An invitation to sequential monte carlo samplers. Journal of the American Statistical Association, 1177639) 2022.
Masoumeh Dashti and Andrew M Stuart. The bayesian approach to inverse problems. In Handbook of uncertainty quantification, pp. 311 311-428. Springer, 2017.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:: /pw.revviww...ee/////pp net./ forum? ddHHkpbnH
Randal Douc, Eric Moulines, and David Stoffer. Nonlinear time series: Theory, methods and applications with examples. CRC press, 2014.
Arnau Doucet, Nando De Freitas, Neil James Gordon, et al. Sequential Monte Carlo methods in practice, volume 1. Springer, 2001.
Idris A Elbakri and Jeffrey A Fessler. Statistical image reconstruction for polyenergetic x-ray computed tomogra- phy. IEEE transactions on medical imaging, 21(2):89-99) 2002.
Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T Roweis, and William T Freeman. Removing camera shake from single photograph. In Acm Siggraph 2006 Papers, pp. 787-794. 2006.
Mário AT Figueiredo, José M Bioucas-Dias, and Robert D Nowak. wavelet-based image restoration. IEEE Transactions Image processing, 16(12):2980-9991, 2007.
algorithms for
Rafael Corsino González, Richard F. Woods, and Barry R. Masters. Digital image processing, third edition. Journal of Biomedical Optics, 14:029901, 2009,
Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye. A review generative adversarial networks: Algorithms, theory, and applications. IEEE transactions on knowledge and data engineering, 2021.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6833 6851, 2020.
Matthew Hoffman and Andrew Gelman. The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo. Journal of Machine Learning Research, 15, 11 2011.
Jonathan H. Huggins and Daniel M. Roy. Sequential Monte Carlo as approximate sampling: bounds, adaptive resampling via co-ESS, and an application to particle Gibbs. Bernoulli, 25(1):584 622, 2019. doi: 10.3150/ 17-BEJ999. URL https::tp/........t.. /ooi org/ 0 31 10/11-BBEJ999
Jérôme Idier. Bayesian approach to inverse problems. John Wiley & Sons, 2013.
Oleg Ivanov, Michael Figurnov, and Dmitry Vetrov. Variational autoencoder with arbitrary conditioning, arXiv preprint arXiv: .23800222222 2018.
Jari Kaipio, Ville Kolehmainen, Erkki Somersalo, and Marko Vauhkonen. Statistical inversion and monte carlo sampling methods in electrical impedance tomography. Inverse problems, 16(5):1487, 2000.
Sebastian Kaltenbach, Paris Perdikaris, and Phaedon-Stelios Koutsourelakis. Semi-supervised invertible neural operators for bayesian inverse problems. Computational Mechanics, pp. 1 1-20, 2023.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: /ttp:///axxvvv/a wrr.iiioooro/oooooooo.oo hasp//ab///a/ 111211111120 6980.
Diederik P Kingma, Max Welling, al. An introduction to variational autoencoders. Foundations and Trends@ in Machine Learning, 1244)3307-392, 2019.
Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43(1 1):3964-3979, 2020.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
Yosra Marnissi, Yuling Zheng, Emilie Chouzenoux, and -hrriihitte Pesquet. A variational bayesian approach for image appporationnaallicaaiion to image deblurring with poisson-gaussian noise, IEEE Transactions on Computational Imaging. 3(4):722-737 -3722 2017.
Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li. Generating diverse strueture for image inpainting with hierarchical vae. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10775-10784, 2021.
Michael K Pitt and Neil Shephard. Filtering via simulation: Auxiliary particle filters. Journal of the American statistical association, 94(446):500-599, 1999.
Teemu Sahlström and Tanja Tarvainen. Utilizing variational autoencoders in the bayesian inverse problem of photoacoustic tomography. SIAM Journal on Imaging Sciences, (6(1):89-110. 2023.
Hyomin Shin and Minscok Choi. Physics-informed variational inference for uncertainty quantification of stochastic differential equations. Journal of Computational Physics, pp. 112183, 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.

---

Published as a conference paper at ICLR 2024
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a. URL StIgiarCHLP.
net/ forum? ?id=
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415-1445 1428, 2021b.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021c. URL https: www./evieee.eeeeeeeeee/ee fortt//ormm?ee wit.mmdddddpdddiee.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. International Conference on Learning Representations, 2022. URL
https: http:////ppeeeveeeeeeeeeee
Andrew M Stuart. Inverse problems: a bayesian perspective. Acta numerica, 19:451-559, 2010.
Jingwen Su, Boyan Xu, and Hujun Yin. A survey of deep learning approaches to image restoration. Neurocomput- ing, 487:46-63 65, 2022.
Brian L. Trippe. Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S. Jaakkola. Diffusion probabilistic modeling of protein backbones in for the motif-scaffolding problem. In The Eleventh International Conference on Learning Representations, 2023. URL net /
Marinus T Vlaardingerbroek and Jacques A Boer. Magnetic resonance imaging: theory and practice. Springer Science Business Media, 2013.
Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. High-fidelity pluralistic image completion with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4692-4701, 2021.
Xinyi Wei, Hans van Gorp. Lizeth Gonzalez- Gonzalez-Carabarin, Daniel Freedman, Yonina C Eldar, and Ruud JG van Sloun. Deep unfolding with normalizing flow priors for inverse problems. IEEE Transactions Signal Processing, 70:2962-2971, 2022.
Luhuan Wu, Brian L. Trippe, Christian A. Naesseth, David M. Blei, and John P. Cunningham. Practical and asymptotically exact conditional sampling in diffusion models, 2023.
Hanyu Xiang, Qin Zou, Muhammad Ali Nawaz, Xianfeng Huang, Fan Zhang, and Hongkai Yu. Deep learning for image inpainting: A survey. Pattern Recognition, 1390:09,46, 2023.
Raymond A Yeh, Teck Yian Lim, Chen Chen, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do. Image restoration with deep generative models. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6772-6776. IEEE, 2018
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas Huang- Generative image inpainting with contextual attention. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5505-5514, 2018
Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Aggregated contextual transformations for high-resolution image inpainting. IEEE Transactions on Visualization and Computer Graphics, 2022.
Guanhua Zhang, Jiabao Ji, Yang Zhang, Mo Yu, Tommi Jaakkola, and Shiyu Chang. Towards coherent image inpainting using denoising diffusion implicit models. arXiv preprint arXiv:2304.03322, 2023.
12

---

Published as a conference paper at ICLR 2024
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1138-1447, 2019.
Xu Zhihang, Xia Yingzhi, and Liao Qifeng. A domain-decomposed vae method for bayesian inverse problems. arXiv preprint 20t.:2300,0570,, 2023.
13

---

Published as a conference paper at ICLR 2024
A SMCDIFF EXTENSION
The identity equation 2. 10 allows us to extend SMOdiff Trippe et al. (2023) to handle noisy inverse problems as we now show. We have that
ou (z,) = (P [1-=-++ (ddx,//=++1) (d(x),
P (ÚF d2,
= f 10- (.rmmITT 1 (dd
where
P- PT Iz 1-120+1)
[11='' P. (F P, (Zn)
bron Er [r+1m)
Dr
(T++Im)
0ū (T, f (ū 2, )dzz
f (Ir+im))
and
Gr Tin (T
pr (F 2- 2++1)
II P. 2 katt))) P, (dzz) s=++1
Next, equation 2.9 implies that
/ Poti(Tot1 (aa1(ao112e11l 2ot1le, T.41 2o+1 (T. 22411 12+1)d201+1=
I Palats 2 (1st1 (Z.
and applied repeatedly, we find that
(T++1m) / p- P-(07 ,,,,,, / 6 (dT,)) II (T.
72
and thus, fir (T++1m)) = 8g. (dz,) II V. (T. In order to approximate of- we first diffuse the noised observation up to time n, resulting -ims and then estimate b. (- (7, using particle filter with p. (x,læ,+1) transition kernel at step 1T + 1 ni] and 9, 2. P.- (I,-1 x 2x)) as potential, similarly
to SMCdiff.
B PROOFS
B.1 PROOF OF PROPOSITION 2.2
PRELIMINARY DEFINITIONS.
We preface the proof with notations and definitions of a few quantities that will be used throughout.
For a probability measure ş and f bounded measurable function, write (uf) = f(z)u(dx) the expectation of f under μ and if K (dæ]=) is transition kernel we write (=))==) = fxx [(x)
Define the smoothing distribution
6y (dron) x 8, (dTo) Poin )
(B.1)
14

---

Published as a conference paper at ICLR 2024
which admits the posterior o time 0 marginal. Its partiele estimate known the poor man smoother is given by
N (drom) = N
Σ
su (dzo) II
1 [kk = 1 Joet (dx,))
(B.2)
No.. ElI:N]++1
s=1
We also let a be the probability measure defined for any B E ((Rd=))nt) by
(B) E B[ (B)],
Onn
where the expectation is with respect to the probability measure
O.n (d(=0m I:N
():N))
II
PM (d
II
II
we w (ddd)))" 1x
w (da (dee
l 1 (dxz)) (B.3)
where w/ = 33 (S+11) Σÿ= (E/+1) and which corresponds to the joint of the random variables
generated by Algorithm 1. It then follows by definition that for any C' E B(R*)).
[ Lu..
IE / de (dzaa) = 1 o0 (C) = 2000 (C).
Define also the law of the conditional particle cloud
N
I:N I:N N (d(ad:n 0l.n
) (2:n) = 6zn (dx,))
II
p% (dx
N II
N
II 2=2
)6N (da
(dx wi (00
(B.4)
k=1
N-1 N
x 8zn (dzy) No (da
I
Σ
wat (da) )2 (dzd (e
of )5,
j=1
what follows Ezg... refers to expectation with respect to PN (-)2o.n) Finally, for [0 n we let 22 denote the sum of the filtering weights at step i.e. S:W and for all & E [1 n]]. Z, = f (elo(x)l)pp((dz))
= E 2s (f:+1) We also write Zn f po 6 (dFo)
The proof of Proposition 2.2 relies on two Lemmata stated below and proved in Appendix B.1; in Lemma B.1 we provide an expression for the Radon-Nin Radon-Nikodym derivative /d and in Lemma B.2 we explicit its leading term.
Lemma B.I. obn and &y are equivalent and we have that
N Zo/ Zm
D (dzzza) ) EE Ezz...
oy (dzon)
(B.5)
I ΩN
15

---

Published as a conference paper at ICLR 2024
Lemma B.2. It holds that
Zn
p-1
=(Nx')
(N-1)
Bizo.
II
N
N
Zu
1==0
+ (N - 1)m11
2 Nn 7.10 (2,l()))))
/ (00
D Don N2 (B.6)
where D... is positive constant.
Before proceeding with the proof of Proposition 2.2, us note that having + (3)))) bounded on Rd for all E [0 n 1] is sufficient to guarantee that Cy.x and D are finite since in this case it follows immediately that Zo....
π ==ر N '2 is bounded and so is the right hand side of equation B.6. This can be achieved with a slight
modification of equation 2.5 and equation ??. Indeed, consider instead the following recursion for E [0 n]
where 8 0,
2% (=n) (ZnlV) p)nn(
7. (F.ly) + dr. dr,t1
(x,))) x ( (d.
2o+1 (I,+1ly) 1yy) +
Then have that
( ob(zn) x f (()r)), (2ol21))
Po (o(()=1) 71 T1 (y) + dæ1
We can then use Algorithm to produce particle approximation of og using the following transition and weight function,
20 x++ 1) 7/ (1)((++))) 1) + 8 (I:s (1++))) + 70(1) + 8 (2ra//00))))
g (xx (osl
0, (x (70(1) + 6)/ (T++10(7.+ (y) +6)
where Yo (v] = J [T (u)) (x+. dd, is available in closed form and pK is defined in equation 2.3. and it is still possible sample from px- since it is simply
Wj is thus clearly bounded for all E [0 mixture between the transition equation 2.3 and the "prior" transition.
Proof of Proposition 2.2. Consider the forward Markov kernel
Pi:n (d21 (Po (P z1)
B Im (zo, dāim) J Pi.m (dž Po (2) (žolz)) =11
(B.7)
which satisfies
Hy (d=am) (dzo) B (20 dzim)
By Lemma B.1 we have for any C E B(Id==) that
26 (C) = / D LC
/ 1o(2))
N 20/ Zn
&y (dzo=))
ΩN
-/10(20)
11((00) /
B 120. d ) Bogn
N" Zn
6 (d2u))
11--1 ΩN
16

---

Published as a conference paper at ICLR 2024
which shows that the Radon-Nikodyee derivative d /d is,
d
/
B (2m dz Exonn
N Zo Zn
(zo)
d
IT==0 Ω
Applying Jensen's 's inequality twice yields
y de
N Zo/ Zn B Imm (20, d2
(zo)
I1 S2
and it then follows that
/
Zn
n-1
II N- 12 ) (d(()))
KL
log
B B1sn (=0 d
Zo
Finally, using Lemma B.2 and the fact that log(1 + xx) for we get
D
KL(08 (
N2
where
/
2o
Σ
(0))) (dzz),
and ())(2)) x pa(2x) f Paja (2.lV)
PROOF OF LEMMA B.1 AND LEMMA B.2
Proof of Lemma B.I. We have that
&... (dz0:a)
= =N-1 /
P.N. (da
I:N da
I:N
Σ
8 gyztto (dzo) II 1 (
a (dzs)
knon el1:N]e+t
= N-1
y (deo) II
kss = a
No- - (d2,)
Roon aim
pl (dar))
(de (
II Wo D" (debl= (
1=1
E=2==1
= -
pd (da 1o_e. en (dzu) II pl (dxx))
II (ds if
Noo J.N
2=2
W (zg
Ap-1
x 1 [(a
ke- kir)
1o- (dx
Ne-
ae
)8
e (dza-))
26-1
II rs
wi 2xx (d=ā)= (dx()))
{ato 1 = -ki]
2o (6(21
(d2o
Ko a.
1 15
2 (dzo).
22
17

---

Published as a conference paper at ICLR 2024
Then, using that for all E [2 : m²]
-.-1
1 (=> 1 (daf-1' (
Ps -d'
[ = (4)
we recursively get that
n pl (dx," ( )6 zn (dzn)
II I[a 1 = k.)
Ds p (dx 1 - )8 k, (dz--1)
2
_-_1
22
1 aa ko =/1
Pol
(dIb ko
)6 8 =
(dzo)
Injo (m
4s 1 (2x-1 (y)
83 (dz
II
1 1 -
I2. 12-111 7.10 (2ely) Po- Pe-1(des-1lo (dx,--')
Zn
2=
Po 21)
x 1faa = =k
Po (džol 1o (d (dr
26 11 (Z lvv)
Zu 12 Pom (dzoen )o,, (dx )
II
1 a hi,) D.
8z, (dr
Zn
==1
Thus, we obtain
N S Po. (dzom.) N /
ΣΣ Hy (dz0mn)
Zo/Zn
82, (de
II jkkn
p% (dot)
IT:-0 SN
Koon 1im
12
1 1s II
ke-1 ke jõzz (dx
II kgflke
W w
f=
1 (a ki) 880 (dxt
II ko
wo ai
(1g oo, (dzb))
Po
Zo/ Zn II: 1 ΩN
NNN Σ Ly (d2om)
) Fpem
kain
where for all ko.n E [1 M Eko... denotes the expectation under the Markov kernel
Roin
(d 1:N 1:N
(000 1m 20:)) 6z, (dx
II ke
p% (dx
N Σ
b (dx he jōkn (da
ke-1
o dz (dx) 12
x
2=2
k=1
x 2zn (d jōk, (d)
II
Σ
w du (da² )e" (dze بلا (dzo))
joko k=1
Note however that for all (Kom, lom) E ([1 : N2
B8 koo 20:n
1==0 Ω
Ω
18

---

Published as a conference paper at ICLR 2024
and thus it follows that
N 2n/2n
Do.n (dzom) Ezonn
(B.8)
11 SN
Denote by (Fs)==o the filtration generated by conditional particle cloud sampled from the kernel p.V equa- tion B.4, i.e. for all E (0:n 1::n-1
F. o (Etin) -
and F, = (SlliN). Define for all bounded f and & E [0: - 1]
n-
N
II
N N
Σ
((
(B.9)
with the convention (E) life Define also the transition Kernel
Q4 1|e41 - B( e (#e+1, A) + / (12
(B.10)
Using eqs. (2.3) and (2.4), it is easily seen that for all é E [0: n0 1],
(Te ()) / (ts
e+ 3x( Q4 (f)(
(B.11)
Define 1 : E IRd + 1. We may thus write that y (f) = N
18+ I. (1) (I) ((C
Lemma B.3. For all l [0
1] holds that
Ezon [y 1mm (f)
N -
Ezo.
(Tn (1
1 Ezp... [TEN (z))()())
N
N
Proof. By the tower property and the fact that (f) is - measurable, we have that
E Ez
N (7-1m 1m (f)] = EF Eoz...
N N
N 1m
(1) zo....
N a (6h))))))) Fet1
Note that for all E [0 : n - 11. (57
,, -1--1) are identically distributed conditionally on Fet1 and
N Σ (rltt)) / / R=1
E Exoo
(G)
W (E))((6)
Fet2
leading to
E Egg...
N-1 Σ
We
(Et) (Et)) Fet1 Fet1
k=1
N N
Σ alletr)) / ( + N (())(())((
1
NΩN
and the desired recursion follows.
19

---

Published as a conference paper at ICLR 2024
Proof of Lemma B.2. We proceed by induction and show for all & E [0: -
E Exa.
[TE. (f))
N n-t
N
2n
(N 1)m-(-1
(B.12)
Na-E
2z/Zn, Zn
/
Dy
(T (1 (0re
kin
N2
== E+2
where f a bounded function and D is positive constant. The desired result in Lemma B.2 then follows by taking f = 0 and f 1.
Assume that equation B.12 holds at step f. show that holds at step 1 we use Lemma B.3 and we compute
E
Ezo..
Vin
N
(Q2 (2x=114+2()) and Exx... [TE, (1)) (z))
Using the following identities which follow from equation B.11
/
= (aa
and
= (e0 (dzz)))))
=
we get by equation B. 12 that
N Ezoin (TE ((-11e+((1))
N - 1) n-2+1 (
N
Zn
(N (N-1) 1-)=-
2
/ benc
Na-
[7e+110 (2)+1l))
Z./Zn
/
DM
(B.13)
( (drelzs.)
2010 (Exl8)
N2
#=
N n1++1 gep (Tel y (xt)
N
Zn
(N 1)m-
/ (a0 (dzol))) +
D
Σ
+
Nn-E+1
I.jo (2.ly)
N
1=
20

---

Published as a conference paper at ICLR 2024
The induction step is finished by using again equation B.12 and noting that
(N 1)=-
Dy
Bza... (T:m (Tmm(1) We (=t)
Na-e-1 (Zz/ 1 +
An
N
N2
and then setting D 1:m = D + DV Emn'
It remains compute the initial value at f= 2. Note that
-
/ (dr ( +
1 N
E Sp:m (n I:n (f)
- (2n)
(B.14)
N
and thus by Lemma B.3 and similarly to the previous computations
Exo.. (0-2:x(())
N 1 )
f PN (dxx (I (In)Q" (f) +
N -
Wn (2n (f)
N2
DV
+ (0n-2(2n1))
pi (dx 1/r (znn)
+
2fam
N2
S 4n 10n-10(2n-11)00, 2 Na (ddEn-))
N
N
Z,
N 1-
N
/
D 2:m
(Tn, -1)
Pn-1 (dxn (dxn- (zn)
(njo(Enly))
N2
B.2 PROOF OF PROPOSITION 2.3 AND LEMMA B.4
In this section and only in this section we make the following assumption
(A2) For all s E [0 1]. Pa (x, )qs+ (Ce41le) (xx) = p.+ (Xs+ ) (It (r++))
We also consider as 0. In what follows we let Tay+1 and write Tidd (I,,.., Tay } and Fl.dy [1 n] Tia Define the measure
dy (, x II ^r (x,, (xx+1)xxx (y (B.15)
Ty (dxp:))= = pn. (dx..)
II sETiddy
Under (A2) it has the following alternative forward expression,
dy
I. (dxp..) = (o(d))
II SETIdd
9s+1 (c+1(dx;110 (x.
II (x,, |x,, 1xxx (dx,, [i]).
(B.16)
==1
Since the forward kernels decompose over the dimensions of the states, i.e.
dz
4s+1 (x+1Xxx11(xx) =
II
di (x.+1
((]
(=1
21

---

Published as a conference paper at ICLR 2024
where (x=+((111])) (Xx+++114] (ast1/aa) x,1,, (ds+1/azs)), we write
d,
To. (Xo:) = (x()
1:m0
(B.17)
where for L E [1 :yy
r [x,x]]]]..,,,,(]x)x]]1 - (x(])xx= (() II
[0))
(B.18)
ofTe
and for l e (dy + dzl.
1-
[x (x = II (x+1(x11)))x111)
Xn
(B.19)
5=0
With these quantities in hand we can now prove Proposition 2.3. Proof of Proposition 2.3. Note that for & [1 dy],
- -a+;)= oy = of (x11))))7
II (dxx
affeee
f (d(xx]()....,,..
and thus by equation ?? we have that
dy
Po Px(xo)9ū x po(xo)
II
N (y (iTe)
t=1
dy
= (xoo)
ry 0.e (.o]....,xx((])xxxx) (x(())xxo(()
{=1
d.
(xo) II
rV 0.e (d(x)
E=1
By equation B.16 follows that
/ Iy (xomm )c
(xo)
JTY
Onn (Romn) )dxon
and hence by equation 16 and equation B.15 we get
dy dy-1
o% (xo) x / Pray (Xray 1oy (dy) (dx [dy])dx ddy Toy
Aalmon (Xer, (X 19 (cxx, 12)) Aapm (xco
This completes the proof.
Let nh. denote the joint time and marginal of the measure equation B. i.e.
(Xo, x,) f (Xo:)) -dxx411
(B.20)
22

---

Published as a conference paper at ICLR 2024
We now prove the following result.
Lemma B.4. Assume (A2) and let To = 0, Tay+1 = n. For all &s [1 dy].
(i) If E Thh + 1: T6t1].
70.4 (Xy, xx)
dy II
f nd..t (350 xx+1) 150+1 10 (0.1
a 12j0+1,0 (x, 10)
(=k+1
(ii) i Tk.
7x0, x0) = 70,041 (x0, x0+1)0 10 (EglEs+1:20) Iul
110+1.0
k-1
dy
o (+.l))
II
o.l
1 (x, (x,[([xxs+114) xo (()
1mk+1
Proof of Lemma B.4. Let l E [1 dy] and assume that E [Te + 1 Th+1 2]. By (A2), equation B.16,
equation B.18 and equation 19 we have that
n. x.) = po(ooo) km))
II oh o
(y
(x.
xo
dy II 1000xx100000/000 (1))))))))
and thus, using the following identity valid for E E [k + 1: dy]
(x. (x. (0)xx (())
2h1 (x. [4] (=11)) of
[())xxxx)+))++++11))1))))))) [C]
- af /ttp://///aaaaaallo/o/////////// [2]
23

---

Published as a conference paper at ICLR 2024
and that 20 (2, ko)g. (Io+1lzts) Lolt110 (Fslo Eu0)2+11 (Es+1 Lo) we get that
nd xs)
/ po(xo)g (1 (d.i+1 (xe)
II i=1
folly (0) (
x II Us
1=++1
dy II
/ nd. (xo, x=+1)2 (2,1 (2 (0 (Tx)
fofo- [x]xxxx=[]x]++11111111111]1]]]]] cxxx
6=++1
If 8 T then
n8. (Xo, xx) (o((x))g, (Zo) II 1: (y [i] [1]xx (xx((]))))
(B.21)
dy *+1 1 [k 1]|xo (k: 1J) II (x11
t=k+2
and similarly to the previous get
so.s (Xy, X²)
dy II L (X](e[xx++11) x000)d)xxx+
f nd. (Xo+ X 12g L :
=
Am 2+
Finally, i = Tk+1 1, then
& nd. (x0, x.) po(xx) (I.)Zo) II
g [x]]xxx)) (x. [i)l
dy
x k+ 1 1//x0 + 11g k+1 inle [y + [k: + 1])
II
k=
and using
' (xolk + 1||xo|k 11) A+1 (y(A + 1||xx|f [k 13)
A. (x [k + 11|x7241 IK: 11, xolk 11)9*t1 R+1 (y + 1||xc|[k (k 1))
24

---

Published as a conference paper at ICLR 2024
find that
To.s (Xo, Xx)
=/ /70 (Xo, X xTa d9 71 (I I Lo (T.)
a.t (xx=1,, |1 xolldd) t=k+1
II
B.3
ALGORITHMIC DETAILS AND NUMERICS
B.3.1 GMM
For given dimension d. we consider Qdata mixture of 25 Gaussian random variables. The Gaussian random variables have mean Miij == (8i, 8j. 8i, 8j) E RRd= for (i,j) E {-2, 0, 1, and unit variance. The mixture (unnormalized) weights wi, are independently drawn according to x² distribution. The paramater of MCGdifE x² = 10-4 We use 20 steps of DDIM for the numerical examples and for all algorithms.
Note that (xx)) = f (o)ooz2 ( As qdata is mixture of Gaussians, {s(xs) is also a mixture of Gaussians with means a//²² Pi, and unitary variances. Therefore, using automatic differentiation libraries, we can calculate V log (x.)). Setting c(z,, 8) = - (/2) (x.) leads to the optimum of equation 1.4.
Score:
Forward process scaling:
linearly decreasing sequence between We chose the sequence of { 1000
B1 = 0.2 and $1000 10-4
Measurement model: For pair of dimensions (dr, dy) the measurement model (v. A, o,) is drawn as fol-
lows:
A: We first draw Á (od,, xd_ xdy) and compute the SVD decomposition of USVT. Then, sample for (i.j) E {-2, -,,0,, 1,,0, St according to uniform in [0, 1]. Finally, we set
A U Diag( (sin, (2.))0-2--112211..))).
oy: We draw By uniformly in the interval [0, max(s1, ,.dy))
y: We then draw
4data and set y Ax, + Tye where ~ N Idy))
Posterior: Once we have drawn both qdata and (y. A, ,,),), the posterior can be exactly calculated using Bayes formula and gives a mixture of Gaussians with mixture components Cig and associated weights Wi.j
C. = N(2 (A / + ms.) E)
G; w A Mi.j. o² Ta, +AAT),
where =: (Ta, 20T2A
Variational Inference: The RNVP entries in the numerical examination obtained by Variational Inference using the RNVP architecture for the normalizing flow from Dinh et al. (2017). Given a normalizing flow fg with & E RR, , E N., the training procedure consists optimizing the ELBO, iee., solving the optimization problem
Not
0. = arg max
Σ
log [Jf((e)) + log n. (fa(e:))
(B.22)
&ERI
where Naf E N. is the minibatch-size, Jf, the Jacobian of fo w.rt dh, and 61:N,, (, I)NN,, All the experiments were performed using a 10 layers RNVP. Equation (B.22) is solved using Adam algorithm Kingma
25

---

Published as a conference paper at ICLR 2024
d₂
d2
d
104
101
KL
107
10a
II
9
10a
1.0
50
100
150
200
50
100
150
200
100
150
200
10"
1.0
100
KL
38
50
100
10"
1.0
10a
KL
000
50
100
150
200
50
1.00
150
200
100
150
200
Iteration
Figure 4: Evolution of KL with the number of iterations for all pairs of (dr, dy) tested in the GMM case.
Ba (2015) with a learning rate of 10 33 and 200 iterations with Nnf = 10. The losses for each pair (dz, dy) is shown fig. 4, where one can see that the majority of the losses have converged.
Choosing DDIM timesteps for given measurement model: Given a number of DD samples Rt we choose the timesteps 1 = t < ta 1000 E [1 1000] as to try to satisfy the two following constraints:
For all : dy] there exists tt such that o, (1 - )
For all : R 1). at. t4ts
1/2 1/2
xs 8 for some
26

---

Published as a conference paper at ICLR 2024
McGdiff
DDRM
DPS
RNVP
122
T1
Figure First two dimensions for the GMM case with dg The rows represent dy 1, respectively. The blue dots represent samples from the exact posterior, while the red dots correspond to samples generated by each of the algorithms used (the names of the algorithms are given at the top of each column).
The first constraint comes naturally from the definition of T₁- Since the poten-
tials have mean ag 1/2 the second condition constrains the intermediate laws remain "close". An algorithm that approximately satisfies both constraints given below.
Algorithm Timesteps choice
Input: Number of DDIM steps R. on 18:)0- 1000
Output: (1,),
Set S, {).
for j + [1:dy] do
Set f, = arg min0o 10ya 1/2 (1 - 122
Add F, S, i & S,.
Set nm = R #S,
and = (a) a 1 nnm-
Set 1, e 1 and i, 1. for E <- [2: 1000 do
if 0e 1/2 aj
1/2 > or E S, then
Set e = t, ie is + and Tg = l.
Set 7n 1000.
Additional numerics: We now proceed to illustrate in Figures 5 to 7 the first 2 components for one of the measurement models for all the different combinations of (dz, dy) combinations used table 1. We also show in fig. 8 the evolution of each observed coordinate in the noise case with dy = 4. We can see that it follows closely the forward path of the diffused observations indicated by the blue line.
27

---

Published as a conference paper at ICLR 2024
McGdiff
DDRM
DPS
RNVP
222
30/2
T1
Figure 6: First two dimensions for the GMM case with d2 = 80. The rows represent dy = 1, 2, 4 respectively. The blue dots represent samples from the exact posterior, while the red dots correspond to samples generated by each of the algorithms used (the names of the algorithms are given at the top of each column).
MCGd ff
DDRM
DPS
RNVP
222
.11
Figure 7: First two dimensions for the GMM case with d,, 800. The rows represent dy 1, 2, 4 respectively. The blue dots represent samples from the exact posterior, while the red dots correspond to samples generated by each of the algorithms used (the names of the algorithms are given at the top of each column).
28

---

Published as a conference paper at ICLR 2024
d
MCGdiff
DDRM
DPS
RNVP
8
1,43 0.55
5.88 1.16
4.86 + 1.01
9.43 0.99
8
2
0.49 # 0.24
5.20 2 1.32
1.99
8.93 + 1.29
8
0.28
2.51 1.29
3.48 = 1.52
6.71
80
1
1.39 + 0.45
5.64 1.10
4.98 + 1.14
6,86 # 0.88
80
2
0.67 + 0.24
7.07 1.35
5.10 1.23
7.79 1.50
80
4
0.28 = 0.14
7.81 = 1.48
4.28 + 1.26
7.95 + 1.61
800
2.40
7.44 1.15
6.49 1.16
7.74 + 1.34
800
2
1.31 # 0.60
8.95 1.12
1.01
8.75 # 1.02
800
0.47 + 0.19
8.39 1.48
5.51 1.18
7.81 1.63
Table Extended GMM sliced wasserstein table.
20
10
10
0-
T.[4]
E [2]
-10
-10
-20
20
10
10
102
10a
10
10a
102
10a
20
20
10
10
3.[3]
E. [4]
-10
-10
-20
-20
100
101
102
103
100
10
102
103
Figure 8: Illustration the partiele cloud of the 4 first observed coordinate in the case (dy. d.) = (4, 800) with 100 DDIM steps. The red points represent the particle cloud, while the purple points at the origin represent the posterior distribution. The blue curve corresponds the curve - / y[(] and the blue dot on the curve to 1/2 [(].
Table 2 an extended version of table 1.
29

---

Published as a conference paper ICLR 2024
dy
dy
dp
10
301
Figure 9: Purple points are samples from the prior and yellow samples from the diffusion with 25 DDIM steps.
d
SW
0.79 0.15
0.87 + 0.07
10
0.96 0.06
Table 3: Sliced Wasserstein between learned diffusion and target prior.
B.3.2 FMM
A funnel distribution is defined by the following density
M( 0, II O,
To generate a Funnel mixture model of components dimension we start by firstly sampling (μi, R2 uniformly in ([-20, 20 SO(R*)) x22 The mixture will consist of 20 Funnel random variables translated by 14 and rotated by R. with unnormalized weights way that are independently drawn uniformly in [0, 1].
Score
The denoising diffusion network c(0) in dimension is defined as a layers Resnet network where each Resnet block consists of the chaining of three blocks where each block has the following layers:
Linear (512, 1024),
1d Batch Norm,
ReLU activation.
The Resnet is preceeded by input embedding from dimension d to 512 and in the end output embedding layer projects the output the resnet from 512 to d. The time is embedded using positional embedding into dimension 512 and is added to the input at each Resnet block. The network is trained using the same loss in Ho et al. (2020) for a iterations using a batch size of 512 samples. A learning rate of 10-3 is used for the Adam optimizer Kingma & Ba (2015). Figure 9 illustrate the outcome of the learned diffusion generative model and the target prior. In table 3 we show the CLT 95% intervals for the SW between the learned diffusion generative model and the target prior.
Forward process scaling We chose the sequence of [8.) as a linearly decreasing sequence between 31 = 0.2 and 31000
10-4
30

---

Published as a conference paper at ICLR 2024
dg
d₂
104
10a
10.
10a
10a
10a
50
100
150
200
50
100
150
200
50
100
150
200
10"
1.0
1.0
KL
10a
1.a
10a
M
10a
50
100
50
100
Iteration
Figure 10: Evolution of KL with the number of iterations for all pairs of (dz, dy) tested in the FMM case.
Measurement model The measurement model was generated the same way as for the GMM case.
The posterior samples were generated by running the No U-turn sampler (Hoffman & Gelman (201 1)) with chain of length t and taking the last sample of the chain. This was done in parallel to generate 10 samples. The mass matrix and learning rate were set by first running Stan's 's warmup and taking the last values of the warmup phase.
Posterior
Variational inference: Variational inference in FMM shares the same details as the GMM case. The analogous of fig. 4 is displayed at fig. 10.
Additional plots: We now proceed to illustrate in Figures 11 to 13 the first 2 components for one of the measurement models for all the different combinations of (dz, dy) combinations used in table 1.
31

---

Published as a conference paper at ICLR 2024
MCGdiff
DDRM
DPS
RNVP
PCA2
PCA₂
PCA2
PCA1
Figure 11: First two dimensions for the FMM case with d₂ 10. The rows represent dy = 1, 5 respectively. The blue dots represent samples from the exact posterior, while the red dots correspond to samples generated by each of the algorithms used (the names of the algorithms are given at the top of each column).
MCGdi
DDRM
DPS
RNVP
PCA2
PCA₂
PCA2
PCA1
Figure 12: First dimensions for the FMM with d= 6. The represent dy 1,3, 5 respectively. The blue dots represent samples from the exact posterior, while the red dots correspond to samples generated by each of the algorithms used (the names of the algorithms are given at the top of each column).
32

---

Published as a conference paper at ICLR 2024
McGdiff
DDRM
DPS
RNVP
PCA2
PCA1
Figure 13: First dimensions for the FMM with d and dy 1. The blue dots represent samples from the exact posterior, while the red dots correspond to samples generated by each of the algorithms used (the names of the algorithms are given at the top of each column).
33

---

Published as a conference paper at ICLR 2024
B.3.3 IMAGE DATASETS
We now present samples from MCGdiff in different image dataset and different kinds of inverse problems.
Super Resolution We start by super resolution. We set 7y = 0.05 for all the datasets and Scoeff 0.1 for DPS. We use 100 steps of DDIM with 17 = 1. The results are shown in Figure 14. We use a downsampling ratio of 4 for the CIFAR-I0 10 dataset, 8 for both Flowers and Cats datasets and 16 for the others. The dimension of the datasets are recalled in table 4. We display in fig. 14 samples from MCGdi ff, DP Sand DDRMovr several different image datasets (table 4). For each algorithm, we generate 1000 samples and we show the pair of samples that the furthest apart in L norm from each other in the pool of samples. For MCGdiff we ran several parallel particle filters with N 64 generate 1000 samples.
CIFAR-10
Flowers (64, 64,3)
Cats
Bedroom (256, 256, 3) (256, 256,3, 3)
Church
CelebaHQ (256, 256, 3)
(W,,CC)
32
(128, 128, 3)
Table 4: The datasets used for the inverse problems over image datasets.
Gaussian 2D debluring We consider Gaussian 2D square kernel with sizes (w/6, h/66 and standard deviation w/30 where (w, h) are the width and height of the image. We set By 0.1 for all the datasets and Scoeff 0.1 for DPS.. We use 100 steps of DDIN with r = 1. We display in fig. 15 samples from MCGdi ff, DP Sand DDRMover several different image datasets (table 4). For each algorithm, we generate 1000 samples and we show the pair of samples that are the furthest appart in L norm from each other in the pool of samples. For MCGdi f we ran several parallel particle filters with N = to generate 1000 samples.
Inpainting CelebA We consider the inpainting problem the CelebA dataset with several different masks fig. 16. We show in fig. 17 the evolution the particle cloud with s.
1000
900
800
700
600
500
400
300
200
100
50
Figure 17: Evolution of the partiele cloud for one of the masks. The numbers on top and bottom indicate the step s of the approximation.
34

---

Published as a conference paper at ICLR 2024
CIFAR- 10
Flowers
Cats
Bedroom
Church
CelebaHQ
sample
observation
DPS
DPS
DDRM
DDRM
MCGdLEE
MCGdiFE
Figure 14: Ratio for CIFAR, for flowers and Cats and 16 for CELEB
35

---

Published as a conference paper at ICLR 2024
CIFAR 10
Flowers
Cats
Bedroom
Church
CelebaHQ
sample
observation
DPS
DPS
DDRM
DDRM
MCGdLEE
MCGdiFE
Figure 15
36

---

Published as a conference paper at ICLR 2024
Original
MCGdiff
MCGdiff
MCGdiff
MCGdiff
DPS
DDRM
SMOdiff
Lorem Ipsum
consectetir ac
eju med temp
et dolore magt
minim uenaam
Figure 16: Inpainting with different masks on the CelebA test set.
37


Reviews:
[{"review_id": "LLBxGbnz2h", "reviewer": "Reviewer_3c1h", "summary": "The authors introduce a method for addressing ill-posed inverse problems by employing a diffusion-based neural network model guided by Monte Carlo sampling. They refer to their approach as the MCGDiff (Monte Carlo Guided Diffusion) algorithm. Their algorithm is specifically applied to Score-based Generative Models (SGMs) and is used for various image-related tasks, including inpainting, super-resolution, deblurring, and colorization. To model prior distributions, the authors utilize Gaussian Mixed Models (GMM) and Funnel Mixture Models (FMM).", "questions": "1. Are there any plans to release your code for the MCGDiff algorithm to facilitate further research and practical applications in the broader community?\n\n2. Have you explored the application of the MCGDiff algorithm in addressing inverse problems beyond image-related tasks, and if so, could you share any insights into its performance and adaptability in those contexts?", "limitations": "", "rating": 10, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "A substantial portion of the research paper is devoted to providing a comprehensive background on the use of diffusion models for solving ill-posed inverse problems. Additionally, the paper includes detailed mathematical proofs regarding the algorithm's performance, covering both noiseless and noisy cases. This comprehensive coverage enhances the clarity and rigor of the research.\n\nThe authors' consideration of the broader applicability of their work beyond their experimental investigation of image data is commendable. It highlights the potential relevance and impact of their findings in a wider context of inverse problems.\n\nThe comparative results presented in the image context are particularly noteworthy. The research demonstrates impressive performance, especially in the way the generated posterior sampling distributions align with the exact ones. This underscores the effectiveness and accuracy of the proposed approach when compared to competing methods.", "weaknesses": "It would have been intriguing to explore the performance of the MCGDiff algorithm on non-image data. While the research focuses on image-related tasks, extending the investigation to other data types would provide a broader perspective on the algorithm's applicability and effectiveness across various domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5JIrjufdf8", "reviewer": "Reviewer_3L38", "summary": "This article propose a sequential Monte-Carlo method to solve linear inverse problems such as deblurring, super-resolution or inpainting with score-based generative priors (aka generative diffusion models) through the design of an efficient sampler.  The method is embodied into their proposed MCGdiff algorithm, which is proved to be sampling conditionally in a consistent manner from the diffusion posteriors. They evaluate the performance of their algorithm on various numerical simulation, demonstrating state of the art results on several imaging inverse problem applications.", "questions": "- How could the results be made more interpretable or controllable.\n- The experiments on GM seem to indicate that MCGdiff does sample from he posterior distribution but not necessarily in a \"thorough manner\" see Fig. 5 and 6 ; how can this be better handled?\n- How do deal with noise present in the training dataset ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper is dense but accessible and sufficiently well written, and although terse at time, thorough and complete. It is also well illustrated althought Fig. 1 is more puzzling than useful. The theory is well-developed, complete with all the proofs. \n\nIn terms of results,  MCGdiff seems to provide very good quality samples in all illustrated problems in comparison with other diffusion-based models. It interesting to see it perform well on difficult cases.\n\nCode is available.", "weaknesses": "The paper is detailed and the appendices can be hard to read. \nThe authors don't really solve inverse problems in the traditional sense. They generate realistic samples learned from a distribution that are consistent with the observations. While this sounds exactly like solving inverse problems, the resulting images, although better than existing methods and very sharp and detailed, only resemble the ground truth. I mention this because these types of methods cannot yet be used in sensitive contexts like medical imaging or science in general.\nThere is a lack of control and interpretability on the generated images, as with all the current diffusion methods.\nThe code issues have not been addressed. From looking at the source, the diffusion code seems to be based on DDPM, which is not acknowledged in the main text or appendices. Speed issues have not been mentioned.\nThe bibliography is excellent except for the first paragraph of the introduction. Linear inverse problems as described have been studied mathematically for a very long time (Fredholm, etc) and in computer vision since the late 1980s at least. Perhaps the bibliography should mention this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C2CyASxIam", "reviewer": "Reviewer_3cyr", "summary": "A method is proposed to solve linear inverse problems using score-based generative models (SGM), i.e. denoising diffusion models, in a Bayesian manner using Sequential Monte Carlo (SMC).  Such statistical approaches allow one to sample from the posterior, or an approximation of it, which facilitates uncertainty quantificaiton.  Experiments are presented on simple distributions where the posterior is known, where validation can be performed.  Further experiments are then presented on real images to demonstrate practical application, although underlying true posteriors for validation are not available in these settings.  Anonymized code is made available.", "questions": "Could the authors propose a metric to summarise performance over a large set of test images (see discussion in Weaknesses)?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Solving inverse problems using SGMs is a topical area of research at present, with a number of recent papers.  The key problem is to faithfully sample from the underlying posterior distribution.  It is not straightforward to integrate a likelihood into SGMs since it is difficult to consider a closed-form for the likelihood due to the dependence on time and thus noise level.  Existing approaches address this issue by various approximations (e.g. data consistency projections) but as far as I'm aware, and as the authors also state, no existing method fully solves this problem.  The authors claim they present the first provably consistent algorithm for condional sampling from the target posterior.  While I haven't checked all of the mathematical details I believe this is indeed the case.  Numerical experiments indeed confirm for the simple distributions that the sliced Wasserstein distance between the proposed method and the posterior recovered either analytically or by NUTS sampling is considerably smaller than for other SGM approaches for solving inverse problems.", "weaknesses": "The authors comment that standard image metrics, e.g. FID, are not suitable for evaluating Bayesian reconstruction methods for solving inverse problems.  Results from a handful of examples of inverse imaging problems are presented, which all look very compelling.  Nevertheless, it would be useful to summarise performance over a larger set of test images, if possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3nJUksQPoY", "reviewer": "Reviewer_68J7", "summary": "This work proposed an interesting method that combines particle filtering with diffusion model, trying to mitigate the issue of ill-posed inverse problem. The idea is novel and the demonstration of the proposed algorithm looks good in two dimensional illustrations, but the results in image datasets doesn't give clear distinctions from other existing methods.", "questions": "1. It seems very difficult to conclude that the proposed method outperforms the others in image dataset, is there any other ways to quantitatively  demonstrate the advantage of the proposed method?\n\n2. Does the intrinsic degeneracy issue from particle filter affect the stability and performance of the proposed method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Interesting novel idea to integrate particle filtering method into the exploration of posterior under the diffusion model framework. \n\n2. Theoretical work looks solid and convincing.", "weaknesses": "1. It seems the contribution point 3 is mainly a support for the first two points, hardly it can be classified as an independent contribution. \n\n2. A pseudo-code or diagram describing the idea of this method would make it much easier to be interpreted\n\n3. This method should be pretty slow as both diffusion model and particle filter are computationally heavy ones. Would be good to clearly state the limitation and also report the runtime etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)