Tweedie Moment Projected Diffusions for Inverse Problems
Authors: Benjamin Boys
Mark Girolami
Jakiw Pidstrigach
Sebastian Reich
Alan Mosca
Omer Deniz Akyildiz
Venue: Accepted by TMLR ()

Abstract: Diffusion generative models unlock new possibilities for inverse problems as they allow for the incorporation of strong empirical priors into the process of scientific inference. Recently, diffusion models are repurposed for solving inverse problems using Gaussian approximations to conditional densities of the reverse process via Tweedie’s formula to parameterise the mean, complemented with various heuristics. To address various challenges arising from these approximations, we leverage higher order information using Tweedie’s formula and obtain a statistically principled approximation. We further provide a theoretical guarantee specifically for posterior sampling which can lead to better theoretical understanding of diffusion-based conditional sampling. Finally, we illustrate the empirical effectiveness of our approach for general linear inverse problems on toy synthetic examples as well as image restoration. We show that our method (i) removes any time-dependent step-size hyperparameters required by earlier methods, (ii) brings stability and better sample quality across multiple noise levels, (iii) is the only method that works in a stable way with variance exploding (VE) forward processes as opposed to earlier works.

Published in Transactions on Machine Learning Research (09/2024)
Tweedie Moment Projected Diffusions for Inverse Problems
Benjamin Boys
Department of Engineering, University of Cambridge, Cambridge, United Kingdom
Jakiw Pidstrigach
jakiu. Siaikuppddtrigacchssttt.......
Department of Statistics, University of Oxford, United Kingdom
Mark Girolami
mag920cum..cc uk
Department of Engineering, University of Cambridge, Combridge, United Kingdom The Alan Turing Institute, London, United Kingdom
Sebastian Reich
sereich@@umi de
Universitat Potsdam, Berlin, Germany
Alan Mosca nPlan
alan
O. Deniz Akyildiz
deniz. Senit..kyildiz hwwpaiiidddiiiimiiaaaa....ooo uk
Imperial College London, London, United Kingdom
Reviewed on OpenReview: https: // openreview. net/ ftttm?? http://///00d0000000/o/o///////0
Abstract
Diffusion generative models unlock new possibilities for inverse problems as they allow for the incorporation strong empirical priors in scientific inference. Recently, diffusion models are repurposed for solving inverse problems using Gaussian approximations to conditional den- sities of the reverse process via Tweedie's formula to parameterise the mean, complemented with various heuristics. To address various challenges arising from these approximations, leverage higher order information using Tweedie's formula and obtain a statistically prin- cipled approximation. We further provide a theoretical guarantee specifically for posterior sampling which can lead to a better theoretical understanding of diffusion- based conditional sampling. Finally, we illustrate the empirical effectiveness our approach for general linear inverse problems on toy synthetic examples as well as image restoration. We show that method (i) removes any time-dependent step-size hyperparameters required by carlier hods (ii) brings stability and better sample quality across multiple noise levels, (iii) is the only method that works in stable way with variance exploding (VE) forward processes as opposed to earlier works.
1 Introduction
Due to the ease of scalability, diffusion models (Song et al. 2020 Ho et al. 2020) have received increased attention, producing variety of improvements in the conditional generation setting such adding classifier guidance (Dhariwal & Nichol 2021), classifier free guidance (Ho & Salimans 2022 conditional diffusion models Batzolis et al. 2021 Karras et al. 2022 2024) and DEFT (Doob's h-transform Efficient Fine Tuning) [Dekerr et al. 2024). These methods are useful for problems where paired (Xn,y) data are available, see, e.g..
Saharia et al. 2022]: Abramson et al. (202]. However, paired data is not always available, and there are cases
where we would like to study alternatives to classical Bayesian approaches to solve inverse problems (Taran-
tola, 22055 Stuart 2010) build samplers for complicated conditional distributions with known observation
operator mapping Xo to the observed data y that used to express known likelihood p(y[xn). With their

---

Published in Transactions on Machine Learning Research (09/2024)
flexibility, diffusion models replace handcrafted priors the latent signal with pretrained and strong empir- ical priors. For example, given a latent signal Xo (say face image), we can train diffusion model to sample from the prior p(xo). The idea for solving inverse problems is to leverage the extraordinary modelling power of diffusion models to learn samplers for priors and couple this with a given likelihood p(y]xo) for a given data y to sample from the posterior p(xoly) for inverse problems. However, designing a diflusion model for the posterior comes with challenges due intractability. Despite its challenges, this approach has been re- cently taking off with lots of activity in the field, e.g. for compressed sensing (Bora et al. 2017 Kadkhodaie & Simoncelli 2021), projecting score-based stochastic differential equations (SDEs) (Song et al. 2021b).
gradient-based approaches (Daras et al. 2022 Chung et al. 2022b), magnetic resonance imaging (MRT) by approximating annealed Langevin dynamics with approximate scores (Jalal et al. 2021). image restoration
(Kawar et al. 2022). score-based models as priors but with a normalizing flow approach (Feng et. al., 2023)
variational approaches [Mardani et al. 2023 Feng & Boumann 2023), Most relevant ideas to us, which we will review in detail in Scction use Tweodic's formula. (Efron 2011 Laumont al.
2022) approximate the 2023 and pseudo-guides diffusion models (IIGDM) (Song et al.) 2023). Similar approaches are also exploited using singular Sanuula-vvauue de- composition (SVD) based approaches Kawar et al. 2021). In this work, we develop approach which builds tighter approximation to optimal formulae for approximating the scores of the posterior diffusion model.
smoothed likelihood, e-g. Diffusion posterior sampling (DPS) (Chung el al.) 2022a
This paper is devoted to developing novel methods to solving inverse problems, given a latent (target) signal Xo Rd=, noisy observed data y E Rd,,, known linear observation map H, and pretrained diffusion prior. The main tool use is Tweedie's formula to obtain both the mean and the covariance for approximating diffused likeli- hoods, to be used for building the final posterior score ap- proximation. This is as opposed to previous works which only utilised first moment approximations using Tweedie's formula (Chung et al. 2022a Song et al., (023). We show that utilising covariance approximation, with moment pro- jections, provides a principled scheme with improved per- formance, which we term Tweedie Moment Projected Dif- fusions (TMPD).
Analytic TMPD
DIMPP
HIGM
DPS
Variance Mean
Error (Waserstein L.2 distance)
10
10²
Exact sampling
TMPD (ours)
To demonstrate our method briefly, Figure demonsttaaes a sampling scenario of a Gaussian random field (GRF) whose mean and variance entries are plotted under "Ana- lytic" columr!] We demonstrate the approximations under this setting provided by our method (TMPD) and its diag- onal (chcaper) version (DTMPD), compared with IIDM (Song et al. 2023). ; and DPS (Chung el al. 2022a). The fig- ure demonstrates the optimality of our method: Our first and sccond moment approximations become exact this case. This results in drastic performance improvement stemming from the statistical optimality of our method for Year-Gaussian settings and also unlocks possible line for thcoretical rescarch for understanding similar diffusion models for inverse problems.
DTMPD (ours)
RGDM (Song al., 2023)
DPS (Chung et al. 2022)
10
10'
10²
10"
N. Number of samples
Figure 1: Error to target posterior for Gaus- sian random field. (Top row) visualisation of the empirical mean and variance of the 1500 samples that were used to compute this er- ror against the analytical moments. (Bottom) Wasserstein distances of different methods w...t. sample size. For details, Appendix F.2
In what follows, we will first introduce the technical back- ground in Section 2 and then describe TMPD in detail in
Section 3 We will then provide some theoretical results about our method in Section 4 and provide discus- sion closely related work in literature in Scction 5 Finally, Section will present experiments Gaussian
We only plot the variances for visualisation while the GRF has full covariance.
2

---

Published in Transactions on Machine Learning Research (09/2024)
mixtures, image impainting and super-resolution, demonstrating quantitative and qualitative improvements provided by TMPD.
2 Technical Background
In score based gencrative models (SGMs) (Song et al., 2020) and denoising diffusion probabilistic models (DDPM) (Ho al., 2020), the goal is sample from larget distribution po Pdata- To that end, an artificial path P: is introduced, with the property that Pt will approach N (N, I) for large i.e,, Pt + V0, Ia) as - co. Then, one learns to reverse this process, in order to transform samples from standard normal distribution into samples from Pdata- More recent advances in generative modelling include methods with artificial paths based on stochastic interpolants or cont inuous normalizing flows, see, e.g., e.g., Albergo & Vanden- Eijuden 2022); Lipman et al. (2022). Conditional flow matching draws inspiration from the denoising score matching approach, but gencralizes matching vector fields directly. We have focused our approach denoising-diffusion models in the SGM paradigm because of readily available pretrained diffusion models.
In the SGM paradigm, stochastic differential equation (SDE) is used to noise the data, and the interpolation parameter will take continuous values in t E [0, T). In the DDPM setting, t is discrete. However, the DDPM Markov chain can be seen as a discretization of the SDE (Song et al. 2020).
Recent developments have improved the noising schedule of score-based diffusion models for image data that parameterise the transition kernels of the forward process in terms of the signal-to-noise ratio of: (xx (xx; SyXo, Id,) (Karras et al.) 2022: 2024). In this paper, we focus derivations on the Variance Preserving (VP) SDE formulation, although our approach SDEs, such as the Variance Exploding (VE) SDE (see Appendix Ch. The VP transition kernel that we choose to focus our derivation is given by setting
be generalised and derived for other
(xx(xx)) N(xx (1xxiv/oss vaxoo la.) where ar a= exp (-f B(o)ds) and 0r == Ot.y
which are the transition kernels for the time-rescaled Ornstein-Uhlenbern process:
dx, == + BB()dw,, Xo ~ Pdata
(1)
The corresponding reverse SDE is then given by
1 dz 2 3(T - t)z,dt + +(T - V), logpr-r(zi)dd + VB(T - t)dw:,
20 PT.
A parameterisation that performs well in practice is {() = 8min + + Pmin). In the diffusion modelling literature, the time-rescaled OU process is also sometimes called Variance Preserving SDE. This is not the only SDE that suitable for the forward process. See Appendix C for details on a time-rescaled Brownian motion (Variance Exploding SDE (Song al. 20201).
There are two usual approximations to solve the SDE in Equation First, we do not know pT, since it is noised version of the distribution Pdata- However for T large enough, Pref = -0, Ta We also do not have Poo which we need for the drift in Equation 2 This can be circumvented by approximating the drift using scoro-matching techniques (Hyvärinen) 2005 Ho ct al. 2020
approximate pr
These methods construct an estimate of the score function by solving the score matching problem in the
form of tx) se Vx, lx:).(x:). This score can also be used in the setting of DDPM (Ho et al. 2020
2.1 Conditional sampling for the linear inverse problem
In the preceding section we introduced diffusion models as method to sample from target distribution Pdata- We now suppose that we have access to measurements, or observations y E R& of xy E IR:::
y = Hxo u, u~ u~N(0,, IT
(2)
3

---

Published in Transactions on Machine Learning Research (09/2024)
We would then be interested in sampling from the conditional distribution of x given y, i.c., Pdata ly). To that end, we have to modify the reverse SDE. We would like to sample from the reverse SDE targeting Pdatal ly); instead of the one targeting Pdata-
Optimally, we would want to replace the score 22, log (r-((z)) in Equation with the posterior score Ve, (2ll)) Written in terms of the forward process, this coincides with
Vx, x; Poly (x:ly) Vx, (x + Vx, log Py)
(3)
The term Py)o(y(x;) is given by the integral
Pyl(y(xx))=
(y)o(y|xx)) (ojo)o (x]|xx))xx
(4)
which involves a marginalizaatin over Xo. The above integral is difficult to evaluate since the term poje (Xo] xx) is only defined implicitly through running the diffusion model. One way around this to train a neural net-
work to directly approximate V Vogg Puly (x, ly) (Batzolis al. 2021 Karras et al. 2022]. Alternatively, if one already has access to approximation of V x, Po
train auxiliary network to approximate the term V log Py|r (y xx) in Equation 3 Dhariwal & Nichol 2021 Denker et al. 2024). However, these methods can be time and training-date intensive, as it is necessary to retrain networks for each conditional task as well have access to paired training data from the joint distribution of (Xo. y). Alternatively, one could try to do a Monte-Cari Carlo approximation the probability flow ODE together with its derivative (Song al. 2020 Section D.2) for each sample, which
the score corresponding to Equation | But this needs evaluating
is prohibitive, also suffers from high variance (Mohamed et al. 2020 Section 3).
3 Tweedie Moment Projected Diffusions
In this section, we first. introduce Tweedie moment projections in Section 3.1 below. Our method relies the approximation Poje xx) x N (xo; (x)) Coj (x:)) to make the sampling process tractable. In that case, since the conditional distribution of given Xo is also Gaussian, we can compute the integral in Equation analytically - Py (y(x;) will just be another Gaussian in that case, its mean and covariance being determined through mojr, Cops, H and Qy- In particular, we can then use this Gaussian to approximate V log Py ( (y)))((())))))))))))))))))))(( since the score of a Gaussian is available in closed form.
3.1 Tweedie moment projections
Instead of just approximating the variance of Poje (Xolx)) heuristically, we approximate it by projecting onto the closest Gaussian distribution using Tweedie's formula for the second moment. Our approximation this stage consists of two main steps: (i) Find the mean and covariance of (xoo x.) using Tweedie's formula, and (ii) approximate this density with Gaussian using the mean and covariance of Poje (x) (moment projection). Due to this approximation, we will refer to the resulting methods as Tweedie Moment Projected Diffussins (TMPD). We will first introduce Tweedie's formula for the mean and covariance and then describe the moment projection.
Proposition 1 (Tweedie's formula). Let moje and Coje be the mean and the covariance of poje po)e(xx00 xx)) respectively. Then given the marginal density Pe (x(x,), the mean is given as
moje [[xo|xx] = -xi + Vx, lx:))(x:)),
(5)
Vo:
and the covariance Coje given by
Coje = E [Xo - my (x - moje)
[x]]
Ut (Ig_ + o2 2 (x = 2tt
(6)
Vx. mojt-
ag
Var
4

---

Published in Transactions on Machine Learning Research (09/2024)
The proof is an adaptation of Meng et al. [2021 Theorem 1), see Appendix A.1 While moje and Coje give us the moments of the density Poji (X (xj x we do not have the exact form of this density. At this stage, we employ moment projection, ie., we choose the closest Gaussian in Hellbackk-eibbler (KL) divergence which is a distribution with the same first and second moments, as formalised next.
Proposition 2 (Moment projection). Let (x)e(Xa|xx) be distribution with mean moje and covariance Cojt. Let Pojo(xo|x,) be the the closest Gaussian in KL. divergence to Poje (xo/xx)) i.e.,
{o)x(xo(x)) = arg min (PL( (xo xx (x))) QEQ
(7)
where Q is the family of multicoriate Gaussian distributions. Then
Po xxx(xx) (xo; moje: Coje).
(8)
This is a well-known moment matching result, see, e.g., Bishop 20066 Section 10.7). Merging Propositions and 2 leads to the following Tweedie moment projection:
Puje (xol xx) se N Xoi Moje:
Vx, moje)
(9)
Va,
where Maje is given in Proposition In the next section, we demonstrate how to use this approximation to obtain approximate likelihoods.
3.2 Tweedie Moment Projected Likelihood Approximation
We next use the approximation in Equation
to compute the following integral analytically
Py)((((x)) (x))
(10)
= =N (y; Hm. H" + ooLa,
Let us recall that mean and covariance terms are a function x, by making them explicit in the notation, i.e., maje (x)) and Coj+(x,) for the mean and covariance respectively. To compute Vx, log (y)x(y)x,), we require further approximations since have x; dependence in both the mean and the covariance of Equation 10 which is computationally infeasible to differentiate through. For this reason, we treat the matrix Coj. like a constant w.r.t. when computing the gradient (which is the case if Pdata is Gaussian). For non-Gaussian Pdesa, this results in a computationally efficient sampler using Coje as a preconditioner for the step size, as otherwise, the resulting terms can be expensive to compute. This leads to an approximation of the gradient
(x,) = Vx, (x) H (H,)) + +o "y Hm))(x))
(11)
~V Vxx, log log
where Vx, only operates maje in Equation Equation which leads to
I1 Another way to write this approximation is to use
(xx = aD Cxr( (x.)) + IgIa, (Imo((x))
3.3 Algorithms
Plugging the approximation in Equation 11 into the reverse SDE in Equation 2 together with the prior score as described in Section 2.1 results in TMPD for conditional sampling to solve inverse problems. The SDE we will approximate numerically to sample from the conditional distribution is given by
1 -T - -lzdd)) + 3(T (V, logpT-.(Z.) + (Z. + VB(T l]dw, dz,
(12)
5

---

Published in Transactions on Machine Learning Research (09/2024)
where Z
Pr and (z+) (y is our approximation to the data likelihood, given by
Equation We call this SDE the TMPD SDE.
We have two options to convert TMPD SDE into implementable methods: (1) score-based samplers (Song & Ermon 2020j, which we abbreviate as TMPD since they are Guler-Maruyamn discretizations of the TMPD SDE; and (2) denoising diffusion models (TMPD-D). The denoising diffusion approach derived from approximate reverse Markov chains and is the approach of DDPM and DPS methods (Ho et all 2020 Chung et al. 2022a). We note that the Gaussian projection can be used in this discrete setting, assuming that the conditional density is available analytically in Ho et al. (2020). and can be written Pojo (Xo) = MieNN(xx,, (xx: Vo, Ur Td, ). The idea is to update the unconditional mean Mojn (X,) of the density Pojn (Xo (x)) with a Bayesian update: p(xo X., y) X
Xn). Given similar formulation as above, Xx,) x (xx;; mgin Coja) and a likelihood similar to Equation 10 where can be replaced by n, we can compute the moments of p(xo|>,, (x.)xx.))) Y) analytically, which
assuming we have a readily available approximation PXon(Xx
we denote m and n The Bayes update for Gaussians gives (Bishop. 2006)
m majn + Coja H (HCoj,H' + To) Hm
(13)
Incorporating Equation 13 for n = N -..,..,, into the usual Ancestral sampling (Ho et al. 2020 steps leads to Algorithm 1 termed IMPD-Denoising (TPMD-D). The update in Equation 13 can be used in any discrete sampler such denoising diffusion implicit models (DDIM) (Song et. al., 2021a).
3.4 Computationally cheaper approximation of Moment Projection
We show in our experiments promising results for TMPD motivating the exploration of less computation- ally expensive approximations to the full Jacobian. In particular, we empirically study a computationally inexpensive method that applies to inpainting and super-resolution, below.
To make the computational cost of TMPD smaller, we can make an approximation of the Gaussian Projection that requires fewer vector-Jacobian products and does not require linear solves. One approximation that we found useful for sampling from high dimensional diffusion models, e.g., high resolution images, is denoted here diagonal Tweedic Moment Projection (DTMPD). Instead of the full second moment, DTMPD uses the diagonal of the second moment Vx, Moje & diag(V moje)- Intuitively, this approxination will perform well empirically since it is similar approximation to IIGDM that assumes dimensional independence of the distribution p(x)/x;), but unlike IGDM, this diagonal approximation is the same as using the closest dimensionally independent Gaussian KL divergence to Poje (Xolx.).
The biggest drawback of our method is that without further approximation, doesn't 't scale up to the high dimensions of image data. This is because even calculating the diagonal of a Jacobian requires computing d, vector-Jacobian products since in general every element of the Jacobian at location x,, Vx, maja depends every element of x,. Therefore we must resort to further approximation that exploits knowledge of the observation operator H.
For the cases of super-resolution and inpainting, a further approximation that allows scaling up to the dimensions of image data is approximating the diagonal of the Jacobian by the row sum of the Jacobian which only requires a single vector-Jacobian product and brings the memory and time complexity of DTMPD down that of HIGDM. We exploit the sparsity of H to make the rowsum approximation of the diagonal more accurate by masking out (zeroing) the values in the Jector-Jacobian product that that will not contribute to the diagonal of HT. We discuss a justification of this approximation in Ap. E. We use this approximation in the image experiments and find that in practice it is only (1.5 + 0.1) x slower than IIGDM and DPS across all of our experiments (Sec. 6), with competitive sample quality for noisy inverse problems and without the need for expensive hyperparameter tuning. Finally, we note very recent work (Rozet al., 20244) that cireumvents our heuristic by applying the conjugate gradient (CG) method, unlocking using the approximation Fq. 11 to be used in practice for non-sparse H (see Ap. E.1 for more details).
6

---

Published in Transactions on Machine Learning Research (09/2024)
4 Theoretical Guarantees
Because of the approximations, our method, as well as HGGDM Song et al. (2023) and DPS (Chung et al, 2022a) do not sample the exact postcrior for general prior distributions. Therefore, one cannot hope for these methods sample the true posterior and a priori it is not even clear how the sampled distribution relates the true posterior. Without further justification, such methods should only be interpreted as guidance methods, where paths are guided to regions where given observation y is more likely, not as posterior sampling methods.
We justify our approximation by showing that the TMPD-SDE in Equation 12 is able to sample the exact posterior in the Gaussian case. One can see that this contrasts with IIGDM and DPS in our numerical experiments or by explicitly evaluating their approximations on simple one-dimensional examples.
Proposition 3 (Gaussian data distribution). Assume that Pdata is Gaussion. Then, the posterior score ez- ervors in the initial condition and drift approzimation
pression using Equation 77 is eract, i.e., if there 5o(x,, t) Vx, log (x;), the TMPD will sample P-ata(-ly) at its final time.
The proof is given in Appendix B.1 However, most distributions will not be Gaussian. The following theorem generalizes the above proposition to non-Gaussian distributions, as long as they have a density with respect to a Gaussian. We study how close our sampled measure will be to the true posterior distribution and give explicit bounds on the total variation distance in terms of the regularity propertics of the density: Theorem (General data distribution). Assume that the data distribution Pdata be written
Pdata (Xo) = exp()(x))) po, En),
(14)
for some pb and Eo. We furthermore assume that for some M 2 1, it holds that 1/M S exp(O(x)) S M and (x)( S L for all x Rd. Then
[l 1lv 5C(++TL1 (115/2- +1)+ 11
(15)
where qi denotes the law of the corresponding reverse-time process for Equation 12 at time and the constant C that only depends on y, H, u. Ho and Eg.
See Appendix B.2 for a proof.
In the limit, when Pdata becomes more similar to a Gaussian, the + Equation 14 converges to zero, and therefore M + and
0. In particular, the right hand side Equation 15 converges and we recover
the result of Proposition When Pdata not Gaussian, the right hand side of Equation 15 gives us upper bound of our sample distribution to the true posterior.
5 Related Work
In this section, we review two closely related methods, that we use as benchmarks, and summarise the relationship between them.
The first work by Chung et al. (202a)), abbreviated DPS-D here introduced the use of Tweedie's formula to approximate (x xx) with Dirac delta (point) distribution centred at moje- In our framework this corresponds to choosing zero covariance, ie.,
DPS-D mojt m
DPS-D 0. and C
(16)
In the work Song et. al. (2023 abbreviated TIGMM-D here the same estimator for the mean is chosen. However, the variance is set to a multiple of the identity, corresponding to choices
Moj
IIG
moj and
CofE = -
(17)
2Siuce the authors run DDPM-type sampler, we re-abbreviate DPS DPS-D in this work (as DPS approximation itself can also be run with Euler-Maruyama schemes).
Sinnee the authors run DDIM-type sampler, we re-abbreviate IGDMM as ITGM-DD in this work (as IGGDM approximation itself also run with Ruler-Marnyama schemes).
7

---

Published in Transactions on Machine Learning Research (09/2024)
The choice of ra is such that it matches the variance of the reverse SDE if pdata would be a standard normal distribution. Since they employ different forward SDE to ours (variance exploding SDE), is set, when the variance of the data is 1, to be equal to 0e/(er + 1). In our case, with the OU/variance preserving SDE as a forward process, rs would be equal to ve/(u, + at), when the variance of the data is 1.
Another relevant very recent work by Finzi et al. (2023) arrives at the approximation matching ours in the context of modelling physical constraints. However, we focus on general linear inverse problems outside the physical domain in this paper together with novel theoretical result. Also, note the work of [Stevens] et al. 2023) who consider the cesimmmm---psstrrion (MAP) approach to find the moments of Poje.
Finally, sequential Monte Carlo (SMC) methods are particle-based posterior sampling schemes that are asymplotically exact in the limit of infinitely many particles. Cardoso et al. (2023) larget the posterior for linear inverse problems, and our first experiment makes a comparison to Cardoso et al. (2023) for Gaussian Mixture model. However, for high dimensional data (such as images), this mothod can exhibit the well studied SMC problem of weight degemeracy, where particle weights collapse. We note a recent state of the art work (Wu et al.] 2024) that tackles weight degeneracy by introducing the Twisted Diffusion Sampler (TDS), with heuristic twisting functions approximating Pyle (y/x)) that increase computational efficiency of SMC. The method is still asymptotically exact as long as the approximation of (y)x(y)x,) converges to p p(y(xu) as approaches O. The authors note that the efficiency of TDS (Wu et al.] 2024) depends on how closely
the twisting function approximates the exact likelihood. TMPD (Equation 10 could be a good twisting candidate.
6 Experiments
In this section, we demonstrate our results as well as the peformance of other approximations to the likelihood provided in CChugg al.] (2022a); [Song et al.] (2023). In particular, we perform comparisons for two of our methods TMPD (an SGM using our approximation) and TMPD-D (a DDPM sampler using Equation 13 We compare these to DPS (an SGM sampler using the posterior approximation in Equation 16] DPS-D (Chung et al. 2022a) (a DDPM-type sampler using Equation 16), IGDMM (Song et al.) 2023 (an SGM sampler using the posterior approximation in Equation
17), and finally IIGD-DD (a DDIM-type sampler
using Equation 17 but in our experiments a DDPM-type sampler since we set the DDIM hyperparameter
7 = 1.0 which is defined in Algorithm 1 by Song et al. (2021a) who show that this is equivalent to a DDPM-type sampler).
The code for all of the experiments and instructions to run them are available at github.com/bbbbb. bb5115 bbtb////d/////jjja Impdjax and github.com// /tmp1///mmooochh
6.1
Gaussian Mixture Model
We now demonstrate a nonlinear SDE example and follow the Gaussian mixture model example of Cardoso ct al. (2023) where the data distribution px() is mixture of 25 Gaussian distributions. The mcans and variances of the components of the mixture are given in Appendix E.3 In this case, for each choice of observation y, observation map H and measurement noise standard deviation dy, the target posterior can be computed explicitly (see Appendix E.3).
To investigate the performance of posterior sampling methods, for each pair of dimensions and observation noise (dx, dy, σy) E [8, 80, 800) x (1, 4) x (10-2, 10-1, 100] we randomly generate multiple measurement models (y, H) IRdy x Edyxd, and equally weight each component of the Gaussian mixture. Further details are given in Appendix E. We chose to control the dimension to gain insight into the performance of posterior sampling methods under varying dimensions. We also chose to control the noise level since the different posterior sampling methods have accuracy that depends on the signal-to-noise ratio. Through randomil varying the observation model, we gain an insight into the performance of the posterior sampling methods with different levels of posterior multimodality. This example is interesting because it allows us to study the behaviour of our methods on non-Gaussian problems in high dimensions whilst having access to the target posterior with which to compare (usually, obtaining a 'ground-truti truth' posterior is not feasible for non- nan-Gausiia problems).
8

---

Published in Transactions on Machine Learning Research (09/2024)
Prior
Target
TMPD-D (ours)
IIGDM-D (Song al. 2023)
DPS-D (Chung et al. 2022)
Figure 2: We display the first. two dimensions of the GMM inverse problem for one of the measurement models tested (H, By 0.1, (dx; dy) (80, 1)). The blue dots represent samples from the target posterior, while the red dots correspond to samples generated by each of the algorithms used (the names of the algorithms are given at the bottom of each column).
Table 1: Sliced Wasserstein for the GMM case.
The full table is in Ap.
B3
dr dy
8
8
80
80
80
800
800
800
2
1
2
TMPD-D
1.6
0.7
0.3
2.7
1.0
0.3
3.1
0.4
DTMPD-D
1.8
3.3
0.4
2.8
3.2
0.7
3.7
3.5
0.7
DPS-D
4.7
1.8
0.7
5.6
3.2
1.2
5.8
3.5
1.4
IIGDM-D
2.6
2.1
3.8
3.2
2.8
0.6
3.5
3.1
0.4
TMPD-D
1.4
0.9
0.3
2.3
1.2
0.4
2.9
1.3
0.4
DTMPD-D
1.8
2.7
0.5
2.6
3.2
0.8
3.4
3.4
0.8
DPS-D
4.7
1.5
0.8
5.1
3.1
1.0
5.7
3.1
1.3
JIGMM-D
2.2
1.6
3.8
2.9
2.7
0.6
3.3
2.7
0.4
TMPD-D
0.9
0.9
0.6
1.5
1.1
0.9
1.5
1.2
0.9
DTMPD-D
0.9
1.7
0.9
2.1
0.9
1.4
2.0
DPS-D
5.2
3.5
2.5
6.9
3.9
1.7
6.8
4.7
0.9
IIDM-D
1.5
2.3
1.8
1.6
1.4
0.9
2.0
2.0
0.6
We use the sliced Wasserstein (SW) distance defined in Appendix E.3 to compare the posterior distribution estimated by each algorithm with the target posterior distribution. We use 10 slices for the SW distance and compare 1000 samples of TMPD-D, HGDM-DD and DPS-D in Tables obtained using 1000 denoising steps and 1000 samples of the true posterior distribution.
Table indicates the Central Limit Theorem (CLT) 95% confidence intervals obtained by considering 20 randomly selected measurement models (H) for each setting (dx, (xx,oy), Figure shows the first two dimensions of the estimated posterior distributions corresponding to the configurations (80, 1) from Table I for one of the randomly generated measurement model (H, By = 0.1). These illustrations give us insight into the behaviour of the algorithms and their ability accurately estimate the posterior distribution. We observe that TMPD-D estimates the target posterior well compared to IIGDM-D and DPS-D. TMPD-D covers all of the modes, whereas IIGDM-D and DPS-D do not.
We perform the same experiment using 1000 samples of TMPD, DTMPD, DPS and HIGMM, obtained using 1000 Fuler-Maruyama time-steps, and results are shown in Appendix E.3
A direct comparison to Cardoso et (223) using their original experimental setup is shown in Table Appendix E.3 which shows competitive performance for posterior sampling compared to Sequential Monte- Carlo, exact sampling method.

---

Published in Transactions on Machine Learning Research (09/2024)
6.2 Noisy observation inpainting and super-resolution
We consider inpainting and super-resolution problems on the FFHQ 256 256 (Karras et al. 2019 and CIFAR-10 32 x 32 (Krizhevsky et al., 2009) datasets. We compare TMPD to IIDGM and DPS. We also compare score-based diffusion models with their denoising-diffusion counterparts (denoted with suflix, -D).
Firstly, we follow the benchmark used by Chhung et al. (2022a) and use a Variance Preserving (VP) SDE, using a DDPM sampler, on FFHQ 256 x 256 using 1k validation images. The pre-trained diffusion model for FFHQ was laken from Chung et al. (202aa) and was used directly without any finetuning. We follow Chung et al. (2022a) and use various forward operators. For resor-iesolution, we use a downsampling ratio of 4 (256 256 + 64 x 64) and bicubic interpolation; for 'box' mask inpainting we mask out 128 x 128 region and for 'random' mask impainting we choose random mask for each image masking between 30% and 70% of the pixels. Images are normalized to the range [0, 1] and it is on this scale that we add Gaussian measurement noise with standard deviation By E (0001,0.05, 0.2). For quantitative comparison, we focus on two widely used perception distances, Fréchet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPIPS) distance. FID evaluates consistency with the whole dataset using summary statistics from the FFHQ-50K dataset. We also evaluate obscrvation data similarity using various distances between a sampled image and ground truth image: LPIPS, mean-squaredd-rroo (MSE), peak signal-to-noise-ratio (PSNR) and structural similarity index measure (SSIM). For IIGDM-D use the algorithm and default hyperparaneters as described in Song et al. 2023). For DPS-D we use the algorithm in the codebase provided by the authors Chung et al.] (2022a) and we use their default hyperparameters such as their suggested step- size hyperparameter for this lask, and static-threshoodding (clipping the denoised image at each step to a range [-1, 1)) whereas TMPD-D does not require hyperparameter tuning or statie-thresholdingg. The results for FFHQ sampled using VP DDPM are shown in the appendix Table 7 We observe that DTMPD-D is competitive with DPD-D over range of noise levels, however, IIGDM-D is not able to produce high quality reconstructions for larger noise levels.
We note that the hcuristics used in the DPS and HIDM implementations have been designed to work with the VP-SDE (DDPM sampler), and therefore the performance may not be robust to the choice of SDE. illustrates this when each method is applied to the VE-SDE on a sample from the FFHQ validation
Fig 14 dataset. We next compare performance to TMPD across VP and VE-SDE samplers and a range of noise levels on CIFAR-10 10 64 64 using 1k validation images.
We use pretrained denoising networks for CIFAR-10 that are available here For impainting, we use 'box' and 'half' mask. For "half-type inpainting, we mask out a 16 x 16 right half region of the image; for box-type inpainting, we mask out an 8 x 8 box region following Cardoso et al. (2023). For super-resolution, we use a downsampling ratio of 2 on each axis (32 x 32 + 16 x 16) with a nearest-neighbour downsampling method; and a downsampling ratio of 4 (32 x 32 - 8 x 8) with bicubic downsampling. Images are normalized to the range [o, 1] and it is on this scale that we add Gaussian measurement noise with standard deviation By E {0.01, 0.05, 0.13. Whereas no hyperparameters are required for our method, we chose the DPS scale hyperparameter by optimising LPIPS, MSE, PSNR and SSIM validation set of 128 images (sce Fig. 9 for example). We found that static thresholding (clipping the denoised image estimate to range [0, at each sampling step) is critical for the stability and performance of both DPS-D and IGGDMDD. Stability was noted as a limitation in Chung et al.] (2022a), and they suggest that devising methods to stabilize the samplers would be a promising direction of research. We find that our TMPD-D method and the diagonal approximation DTMPP-DD is stable across SDE, noise-level and observation maps, without the need for static thresholding. Whilst for VE IIGM-DD we found the original algorithm in Song et al. (2023) to be stable, for VP IIGMM-D, the original algorithm, whilst stable for FFHQ, was not stable, even with statie-threshodimn for CIFAR-10. We chose to bring IIGDM-D step closer to our algorithm by substituting their likelihood score into an Ancestral sampling algorithm, instead of a DDIM algorithm as suggested in Song et al. (2023 which produced stable samples.
The methods TMPD, IIGDM and DPS all have the same mumerical solver of their respective reverse-SDB and DTMPD-D, HGDM-D and DPS-D all use DDPM since DDIM and DDPM are equivalent algorithms with our chosen DDIM hyperparameter 7) = 1.0. Therefore, the sampling methods being compared only differ in the Vx, log (y)x)) term in their reverse-SDE, and so this experiment. allows us to study the
10

---

Published in Transactions on Machine Learning Research (09/2024)
Table 2: Summary of results using the VE and VP-SDE for increasingly noisy Øy E {0.01, 0.05, 0.13 obser- vation inpainting and super-rcsoltion problems on FFHQ 1k validation set. The full results are in Ap. E.4
SDE
VE-SDE
VP-SDE
PROBLEM
METHOD
FID 4
LPIPS
4
FID +
LPIPS +
0.01
DTMPD-D
32.3
0.203
0.099
29.6
0.230
4x 'BICUBIC'
DPS-D
47.0
0.273
0081
31.4
0.234
0.008
SUPER-RRSOLUTION
TGGM-D
37.4
0.244
1 0.00
29.7
0.198
0.037
yy 0.05
DTMPD-D
32.1
0.268
0.005
32.7
0.304
= 0.033
4x 'BICUBIC'
DPS-D
105.9
0.590
A 0.096
29.3
0.280
= 0.051
DEPER-RRSOLUTIOO
TIGM-DD
106.8
0.592
1
45.1
0.311
= 0.017
yy 0.1
DTMPD-D
32.7
0.310
0.003
38.0
0.348
=
4x 'BICUBIC'
DPS-D
114.0
0.569
30.9
0.318
IIDM-DD
206.0
0.724
0.084
119.6
0.589
By 0.01
DTMPD-D
30.2
0.114
25.7
0.153
'BOX' MASK INPAINTING
DPS-D
23.9
0.093
0.019
31.5
0.175
IIDM-DD
27.1
0,108
0.025
143.8
0,247
0.024
8y 0.05
DTMPD-D
33.6
0.186
0.006
27.0
0.240
0.038
'BOX' MASK
DPS-D
39.7
0.318
+ 0.01
30.7
0.228
0.016
INPAINTING
IIGMM-D
49.5
0.354
+ 0.04
159.3
0.448
0.046
By 0.1
DTMPD-D
34.0
0.223
0.041
29.6
0,292
= 0.000
'BOX' MASK
DPS-D
59.1
0.467
+
29.3
0.259
INPAINTING
IIGMMDD
72.6
0.529
165.7
0,539
behaviour of our method on inpainting and super-resolution compared to the different approximations of the smoothed likelihood. A summary of the results for CIFAR-10 sampled using VE DDPM are shown in Table [
The complete results are in Tables S and 12 for VP and VE DDPM respectively, and Tables 9 and 13 for score-based VP and VE-SDE, respectively. For more experimental details including illustration of samples used to generate the tables can be found in Appendix E.4 Our method is the only method able to provide high-quality reconstructions independently of the SDE, time discretization or noise level used. On the other hand, we see that DPS-D and TIGDM-D are not able provide high-quality reconstructions for the VE-SDE. For the continuous time methods, IIGM and DPS are outperformed by TMPD for both VE and VP-SDEs in the majority of tasks.
7 Discussions, limitations and future work
In this paper, introduced TMPD, diffusion modelling approach solve inverse problems and sample from conditional distributions using unconditional diffusion models. On various tasks on the VP-SDE, TMPD achieves competitive quality with other methods that aim to solve the noisy, linear inverse problem while avoiding the expensive, problem-specific training of conditional models. Our method is more versatile since it can also be used for the VE-SDE, and for large noise and different time discretizations.
TMPD is slower, as each iteration costs more memory and compute due the Jacobian over the score model. Even with diagonal and row-sum approximation to the Jacobian, the method is around 1.5x slower than DPS and IIGDM. The row-sum approximation is not suitable for inverse problems with more complicated, non-diagonal and nonlincar observation operators, therefore, it would be helpful to explore methods to circunvent heuristics. For example, heuristics can be circunvented by noting in the definition of Equation does not require the inverse but rather solving a system of linear equations with right hand side y Hm (x,). very recent work (Rozet al. 2024), uses the natural choice of the conjugate gradient (CG) method solve this linear system with success on non-diagonal and nonlinear observation operators, even for small number of iterations of the CG method.
11

---

Published in Transactions on Machine Learning Research (09/2024)
Using flow based methods (Albergo & Vanden-Eiinden 2022 Lipman al. 2022 as unconditional priors and building conditional sampling methods that leverage flows as pretrained models (Ben-Hamu et al. 2024
Pandey et al. 2024) is related to the approach used in this paper. For example, it is possible to directly substitute in the TMPD approximation to the PiGDM method applied to flows (Pokle et al. 2023 which would be a fruitful direction for future work.
Many state of the art diffusion models operate in a latent space which would make any observation operator nonlinear, and thus linear observations maps do not suffice in the latent diffusion setting. Furthermore, for inverse problems in general it may be excessive to train diffusion model for the full data distribution, and it could be difficult to outperform a diffusion model trained for specific inverse problem. For example, using diffusion models trained specifically for the super-resolution problem in a cascade together with noise conditioning augmentation (Ho et al., 2022 has been extremely effective in progressively generating high- fidelity images, see e.g., (Saharia et al. 2022
A limitation of our method is that without approximations, the full method may be costly to implement in high dimensions of image data. On the positive side, our approach does not require any hyperparameter tuning, does not require static-thresholdingg of the denoised image, and is more principled when compared to existing approaches, as shown in Section 4 We show that our method, unlike IIGDM, does not fail for the cases where the additive Gaussian noise is significant. We provided a way to analyse similar methods and our moment approximations can be analysed more rigorously to provide deeper theoretical results for these kinds of methods. Our future work plans include expanding the analysis we provided in this work.
Acknowledgements
This work has been supported by The Alan Turing Institute through the Theory and Methods Chal- lenge Fortnights event Accelerating generative models and nonconvez optimisation, which took place 6-10 June 2022 and 5-9 Sep 2022 The Alan Turing Institute headquarters. JP and SR acknowledge funding by Deutsche Forschungsgemeinscaattt (DFG) Project-ID 318763901 SFB1294. M. Girolami was supported by a Royal Academy of Engineering Research Chair grant RCSRF1718/6/1 6/34, and EPSRC grants EP EP/WOOO PP//000586/11 EP /V/056441//, EP/ /V056522/1 1, EP/RO18413/2, EP EP/RO34710/ EP JEP/Y22805/11, and EP EPPR004889/11 The authors would also like to thank the Isaac Newton Institute for Mathematical Scienees, Cambridge, for support and hospitality during the programme The Mathematical and Statistical Foundalion of Future Dala-Drived Engineering where work on this paper was undertaken. This work was supported by EPSRC grant EP EP/R014604/1. BB gratefully acknowledges the EPSRC for funding this rescarch through the EPSRC Centre for Doctoral Training in Future Infrastructure and Built Environment: Resilience in a Changing World (EPSRC grant reference number EP/ JPP/S2302X/1);; and the support of n nPlan, and in particular Damian Borowiec and Peter A. Zachares, for the invaluable facilitation of work that was completed whilst on internship with nP nPlan and access to A100 GPUs.
References
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolec- ular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Michael S Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. arXiv preprint 2200. 15571, 2022.
Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann. Conditional image generation with coore-based diffusion models. arXiv preprint arXiv:2111.19606 2021.
Heli Ben-Hamu, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, and Yaron Lipman. D-flow: Differentiating through flows for controlled generation. arXiv preprint arXio:2402. 14017, 2024.
Christopher M Bishop. Pallern recognilion and machine learning, volume 4. Springer, 2006.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using generative models. In International conference on machine learning, pp. 537-546. PMLR, 2017.
Gabriel Cardoso, Yazid Janati El Idrissi, Sylvain Le Corff, and Eric Moulines. Monte carlo guided diffusion for bayesian linear inverse problems, 2023.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is as easy as learning the score: thcory for diffusion models with minimal data assumptions. In International Conference on Learning Represenlations, 2023.
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffu- sion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2022a.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 35:25683 -26696, 2022b.
Hyungjin Chung, Jeongsol Kim, Sehui Kim, and Jong Chul Ye. Parallel diffusion models of operator and image for blind inverse problems. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 6059 6069, 2023.
Giannis Daras, Yuval Dagan, Alex Dimakis, et al. Score-guided intermediate level optimization: Fast langevin mixing for inverse problems. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022.
Alexander
Denker, Francisco Vargas, Shreyas Padby, Kieran Didi, Simon Mathis, Vincent Dutordoir, Ric- cardo Barbano, Emile Mathieu, Urszula Julia Komorowska, and Pietro Lio. Deft: Efficient finetuning of conditional diffusion models by learning the generalised I--raasform. arXio preprint arXiv:2406.00 017711 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-878 8794, 2021.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106 (496):1602 1614, 2011. doi: 1a.119//jaa.221Lmmm111 URL https://doi. 11110..11///aaa 11111/////...11........1 2011. tm11181 PMID: 22505788.
Berthy T Feng and Katherine L Bouman. Efficient bayesian computational imaging with a surrogate score- based prior. arXiv preprint arXiv:2309.019/ 01999, 2023.
Berthy T Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine L Bouman, and William T Freeman. Score-based diffusion models as principled priors for inverse imaging. arXi preprint arXi::334.11151, 2023.
Marc Anton Finzi, Aundhyan Boral, Andrew Gordon Wilson, Fei Sha, and Leonardo Zepeda-Núñez. User- defined event sampling and uncertainty quantification in diffusion models for physical dynamical systems. In International Conference on Machine Learning, pp. 10136 10152. PMLR, 2023.
Clark R. Givens and Rac Michael Shortt. A class of Wasserstein metrics for probability distributions. Michigon Mathematical Journal, (1(2):231 240, 1984. doi: 10.1307/mmj/1020000 //doi.org org/oo//1/00 10. 1307/mm3//029003002
URL https:
Jonathan Ho and Tim Salimans. Classifier-free diflusion guidance. arXiv preprinl arXiv:2207. 12598, 2022.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Floet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249 2281, 2022.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, (124))695-709, 2005. http::/jjmm httg:/aapprr/////yyaareeeeeooooe html
Ajil Jalal, Marius Arvinle, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advonces in Neural Information Processing Systems, 34:14938-14954, 2021.
Zahra Kadkhodaic and Ecro Simoncelli. Stochastic solutions for lincar inverse problems using the prior implicit in denoiser. Advances in Neural Information Processing Systems, 34:13242 13254, 2021.
loannis Karatzas and Steven Shreve. Brownian motion and stochastic calculus, volume 113. Springer Science & Business Media, 2012.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generalor architecture for generalive adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basod generative models. Advances in neural information processing systems, 35:26565 26577, 2022.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24174- -41444, 2024.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. Aduances in Newral Information Processing Systems, 34:21757-2179,, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593 23606, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra.
Bayesian imaging using plug play priors: when Langevin meets Tweedie. SIAM Journal on Imaging Sciences, 15(2):701-7311 2022.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.027: 2022.
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. arXiv preprint ar Xiw:2335.04399. 2023.
Chenlin Meng, Yang Song, Wenzhe Li, and Stefano Ermon. Estimating high order gradients the data
distribution by denoising. Advances in Neural Information Processing Systems, 0925359-25360 2021.
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. The Journal of Machine Learning Research, 21(1):5183-522 5244, 2020.
Kushagra Pandey, Ruihan Yang, and Stephan Mandt. Fast samplers for inverse problems in iterative refine- ment models. arXiv preprint arXiv:2405.11 1673, 2024.
Jakiw Pidstrigach, Youssef Marzouk, Sebastian Reich, and Sven Wang. Infinite-dimensionaal diffusion models for function spaces. arXiv preprint arXiw:2302.10130, 2023.
Ashwini Pokle, Matthew J Muckley, Ricky TQ Chen, and Brian Karrer. Training-fres linear image inversion via flows. arXiv preprint arXim:2310.0444 04432, 2023.
Herbert Robbins. An Empirical Bayes Approach to Statistics, pp. 388-38 394. Springer New York, New York, NY, 1992. ISBN 978 1-681--191-- doi: 10.1007/998-11461222 10.
26. URL https:///oi..co
14

---

Published in Transactions on Machine Learning Research (09/2024)
François Rozet, Gérôme Andry, François Lanusse, and Gilles Louppe. Learning diffusion priors from obser- vations by expectation maximization. orXiv preprint or arXiv:2405. 13712, 2024.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding, Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. 2021a. URL https: (loppp/wwr.er...effooooraaaaa/aa
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Psendoinverso-guided diffusion models for inverse problems. In International Conference Learning Representalions, 2023. URL https:/7
openreview.net/forurrre ggMMAMRKK
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with scorc-based generative models. arXiv preprint vrXim:2111.00000 2021b.
Tristan SW Stevens, Jean-Lue Robert, Faik C Yu, Jun Seob Shin, and Ruud JG van Sloun. Removing structured noise with diffusion models. arXiv preprint Xrxi2:2302. 05290, 2023.
Andrew M Stuart. Inverse problems: a bayesian perspective. Acta numerica, 19:451 559, 2010.
Albert Tarantola. Inverse problem theory and methods for model porameter estimation. SIAM, 2005.
Luhuan Wu, Brian Trippe, Christian Naesseth, David Blei, and John P Cunningham. Practical and asymp- totically exact conditional sampling in diffusion models. Advances in Newral Information Processing Sys- tems, 36, 2024.
A Proofs for Section 3
A.1 Proof of Proposition
In order to prove this result, adapt Theorem of Meng et al. (2021 We write the proof for generic exponential family which can be adapted to our case easily. Let consider
(x) = (xx: VaiXo, Ia
and write (x.) = (x)o(x//x))) We are interested in finding the moments of the posterior
(o)x(x)(x;) = (x (x)))(
px(x))
Let us redefine the right handside this Bayes' rule using the exponential family paramoterisation of the Gaussian Pejo (x)
p(no|x:)) (x (7))
(x())
where 7o Xova,/u and
Pxlo(x//m)) no x xi ((0)) nd
15

---

Published in Transactions on Machine Learning Research (09/2024)
where
g(x:) (dd
-1/2 e-xxxxxxiii x1/20
Now let
X(xi) = log Px(
g(xx))
we can rewrite
x()))x)) = eo - x ( )(xx)) qo (70).
It is then easy to show that the moments of nuu given by (Meng et al. 2021
Elmo xx] = (x(x),
E 1o [x1 = JA(x))
E 7
T Xx.] -x((x)) + (xxx))
where JA is the Jacobian of X w.r.t. X₁ and S is the Hessian of w.r.t. Xt. Recall now that no xo /o/o
and
x x: +
A(x() = (x:) +
C
2v:
X1
Jx(x,) Xx, (x
1
Sx(x.) = V log
- Ia.
1t
This implies that
E xx =
(xi + ue Vx, log (x::))
var
which proves the Tweedie's formula for the mean. For the covariance, note that
Cov(nooxx)) = E(nond [x.] - E/yol [xooo[///xx]]
= Vo + Id.-
Since
(ov(nolx:): 04 Cov(xo|X:),
we conclude
Cov(xolx:) ai
01 (Id, + 24 2 log (x1)),
which concludes the proof.
B Proofs for Section
B.1 Proof of Proposition
II Pdita is Gaussian, then the full process x, is Gaussian process. In particular, (Xo, x.) are jointly Gaussian:
X Xx ([E[xx/Xx) V [[xo|x.)).
where V denotes the covariance. However, the right-hand side preciscly the approximation we make, due Proposition Therefore, the approximation we make in Equation [1 correct. Adding this the learned drift (x,, V (x,,, get expression for V log yx, by Equation 3
16

---

Published in Transactions on Machine Learning Research (09/2024)
B.2 Proof of Theorem 1
We first introduce three SDEs.
Conditioned SDEs
The first pair of SDEs is given by an OU process, but started in the conditional
distribution:
1
dxf 2 xt dw,, x5~ p6 == Pdata (xoly).
(18)
We denote the marginals of x by P to differentiate them from the marginals Ps of the OU process started in the correct distribution, defined in Equation 2 Note that pi = pr(xlly). The time reversal 2, = XT- then satisfies the SDE
dz: z + Vz, 1 + dw:
2
(19)
2 dt + Vz, (z) + Vz, log Porgy/r-((y/22)) + dw
20 ~PT,
where we used Equation 3
Solutions to the above reverse SDE will sample target measure p6 Pdata Pdata(Xoly) at final time. Therefore, want to study how our algorithm approximates solutions of x².
Intermediate Gaussian SDE: Instead of bounding the distance of solutions to the above conditioned reverse SDE to our algorithm, we will instead introduce an intermediate process, which we will later use in a triangle inequality. The process will be a Gaussian process. Therefore, we denote it with a superscript G.
The process x will be defined analogous to Equation but assuming that is started ((xx; No, En) instead of Pdata- Since the forward SDE in Equation 2 linear, all of the marginals of x which we denote by Pt, will also be Gaussian.
Again, we define conditioned version of x, called Gc analogous Equation 18 i.e. xee will be distributed x conditioned y y, ie. px(xxly ly y). Since we have linear observation model, (xoly y) is still Gaussian, and therefore GGe will also be Gaussian process. Its reverse SDE 2 is defined through Equation 19 just that every appearance of will also have a superscript G,
1 dz, 7 52, + Vn, log b + Vz, logrr-ri [z) + dw, 20 pr,
(20)
Algorithm SDE: Finally, we define a third reverse SDE, which is the SDE that we are discretizing when implementing our algorithm. It is given by
dz = zidt + Vz, (z))r + (z)) dw. Zo ~ pi
(21)
where
f((z)) Eix 2, H (HV[x,)x; 24 H + - HE x; 21))
(22)
and V denotes the conditional covariance. Except for approximation errors due to time discretization and the initial conditions, this is the SDE we are sampling from in our algorithm. We denote the marginal of z by de- Now we are ready to write our proof.
Proof of Theorem 1 We have that Pasta = =
11 -r S llp - po "''llr + G. gr l|
We will bound the sccond term using Pinsker's incquality get KL term on the path space, as done Chen al. 2023). The proof then consists of bounding the first total variation term and the resulting KL term.
17

---

Published in Transactions on Machine Learning Research (09/2024)
Bounding the first term: The first term is the total variation distance between po(xuly y) and Po (x = y). Now,
Px soly v) Pyjoly = 8|xo zolPo = xo)
Py 1/)
pyjoly y/xo zo)Po (xo x0) po px 00) py(y = y)
pg ly = y)
Pu (xg =0) Py(y = y) Py py =1)
- zoly y) exp 4 (zo))
pyly =y)
where we used that the conditional distribution of given xo does not depend on the distribution of Xo- For the last term we get
/ Pyjoly = 8/x0 xx) = xo)dro
Py(y = y)
/ Pyjo(y (y = yixo =x) xo) ex)(((((()))
NNp(y = =))
with N E M1. Therefore,
Ip6 1 S /
(20)
pb
1pp G.e (xo) S M² - 1,
Pn
where we used that M - is always greater or equal to 1 - 37 since M 2 1.
Bounding the second term: For the socond term we get that po G.C is the final marginal of Equation 20 while @r is the final marginal of Equation 21 We now use Pinsker's inequality,
IIpo G.c 4Tll v (
We denote the path measures induced by Equation 20 and Equation 21 by pG.c and Q respectively. They have final time marginals G. and qr. Therefore, we can bound the KL-Divergence of the marginals p and or by the KI--Divergence on the full path space:
KL G.e (4r) S KI((PG
We will assume that we can apply Girsanovs theorem and show later that this is justified. Using Girsanovs thcorem, we can evaluate the Radon-Nikodym derivative apc.e on the path space and therefore calculate the KL-divergence:
diG,e
KLC =Epo. [log
dQ
= Epc.
Vlogp. + (x)) ((( - log (x, )dw!
1/" [1llogg ST-
+ +Epo.o
v log 1og (2rx) 201
G.c
= -1" E (l +
Voo G.C (x))) jdd
18

---

Published in Transactions on Machine Learning Research (09/2024)
where the term on the second line drops out because stochastic integrals have expectation 0. Now we have the drift of the Gaussian SDE given by
V logpt""ax
=V logg (x²) + v logp (y/x; = x1)
= -Vlog ( (1a)
(23)
+ vv, E|x X G H
(HV|x) H oa
H x xz))
=V (x)) + fz)))
see Equation 48
while the drift of the algorithm SDE is given by
V Vogge(()) + Se(2)),
(24)
where fa is given by Equation 22
We see that the difference in the SDE drifts mainly consists of differ- ences between conditional moments of Pt and pf", as well as the derivatives of the conditional expectations. Therefore, the main difficulty of the proof is to bound these.
The density of the conditional distribution of Poje given by
(o)e(2o((11 po.e( v4)
pa. (T. 2z) exp(((xo)) Poje (eol=))) exp(((x0))
(r(e)) =p exp()(200) - (xx))),
p(xx)
p(x2))
(25)
where
Pr(x,) (xxx)(((xx)))x xz].
exp((,(<x))
(26)
pi (x()
In the last equality, we used that
dP
(2)0.1) = exp(
JPG
where we denoted by the path mcasure induced by Equation 18 Therefore, also their marginals pt P and Pt P. have relative densities, which are given by integrating out the density to time l, as did in
Equation 26
By assumption exp($) is bounded from above and below by M and 1/M respectively, and by Equation 26
the same holds for exp( Therefore, by Equation 25 Pejo is absolutely continuous with respect to Pijo
with a density that is bounded from above and below by M and 1/M² respectively. We now obtain
E[xolxx = xz] - |0[x01 xz]1
-Iʃ no (p (xn)) - p (xo (o )dro = =Iʃ mup / (xx) - +(x,)) - 1)d20
5(M² 1 = x1
The same holds for every entry of the covariance matrix. We denote
Vij Vixo|xelis.
Ve = V1x01x91 = Nej.
VV vol s (M² - 1)NNs,
Ny = =1120,000011 [x{]]
Now we get that
1/2
I
V F
Σ (Vg vg922/
(M² (MP-1)Il 1) ma
max (Ny).
19

---

Published in Transactions on Machine Learning Research (09/2024)
We can bound Ni by
N1 ((( (xx Tr(V[x0xx =x))).
(27)
The latter term does not actually depend on 2t, but only on t. Using the formulas in Equation 44 we see that it can be bounded independently of (only depending on my and Eo). Therefore, we can put it into a constant. We get that
I - volle S (M² -1)l
where
o Cb,
with a constant C' only depending on mo, Co. H, By and the observation Now let v be a vector with ||a|| = 1. It holds that
T Vij 0
(u (TT(pn) (ro F|xo,x, (x) exp( - (F(()))
M 11 T v N / (oT (B = xz) E(xolx: po/))))0000001111/0111))
1
(28)
20 Vg
M
with 2 1/M. Since and Vc are positive semidefinite, symmetric matrices, this implies that all eigen- values of are bounded by the lowest eigenvalue of VG times 1/M. We define
Tj = H. H + L
(29)
T2 HVH + +oy
(30)
We then need to bound
= [ - +|
We start with
T - Tellop < 2 - lo $ V - -V-Ipe S (M² 1) Ir.
We also used the equivalence of the Frobenius and operator norm here. Due to eigenvalue bound (Equation 28 on V, we also get an analogous bound on V + oyIa, Therefore,
|T -T & (M² 1
M) s (M² 1) + 1) S 31
where we used that the operator norm the inverse, the inverse of the operator norm and the equivalence of the operator norm to the Frobenius norm.
Finally, need to bound
Vx, rop(zol xx )x
f To Vn, exp(((2o) - (fxx)) )dro
(31)
f so exp ( x0) 2 xz) ) (zol 2 daoo + V., (xxx) exp(#(20) - (xoll))o
/ exp((((z) ((z;)) 2 (Hol + V (2x 2z).
20

---

Published in Transactions on Machine Learning Research (09/2024)
Also
|f zo exp((((1)) ((x)) (. (zolzz)dzo Vz, B(x8 mm)
/ o. (z0l (exp(((xx)) -(x))) 1)220
S(M² 1) / p (=ook
V (oo d2z
S(M² 1)( / (
(32)
|d2
S(M² 1)( / ( 0/c (2o - m
)xx)) )
S(M2 1)( / (2oolz))) - mot) Eoj (2o moj)d z | 1/ <1aae|(MM=1)
here we used that we can upper bound the operator norm of a positive semidefinite matrix by its trace from the third to the fourth line. We denoted by moj and Egjs the mean and covariance of p (x8 x 24). From the second to last to the last line we used that the møja depends on Ie linearly, and the magnitude of the linear dependence can be bounded uniformly in t (sec Equation 44h. The integral in the last line is the variance of a standard normal random variable, which evaluates to dg. Furthermore,
v0,(x;) = (x x = 21] S L.
see for example the Proof of Theorem 1 in Pidstrigach et al. 2023) Therefore,
E[x G x = xz] V., Eixo x = xi]
2z (M2 1) + L | Xo Xx 2z]ll
2 (M² 1) + L( | N x 21] E[(x xx x1]1 + |B| x xxlll)
2 IMM2 - 1) + L(M² 1 [|æell) = || [ - 1 + L) + (M² - 1)L =: B
Furthermore, from Equation 31 we see that
V E|xo = æx]ll و [M
- 1) 2 ||0 II) = B.d
too. We now subtract the full drifts in Equation 23 and Equation 24 and arrive at
| fxx,) - fz.)ll
Brg ( - HTE x x x:))ll
+ Br, | T looplyy - H1E G G = æx]1
+ B., | | | xxy xx{ 21] HEE X2: 2ell
SBn + Bz, (M3 1) + Bz, M(M² -1)
<Bn (M3 - 1) = =(M ()( +L) + +MM 1)
Furthermore, we get that
pi (x+)
|V (xx) - V log V log
< V. (x())) $ L.
pr(x,)
Plugging in Equation 23 we get that
V (z,) + f((x,) - Pi (x,)) S (M5 (105-1)(0 1)((= + + (M - 1) + L.
Using this expression, we see that the drift change is linear in | We now want to apply an iterated version of Novikov's condition to show that our application of Girsanov's Theorem is justified. To that end,
21

---

Published in Transactions on Machine Learning Research (09/2024)
follow the argument see that
Karatzas & Shreve 2012
Corollary 3.5.16). We repeat it here for completeness. We
fi12 exp
Epc.c
(2
(Vlogp(( + f(())) V log Pt (x, (xt))]
fitz
<poo
exp
c 1+ [
|,0 1
Since the expectation is regarding Gaussian random variable 2i, we can make it finite as long as we pick A; f441 - t; small enough. By setting to 0 and > can show equivalence on [to, f1]- We can then iterate this procedure, to get equivalence on [41 (2] and so on. Since the lowest and highest eigenvalues of Σ, are bounded from below and above respectively, and is also bounded, the A, can be bounded from below. Therefore, we get equivalence on [0, this way in at most [T/A] steps.
Furthermore, we see that the likelihood Ki-divergence can bounded by
1 / V log (x.) + f(())) V Vlogpi G.C. (x))
Epa.c
2
2 2 ST((M - 1))++-1))1+))) + (M - 1)L D) T(M 1))
where we used that the mean and covariance of the Gaussian process 2, (under (PC.)) can be bounded by a constant only depending on y, Σg and my Taking the square root proves our theorem.
C Variance Exploding SDE (time-rescaled Brownian motion)
A second example of an SDE used frequently in denoising-diffusion is the Variance Exploding (Song et al. 2020 or timo-rescaled Brownian motion,
dx. = V du(t)
dw., Xo ~ Po = Pdata
(33)
dt
where (Wa) is Brownian motion and R20 - Rz0 is increasing function where c(0) = 0. The transition kernel of the time-rescaled Brownian motion is
Pxo(Xx(Xx)) N(xx, I.Ta))
(34)
denoting v(t) vt. The signal Xy is sampled from a second SDE, the reverse process, given forward time
de dw,, XT ~ Pr.
dx; Vx, lx:
V de
(35)
Tweedie's formula
In this section, we give an alternative derivation of the Tweedie's formula for VP-SDEs and VE-SDEs.
Tweedie's formula (Robbins 1992 Efron 2011) gives the minimum mean squared error (MMSE) estimator of xo|x; when X2 (gg is Gaussian and the score Vx, log:((x;) is available. Tweedie's 's formula applied to the time-rescaled SDEs is given below.
D.1 Variance Preserving SDE (time-rescaled Ornstein-Uhlenber process)
Using the Variance Preserving SDE or time-rescaled Ornstein-Uhleniee process Equation 2 given the random variable
Xp = Vo ~ Pves
U1
22

---

Published in Transactions on Machine Learning Research (09/2024)
where, now define Pvo (vo) the probability density of the random variable vo evaluated at the realization Vo, then the random variable
X = v, ~ NV Is,):
has marginal probability density evaluated at the realization v, is the convolution of the random variable with the Gaussian kernel
Pv. (vi) f Φ(v, - Po)poo
Then, Tweedie's formula (Robbins
1992:
Ffron
2011 gives
EvorPvolve (vo) Vt Vv, Prg (v:)
dx: From continuous change of random variables, Pv. (ve) = Px, (z) Px, (x+) + log /uz, which yields
Px, (xi) giving (v.))
dv.
Vv, lo, = V Vx, log (x.
Now, substituting into Tweedie's formula,
E Poo
[vo]
+ uv Vx, lxggx((x))
Voz
giving
1
E Xx [x0]
X2
+
Vxi Pxg (xe).
Vaz
Vaz
The covariance is given as Robbins
1992
Efron
2011
V Voo~Poo volvv]== = Vv, Vv, log (V+), where
V., V logpv.
V., (v Uy V. lxg
(36)
t. V2 x,
log (xx)
(37)
(38)
and
V Vo Vor-Poolve [vol -Lde + 21 V2. log Px. (xi)
giving
V1 (Idt, Ur Xx, log px, (xx)).
V Pxomee
[xo]
at
In practice, we make the approximation Vx, log Pxe (xx) so so 1), giving
1
E Exo~Paolx
[Xo] maje
+
(x.,)
Va,
Vaa
and
Ut (Ia, + ve Vx, f)),
Vx Vxp~Pxooix [xo]
az
which yields a Gaussian approximation that matches the first and second moments,
Pxolx, (Xu (x))xx) N Moorr at
Ut (Id, + 2 Vx, t))
(39)
=
moje:
Vx, Mo
(40)
fas
The matrix Et (I,, + Ug Xx, Sx(,,)) t)) needs be inverted to calculate the log likelihood, and therefore must be both symmetric and positive definite for all time, which puts the requirement that 50 t) can be written the negative gradient of potential, however it is only approximated as such so (x,, A V log Px, (xx).
23

---

Published in Transactions on Machine Learning Research (09/2024)
D.2 Variance Exploding SDE (time-rescaled Brownian motion)
Using the Variance Exploding SDE (Song el al. 2020 time-rescaled Brownian molion, given the random variable
Vo
Puo
(41)
similar calculation in D.1 gives
Xx + 2ee Vx, log Pxi (xx); B Pxoie [xc]]
and
Pxol xo] = Ve(Ta, + 12 Vx, log Px, (x:)) Again, in practice, we make the approximation Vx, log Px. (xx) so (x t), giving
Exo xe [xo] z Maja X, 0150 (x),
and
V Xxo-Pa [xo] R (To, + 02 So,,,
which yields a Gaussian approximation that matches the first and second moments,
Pxo |xc, (xolxx) x N (mojs Ue (Ia, + Vx, so (xx,
(42)
N (mals 21 Vx, moje).
(43)
E
Algorithmic details and numerics
The code for all of the experiments and instructions to run them are available github.com/bbl www.oo/m//////o//ooo////... /bb51///mm////aaa tmpdjax and ggthub.com/bb515,
Algorithm TMPD-D (Ancestral sampling, VP)
input y, Fy
XN -O, I
for N I,..., do
mmj+ + 1
van (xn Un Un tx))
my + moje + van
Vx., mo H (H Van
Vx, Ho H + - 'yy- HHHo))
Zn N(O.Ie,)
On ((1 (1-an-1)a) 1/m/ an
11-3n(1-an) + I- mjje + kn
van
Xn-1- 1-0n
end for output X0
E.1
Computational Complexity
Let N is the number of noise scales, and let dy is the dimensions of the observation, dy are the dimension of the image and T3 is the time complexity of evaluating the score network. Computing a vector-Jacobian product has time complexity T.. Then, the time complexity of TMPD (not including fast matrix-vector products) is (N + T.,d. + +T, +T)) IIGM comes at a smaller computational cost due to needing only O(1) neetor-jacobian-proddee comes at the same complexity as I1GDM.
instead of O(dy), resulting in time complexity of O(N(T, + T.)). DPS
Whereas TMPD requires calculating the Jacobian which has memory complexity of atleast Oddddy). This is too large for high resolution image problems where the dimension of the observation is large. Tn comparison,
24

---

Published in Transactions on Machine Learning Research (09/2024)
the memory complexity of IIGDM depends on the observation operator, but for the class of problems that are explored in [Sog et al. (2023 is (dd)). DPS comes at the same memory complexity as IIGDM
As mentioned in the text, we make the following approximation
DTMPD Vx, loggy)(()xxx) Vx, Hoj (H Ur
diag(( Xx, )o + +o]Ta, Var
Hmoj))
Whilst this approximation does not require a linear solve, taking out the O(Ng)) time complexity term, we would still like to take out the O(NT,d,)) time complexity and O(dxdy) memory complexity term from caleulating and storing the Jacobian, respectively, since this is too large for solving high resolution image applications. A further approximation approximates the diagonal of the Jacobian by the row sum of the Jacobian which only requires one vector-jacobian product and brings the memory and time complexity of DTMPD down to that of IIGDM and DPS. We use this approximation in the image experiments and find that in practice it is only (1.5 = 0.1) x slower than IIGDM and DPS across all of our experiments. The row sum will be a good approximation of the diagonal when the Jacobian is approximately diagonal, which happens when there is small linear correlation between observation pixels, which we found to work well for super-resolution and impainting. In impainting, we have further improved the accuracy of the rowsum by instead of calculating the vector-Jacobian product evaluated at H1, masking out values of the vector in the vector-Jacobian product using the inpainting observation operator since they won't contribute to the diagonal values of the of the variance.
However, heuristics can be circumvented by noting in the definition Equation 11 does not require the inverse but rather solving a system of linear equations with right hand side y - H A very recent work (Rozet et al. 2024), uses the natural choice of the conjugate gradient (CG) method to solve this linear system noting that HC + of is symmetric positive definite (SPD) and is therefore compatible with the conjugate gradient (CG) method. The CG method is iterative algorithm to solve linear systems of the form Mo = b where SPD matrix M and vector b are known. Importantly, the CG method only requires implicit access to M through an operator that performs the malrix-vector product Mo given a vector v. In our case, the linear system to solve is y Hmooj = (HCoj + oz H o
Vx, mo HHT v + ogIv. The vector-jacobian product v ""VV x, maje can be cheaply evaluated. In practice, there is no restriction on the score network to be the gradient of a potential, and the gradient of the score need not be SPD. Due to this and numerical errors, RRzet al. (2024) observed that CG method becomes unstable after large number of iterations and fails to reach exact solution. Fortunately, the authors find that using very few iterations (1 to 3) of the CG method as part of the computation of the posterior score approximation leads to significant improvements over using heuristics for the covariance. Rozet et al. (2024) have successfully applied the CG method lo non-sparse H, such as accelerated MRI (where H is the composition of the Fourier transform and frequency subsampling).
us
E.2
Gaussian
When the data distribution (x() is (multivariate) Gaussian, then the reverse SDE is linear SDE and we can calculate all of the terms needed to sample from the target posterior using diffusion explicitly. Moreover, we can sample from the target posterior using direct or implicit method such as Cholesky decomposition. In this simple example, we compare samples from the direct method to various conditional diffusion methods (TMPD, TIGM,, DPM), by plotting a sample estimate of the I,2 Wasserstein distance between the sample and the target Gaussian measures (Givens & Shortt, 1984) by using the analytical mean and covariance of the target distribution and empirical estimate of the mean and covariance of the sample distribution. To generate px(),),
use equally spaced grid of vectors u, E [-5.0, 5.0)²
for i e 1,2, .... 1,,, 32?, pick Matern u, + uu u2))) exp (-v3 u, - uy)) which
5/2 kernel for the covariance function k(u; u,) = (1 + v5 defines the prior po (Xo) = (m Co) covariance C E IR 10002x10024 where Co.3 = k(u, uy) and we define the mean as a zero vector mg = 0 E 11024 To compute analytically the distribution of Poly Px (xoly), we sample y = Hx, + z, z ~ NN0, Ia) ) and use the standard Gaussian formula to calculate the mcan and covariance of xoly which in this case are complete description of Poly (xely). The I22 Wasserstein estimate is plotted over increasing sample size N E [9, 1500] and for ay = 0.1 in Figure]
25

---

Published in Transactions on Machine Learning Research (09/2024)
We provide an illustration of the mean and uncertainty captured by the diffusion samples using 1500 samples of each diffusion model to produce a Monte-Carle estimate of the mean and diagonal variance vector, and compare these to the exact mean and diagonal variance.
Below, we provide a calculation comparing the exact diffusion posterior for the linear diffusion posterior sde to the approximations used in IGDM (Song et al.) 2023) and TMPD. Tet us assume that the target distribution be known (x() (mmo Co). Then, via Bayes' rule,
log Pojr (Xo x+) = Pogo (x, (x0) + log + constant
1
2 (x: vaixo) (vcIo_) (x v)ix)
1
2 (xo mo) TC. (xo -mo + constant
(vxx)) (vrIs.) ( (Vaxx)) + x, (velo,) (
2
1
Xo CT Xo + ma Cp Xg + constant.
2
1
(xo mi) TE (xu my),
2
So
Poje(Xu xx) Nmm Ee)
(44)
where
E. = ((0) -(0t) 04
IIa Ia,))=1
Cp
(45)
and
(vor,
m; = Eg
X1 + -C-imm m0
11
We also have that,
pe(x,) = =N(a,moo C+)
where C₁ = ar Co + vela, and thus
log (x,,)(x;)= = --((-- 7(x, moyas) T C
moya.).
From
Pxie(y(x1)= = f
N (Hm, HΣ,H + +o
we have that
(ogr<(y(x)) -2 Hm:) T + L ty Hm.).
1
The posterior score can be calculated (see Cardoso et al. (2023) from applying Bayes' theorem,
log
(xeelyyyXle ly) logpr(x)) - log (y)xx) + Constant,
and taking gradients with respoct the statc xe gives
Vx, Prg =V lx, + Vx, log (yg
(46)
= Ca,Co + 2z Ia,) ) -1(( movay
(47)
+ Val MT. (HE,HT + +oG HΣ, Vat Xr + Cp' mm'm)))
(48)
Setting mo = in 48
gives,
Xx log Paly (xxly) = Ca,Co + vzlaa))
(49)
Vai
Var HE
Σ H H -
(50)
1/
2
26

---

Published in Transactions on Machine Learning Research (09/2024)
E.2.1 Comparison of the exact posterior to the approximation made in Seeudo-Invere--Guidaa
We aim to compare
(50 to the approximation used in IIGDM (Song et al. 2023
Po (xolxe) R (moj, of (1,))
where moje = X + u xx, (x,), which is the minimum mean squared error (MMSE) estimate of x Xo|x; and re is chosen empirically. This gives Pyje(v(x) = (Hmoj r HH + ogJa,), which in turn gives
Vx, x, log (x)ly) = xx. (x())
(51)
vx mo H (r HH + (y Hmoo)
(52)
which is computationally tractable in the standard diffusion model setting where the score nonlinear and approximated Vx, log)((x;) sx(,, Substituting in the known score and again setting mg- 0 to compare to the linear case 50 gives,
Xx, log ( (xll) ~V x, log Pr (x,)
(53)
Xx maji H (r HH +o Hmoo)
(54)
= Ca,Co + vrld,)) -
(55)
(Id, ve Co t H ( (HH +oīTa
(56)
Vai
(y
H - Co + vs II, I
(57)
Vau
comparing terms, IIGDM is making the approximation
Vx moj+
1
Ida
T1
(ox Co + vxIa,))
y Σ. 14
Vas
Var
Note that since,
Va
y
(C
+ - Id, (=1 (--))~11- --1--
(58)
1/
1
Co) + (v: (
(59)
Vi
1
(vll - 0z (a + velo,)
Woodbury identily
(60)
vn vrTaa)
(61)
0
V. mo
(62)
this is exact. The other approximation ITGD is making is rII E,, and note that this approximation is accurate with r? Us as - 0 but inaccurate as t + 1. But the exact linear SDE can be recovered by instead using TMPD,
V log Ptly (xil) =Vx, log (x),)
(63)
H Vt Vx. mg .. + ozI1,, Hmo
+ Mo
(64)
Var
=vx, (x(x))
(65)
Vau
Vai HE,x)
E, (HE,H oo
(66)
21
27

---

Published in Transactions on Machine Learning Research (09/2024)
Table 3: Sliced Wasserstein for the GMM example using the reverse VP-SDEs discretized with Euler-
Maruyama.
Cy.
dx
DEMPD
TIGHS
TXPPH
PUNPPI
HIGH
DPS
TPPS
1.4
0.6
1.5
14
14
3M = 2.1
15 0.3
0.
0.3
0.5
0.3 D.3
0.3
3.2
0.
0.2
0.3
04
0.1
6.2
0.3
80
23
03
0.8
4.7
1.
11
0.
11
E.3 GMM
For a given dimension dx, we consider Po a mixture of 25 Gaussian random variables. The components have mean Ms.j == (8i, 85, .... 8i, 8j) E Rd for(i, -2 0, and unit variance. have set the associated unnormalized weights Wa,j 1.0. We have set o3 10-1
Note that (x(),) P (x))xxx)) As (xo) is mixture of Gaussians, Px(x,) is also mixture of Gaussians with means youlis and unitary variances. Therefore, using automatic differentiation libraries, can calculate Vx, (o We chose fmax 500.0 and 8min 0.1. We use 1000 timesteps for the dine-discretizatio.. For the pair of dimensions and chosen observation noise standard deviation (dx, dy Ty) the measurement model (y. H) is drawn as follows:
H: We first draw H ~ NNOs,, x xd,) and compute the SVD decomposition of USVT.
Then, we sample for (i.j) E -2, -1,0, 1,22, st, according to a uniform in [0, 1]. Finally, we set H U.daag(a.n..ee----....
We then draw x, ~ and set = Hr. + where 2 ~ NO0, o(,,0 la,,
Once we have drawn both x, and (y, H, oy), the posterior can be exactly calculated using Bayes formula and gives a mixture of Gaussians with mixture components Cij and associated weights Wi.j.
4. : y/o pis), E), : Hur,, oL + HH"),
(67)
(68)
where E = (Ld -1
Euler-Maruyama solver To compare the posterior distribution estimated by each algorithm with the target posterior distribution, SDEs defined by the TMPD, DTMPD, Song et al. (2023 Chung et al. (022aa) approxinations obtained using 1000 Fuler-Maruyana time-steps with 1000 samples of the true posterior distribution. Table indicates the Central Limit Theorem (CLT) 95% confidence intervals obtained by considering 20 randomly selected measurement models (H) for each setting (dz, dx, G.).
10* slices for the SW distance and compare 1000 samples of the continuous
DDPM Table compares 1000 samples of TMPD-D, IIGM-DD and DPS-D which are obtained using 1000 denoising steps and the extended version of Table We follow Cardoso et al. (2023) and compute the sliced Wasserstein distance using Wasserstein-1 distance. Figure 2 shows the first two dimensions of the estimated posterior distributions corresponding to the configurations (80, and (800, 1) from Table for one of the randomly generated measurement model (H). These illustrations give us insight into the behaviour of the algorithrms and their accuracy in estimating the posterior distribution. We observe that TMPD-D (and the Mallya-aa method TMPD) is the only method that covers the modes of the posterior distribution. Finally, dircct comparison to Cardoso et al. (2023) using their original experimental setup is shown Table 5 which shows competitive performance for posterior sampling compared Sequential Monte- Konto-Carlo, exact sampling method.
28

---

Published in Transactions on Machine Learning Research (09/2024)
Table 4: Sliced Wasserstein for the GMM example using VP DDPM.
Oy
Cy
4x
DUMPIR
IIDDAD
DFS-D
DEMPER
RIGOADD
DFS-D
DYMPISD
RIGOMDD
DES-O
16
1 6.5
1-5
0.6
38
63
5:7
2 1.5
11103 0.3
18 2.0.T
3.2 12200 0.8
3.7
1.
03
0.3
0.8 2.62
15 15:3.:
5-2
1-3
0.7
+
0.3
3.8
1.6
2.1
=
1.8
1.5
2 0.3
277
1.1
=
1.5
0.5
0.2
1,7 2 0.8
8.6
1.2
4:
0.3
0.4
2:00 0.2
=
0.7
2
2
0.5
2 0.3
=
0.8 0.6
26400
0.2
0.0 2 6.5
4 0.3
2.5
6.0
80
80
32
0.7
111.0
Table 5: Comparison to Cardoso et al. (023 using their original experimental sctup.
TMPD-D and
DTMPD-D use 1000 steps of DDPM.
d
d
MCGDIFF
TMPD-D
DTMPD-D
8
1.43 + 0.55
1.83 + 0.5
1.82 + 0.5
8
2
0.49 + 0.24
0.95 + 0.3
2.27 1 0.9
8
0.38 = 0.25
0.61:000
0.72 + 0.4
80
1.39 + 0.45
2.81 0.8
2.81 + 0.8
80
0.67 = 0.24
1.14 0.4
2.82 + 0.9
80
0.28 + 0.14
0.95 + 0.5
0.95 + 0.5
800
1
2.40 = 1.00
2.96 + 0.6
2.96 + 0.6
800
2
1.31
+ 0.60
1.60 + 0.5
3.07
+ 1.1
800
4
0.47
+ 0.19
0.60 + 0.2
0.84 + 0.5
E.4 Inpainting and super-resolution
Since the DPS-D method was derived and tuned specifically for VP-SDE, we look the VP-SDE experiments the same score network for each method and the same sampling or discretization numerical method. All methods are discretized using 1000 denoising steps. For the Markov chain methods we use DDPM and for the SDE methods we use an Euler-Maruyama discretization. In contrast to DPS(-D) HIGM((-D), we observe the robustness of our method across both SDEs and inpainting and super-resolution observation maps.
in Section E.4.1 separately from the VE-SDE experiments in Section B.4. In each comparison,
E.4.1 VP-SDE
Imagenet 256x256 For VP-SDE, our Imagenet 256 x56 experiment compares SNIPS (awar et al. 2021 diffusion methods (DDPM) DTMPD-D, DPS-D and IIGDM-D, and results are shown Table
FFHQ 256 256 For VP-SDE, our FFHQ 256 256 experiment compares diffusion methods (DDPM) DTMPD-D to DPS-D and IIGM-DD, and results are shown in Table 7 Fig 3 Fig and Fig 5 are a visual summary Table 7 plotting the LPIPS, SSIM and FID metrics against increasing noise for different obsorvation maps. Some uncurated samples that were used to generate Table 7 are shown in Fig G and We observe that all methods can successfully produce high quality reconstructions in the low noise regime
Table 6: Comparison to SNIPS. 4x noiseless and noisy (images have additive noise of ay = 0.05) super- resolution results on ImageNet 1K (256 x 256).
4 x noiseless super-resolution
x noisy super-resolution
Method
PSNRT
SSIMT
KIDI
PSNR1
SSIM
KID
SNIPS
17.6
0.22
35.2
16.3
0.14
67.8
I1GDM-D
26.0
0.75
1.30
20.7
0.43
17.8
DPS-D
25.1
0.69
4.08
23.4
0.63
3.09
DTMPD-D
26.0
0.75
1.10
23.1
0.62
8.85
29

---

Published in Transactions on Machine Learning Research (09/2024)
Table 7: Quantitative evaluation of solving linear inverse problems for VP DDPM with increasing noise on FFHQ 256x256-1 1k validation datasct.
PROBLEM
METHOD
FID
LPIPS
MSE +
PSNR 1
SSIM 1
By = 0.01
DTMPD-D
29.6
0.230
2000
1.60F-03
7.7t-01
28.4 1 19
0.784
0.016
4x 'RICURIC'
DPS-D
31.4
0.234
0.01
1.90F-03
= 1.00-u3
27.8 2.
0.776
=
SUPER-REE RESOLUTION
LIGDM-D
29.7
0.198
1.56c-03
8.72004
28.6 +
0.809
By = 0.05
DTMPD-D
32.7
0.304
0.003
2.90E-03
= 5.811-0
26.0 +:
0.699
= 0.000
4x 'BICUBIC'
DPS-D
29.3
0.280
0.071
2.90E-03
26.0 4
0.719
SUPER-RESS RESOLCTION
IIGM-DD
45.1
0.311
0.047
3.08E-03 03
5.79t-
25.7 +:
0.682
= 0.062
By 0.1
DTMPD-D
38.0
0.348
0.018
4.33F-03
= 4.72r-03
24.0 1:
0.635
=: 0.000
4x 'BICURIC'
DPS-D
30.9
0.318
0.031
4.06F-03
3-33--03
24.4 +:
0.664
0.000
RUPER-RESOLUTER
IIGDM-D
119.6
0.589
0.147
1.10E-02
5.50c.05
19.7 +
0.376
+ 0.056
By 0.2
DTMPD-D
45.6
0.401
7.03r-03
21.8 4:
0.559
4x 'BICUBIC'
DPS-D
38.1
0.385
0,051
7.59E-03 03
a.00c.0x
21.6 +
0.570
0.051
SUPER- RESOLUTION
IIGM-DD
295.7
0.780
5.65E-02
= 5.20c.03
12.5
0.117
By = 0.01
DTMPD-D
25.7
0.153
0.033
9.04F-03
7:8te-03
21.4 4
0.829
'BOX' MASK
DPS-D
31.5
0.175
0.038
7.79F-03
= 89m-00
22.4 4 3.
0.833
=: 0.033
INPAINTING
IIGDM-D
143.8
0.247
0.024
2.58E-02
7.11c.05
16.1 +
0.759
By = 0.05
DTMPD-D
27.0
0.240
0.008
9.68F-03
= 6.652-03
20.9 +
0.760
= 0.030
'BOX' MASK
DPS-D
30.7
0.228
8.28E-03
= 7.93c.03
22.0
0.782
INPAINTING
IIGMM-D
159.3
0.448
0.046
2.71E-02 02
7.000.03
15.8 +
0.504
0.050
By 0.1
DTMPD-D
29.6
0.292
0.94
1.06E-02
= 1.115-02
20.6 4: e.
0.709
= 0.053
'BOX' MASK INPAINTING
DPS-D
29.3
0.259
6.0m
8.03E-03
6,97v.03
22.0 4
0.746
0.051
IIGM-DD
165.7
0.539
0.053
2.84F-02
6.57m-03
15.6 4:
0.418
=: 0.185
By 0.2
DTMPD-D
33.8
0.346
1.13F-02
20.2
0.649
'BOX' MASK
DPS-D
34.9
0.337
= 0.006
8.01
= 4.22-033
21.5 4: 2.0
0.662
= 0.067
INPAINTING
HIGDM-D
199.7
0.590
0.009
3. 3.17E-02
L.00G-02
15.2
0.461
By = 0.01
DTMPD-D
24.6
0.090
0.033
4.87E-01
6.277-04
34.3 + 28
0.931
+ 0.030
'RANDOM' MASK
DPS-D
32.7
0.137
0.033
5.57E-04
= 5.0000.0L
33.4
4: 9.6
0.913
= 0.031
INPAINTING
HGDM-D
24.7
0.069
0.023
4.71E-04
5.188.01
34.5 4
0.940
0,026
By = 0.05
DTMPD-D
27.1
0.187
o 000
8.86F-04
= 4.8tr-01
30.9 1
0.851
0.033
'RANDOM' MASK
DPS-D
38.0
0.256
1.74F-03
28.1
0.794
INPAINTING
IIGDM-D
44.8
0.315
0,043
2.15E-03
6.05c.04
26.8 +
0.647
0.068
By :g=00 0.1
DTMPD-D
29.9
0.247
= 0.008
1.7/E-03
= =8.90 8.911-03
28.7 1:
0.787
0.049
'RANDOM' MASK INPAINTING
DPS-D
34.8
0.274
0.054
2.21E-03
L.17c.03
27.1 +
0.763
0.063
IIGM-DD
62.7
0.490
0.070
6.71E-03
= 1.90c-03
22.0 4:
0.410
= 0.120
By 0.2
DTMPD-D
34.6
0.306
2.71E-03
26.1 4
0.714
'RANDOM' MASK
DPS-D
35.9
0.317
0.030
3.59F-03
= 1.730-03
24.9
4: 1.9
0.701
= 0.073
INPAINTING
HIGDM-D
104.1
0.623
0.109
1.77k-02
8.026.03
18.6
0.316
but, visually, only DPS-D and DTMPD-D successfully produce high quality reconstructions the high noise regime,
Whereas DPS-D requires a hyperparameter search, there are hyperparameters for DTMPD-D D. For HIGM-
D we use the algorithm and default heperparaeeters as described in Song et al. (2023). For DPS-D we use the algorithm in the codebase provided by the authors Chung et al. (2022a) and we use their suggested hyperparameters for this task, such as step-size and using static-threshoddng (clipping the denoised image at each step to a range [-1, 1]) whereas DTMPD-D does not require any hyperparameter tuning or static- thresholding.
CIFAR10 64x64 Our CIFAR-10 64 experiment compares TMPD to DPS and IIGMM, and also com- pares diffusion methods (DDPM) in Table S and score-based methods (discretized with Fuler-Maruyama)) in Table 9 Various samples used to produce the figures in Tables and 9 are shown in Fig 10 and
We found the hyperparameter suggested for the DDIM method for the VP-SDE in Song et (0223 rf va/(a vz), which is calculated by assuming the data distribution (x(x)) is a standard normal and
30

---

Published in Transactions on Machine Learning Research (09/2024)
300
0.-
LPIPS
SSIM
05-
0.-
02-
noise std,
noise std, ay
Figure 3: 4x bicubic super-resolution FID vs LPIPS (left) and SSIM (right) using the VP-SDE on FFHQ-1K validation dalaset for increasing observation noise.
TMBDDD
TMPD-Q
230
TMPD-9 DeS-0 (ODDAO
TMPD-1
210
DPS.D
DP5.D
DCDDSDD
DIDODD
175
[kIDM-3
175
LPIPS
1755
SSIM
175
0.6
05
0.
10
16
noise std,
noise std, dy
Figure 4: 'box' mask inpainting FID LPIPS (left) and SSIM (right) using the VP-SDE FFHQ-1k validation dataset for increasing observation noise.
then calculating the posterior variance, to give unstable solutions for the algorithm given in Song et al. (2023) To make the method stable, we instead plug the IGDM posterior score approximation into a DDIM sampler in a similar way to Algorithm [ which, for the VPSDE, brings the algorithu IIGM-DD closer our method; we are then able to choose r? vz/(a, + v)) for both VP DDIM and VP-SDE methods.
For DDPM
the step-size constant suggested in Chung et al. (2022a) for inpainting. S <'/lly
Hmojell where we tune Ç' over the suggested range of (" E 1.0] in Chung et al. (2022a) across LPIPS, MSE, PSNR and SSIM, as shown in Fig. 9 for each inverse problem (each line in the Tables and
E.4.2 VE-SDE
FFHQ 256x 256 For VE-SDE, our FFHQ 256 experiment compares diffusion methods (DDPM) DTMPD-D to DPS-D and IIGD-DD, and results are shown in Table [0] Fig [2 and Fig 13 are a visual summary of Table 10 plotting the LPIPS, SSIM and FID metrics against increasing noise. Some uncurated samples that were used to generate Table I0 are shown in Fig 14 and 15 Since the DPS-D and PIGDM algorithms were developed for the VP-SDE, the method is not performant for the VE-SDE, and the IGGDM
31

---

Published in Transactions on Machine Learning Research (09/2024)
Table 8: Noisy observation inpainting and super-resolution for VP DDPM on CIFAR-10 1K validation set.
PROBLEM
МЕTHOD
FID
LPIPS +
MSE +
PSNR
SSIM
By 0.07
DTMPD-D
33.7
0.090
0.018
0.007
0051
23.8 3
0.784
0.033
'BOX' MASK INPAINTING
DPS-D
31.5
0.064
0.004
25.8
0.841
IGDM-D
37.1
0.316
0.012
19.9
0.546
Ty = 0.01
DTMPD-D
40.4
0.272
0.028
16.4 =..
0.584
'HALF' MASK INPAINTING
DPS-D
33.1
0.221
0.072
0.028
0.021
16.8 8.6
0.637
0,008
IGDM-D
35.8
0.397
0.103
0.031
0.020
15.9 228
0.419
0.132
y = 0.05
DTMPD-D
35.9
0.128
0.001
0.006
0.011
23.5 a.8
0.763
0,079
'BOX' MASK
DPS-D
31.1
0.078
+ 0.007
0.004
+ 0.003
25.5 4.8
0.830
0,070
INPAINTING
IIDM-D
36.3
0.319
0.110
0.012
0.081
20.0 1.9
0.545
0.142
By = 0.05
DTMPD-D
40.4
0.292
0.078
0.029
0.020
16.4 a.1
0.572
0,070
'HALF' MASK INPAINTING
DPS-D
32.5
0.230
0.673
0.028
0.023
16.7
0.626
0.098
IIDM-D
35.6
0.398
0.032
15.9
0.421
0.133
Ty = 0.1
DTMPD-D
38.9
0.168
0.007
22.5
0.728
'BOX' MASK
DPS-D
31.6
0.101
0.004
24.5
0.807
INPAINTING
IGDM-D
36.3
0.318
0.012
6.0
19.7
0.546
0.140
Ty 0.1
DTMPD-D
43.8
0.350
0.088
0.030
0.025
16.2 2.9
0.547
0,076
'HALF' MASK
DPS-D
33.0
0.522
0.007
0.031
0,02
16.3
0.602
0.007
INPAINTING
IIDM-D
36.2
0.276
0.085
0.034
1 0.033
15.5
0.412
0.128
By = 0.01
DTMPD-D
33.2
0.117
+ 0:051
0.004
1: 0.00
24.7 2.9
0.835
0.071
2x 'NEARES''
DPS-D
32.5
0.099
+ 0.904
0.004
i 0.003
25.1 a.1
0.847
0.073
SUPER-RESOLLUION
IIGM-DD
35.6
0.407
0.118
0.016
0.005
18.2 =
0.442
0.152
Ty 0.01
DTMPD-D
41.5
0.278
0.081
0.011
0.00
20.4 =
0.563
0.118
4x 'BICUBIC"
DPS-D
33.9
0.220
0.010
20.8
0.609
SUPER-RESOLUTION
IGDM-D
39.4
0.279
0.011
20.1
0.546
Ty = 0.05
DTMPD-D
33.9
0.156
0.005
24.1
0.810
2x 'NEAREST'
DPS-D
34.4
0.127
0.048
0.004
0.003
24.4 2.7
0.825
0,070
COPERRRESOLUTION
IGDM-D
35.1
0.407
0.118
0.016
0.005
18.2
0.440
0.158
Ty = 0.05
DTMPD-D
38.3
0.332
0.005
0.013
0.082
19.6
0.501
0.116
4x 'BICUBIC'
DPS-D
38.2
0.265
# 0.087
0.011
+ 0.005
20.2
0.567 0.128
SUPER-RESOLUTION
IGDM-D
23.2
0.522
0.101
0.034
0.122
15.0 =
0.215
0.114
Ty = 0.1
DTMPD-D
35.0
0.208
0.085
0.006
0.005
23.1 2.6
0.760
0,001
2x 'NEAREST'
DPS-D
32.1
0.152
0.888
0.005
0.003
23.8 22
0.802
0.075
IGDM-D
35.0
0.407
0.017
18.1
0.435
Tg = 0.1
DTMPD-D
36.8
0.378
0.015
18.7
0.435
4x 'BICUBIC'
DPS-D
34.8
0.308
0.013
19.4
0.513
SUPER-RESOLUTION
HGDM-D
23.0
0.521
0.101
0.035
0.013
14.9
0.212
6.117
32

---

Published in Transactions on Machine Learning Research (09/2024)
Table 9: Noisy observation inpainting and super-resolution for the reverse VP-SDEs on CIFAR-10 1k vali-
dation set.
PROBLEM
METHOD
FID
LPIPS +
MSE
PSNR
SSTM t
y 0.01 'BXX' MASK
TMPD
33.2
0.088
0.007
23.8
0.785
DPS
61.8
0.646
0.078
0.111
9.9 +1
0.050
0,008
INPAINTING
IIDM
34.8
0.073
0.055
0.004
0.005
25.1 5.5
0.816
0.005
By = 0.01
TMPD
37.4
0.267
0.008
0.030
0.006
16.3 3.0
0.581
0.070
'HALF' MASK
DPS
65.8
0.645
0.079
0,112
0,000
9.9 +
0.052
0,009
INPAINTING
IIDM
33.3
0.236
+ 0.088
0.027
0.021
17.0 s.8
0.620
0.087
By = 0.05
TMPD
33.5
0.117
+ 0.052
0.006
0.00
23.5 s.2
0.763
0.074
'BOX' MASK
DPS
62.2
0.647
9.078
0.110
0.018
10.0
0.055
0.007
INPAINTING
IIDM
35.2
0.103
0.007
0.004
0.003
24.8
0.798
0.074
By 0.05
TMPD
37.9
0.286
0.030
16.2
0.567
0.075
'HALF' MASK INPAINTING
DPS
65.1
0.647
0.117
9.7
0.052
HGDM
34.1
0.256
0,078
0.027
16.8
0.605
0.064
By = 0.1
TMPD
35.1
0.159
0.007
22.6
0.731
'BOX' MASK INPAINTING
DPS
61.9
0.647
9.074
0.112
0.052
9.9 1.9
0.054
0.071
IIDM
36.2
0.140
0.050
0.005
0,008
23.7 +
0.763
0.083
By = 0.1
TMPD
41.9
0.310
+: 0.077
0.031
0.000
16.0 1
0.541
0.078
'HALF" MASK
DPS
81.0
0.646
+ 0.De
0.112
0.052
10.0 1
0.052
0.009
INPAINTING
HIGM
38.2
0.279
0.074
0.029
0.021
16.4 8.2
0.577
0.005
y 0.01
TMPD
32.4
0.123
0.003
0.005
0.009
24.5
0.826
0.077
2x 'EEAREST'
DPS
71.6
0.645
0.118
9.7 +.20
0.049
SUPER-RESOLUTION
HIDM
41.7
0.192
0.005
23.3
0.773
Y = 0.01
TMPD
22.9
0.525
0.035
14.9
0.206
4x 'IICUBIC'
DPS
77.9
0.640
o.076
0.120
9.7
0.046
0,070
SUPER-RESOLUTION
IGDM
36.7
0.461
0.0R3
0.023
0.000
16.7
0.305
0.308
Y = 0.05
TMPD
32.7
0.158
0.000
0.005
0.003
23.9
0.800
0.075
2x 'NEAREST'
DPS
72.5
0.648
o.078
0.118
0,058
9.7
0.050
0.009
IIDM
37.3
0.229
0.0N
0.006
0.005
22.6 2.1
0.735
0.086
oy = 0.05
TMPD
35.7
0.328
0.09L
0.013
0.001
19.5 2.
0.495
0.117
4x 'BICUBIC'
DPS
77.9
0.641
QU071
0.120
0.001
9.7 +
0.047
0,070
SUPER-RESOLUTION
IGDM
34.8
0.479
0.09
0.026
0.000
16.2
0.276
0.112
Og 0.1
TMPD
32.9
0.207
0,08L
0.006
23.0
0.753
0.088
2x 'NEAREST'
DPS
72.2
0.644
0.118
9.7
0.051
SUPER-RESOLUTION
IGDM
35.9
0.273
0.007
21.7
0.682
Ty = 0.1
TMPD
34.7
0.379
0.096
0.016
0.007
18.5
0.429
4x 'RICUBIC'
DPS
78.3
0.639
0.074
0.120
0.050
9.7
0.050
0.065
SUPER-REE RESOLUTION
HIGMM
34.4
0.492
0.097
0.028
0.0LD
15.8 1.7
0.258
0.112
33

---

Published in Transactions on Machine Learning Research (09/2024)
08
Rapes
LPIPS
FID
SSIM
noise std,
noise std, ay
Figure 5: 'random" mask inpainting FID vs LPIPS (left) and SSIM (right) using the VP-SDE on FFHQ-IK
validation dalaset for increasing observation noise.
Ground truth
Measurement
TMPD-B jound
DPS ahaagaa
HIGDM-D 2021
78
Figure 6: 4x bicubic super-resolution samples from the VP-SDE on FFHQ across a range observation noise.
algorithm was only performant in low noise settings, whereas DTMPD-D is performant across a wide range of noise levels.
It has been observed that IIGDM-D performs worse for 1000 steps than 100 steps (scc (Mardani et al. 2023 Section 5)). We also provide results in Table [1 for 100 steps of DTMPD-D and IIDM-D for the same FFHQ VE-SDE model, to compare to TGDM-D in its optimal setting.
34

---

Published in Transactions on Machine Learning Research (09/2024)
Ground truth
Measurement
TMPD-D jound
DPS-D (chaag
IIGMMDD
70
Figure 'Random' mask inpainting samples from the VP-SDE on FFHQ across a range observation noise.
Table 10: Quantilative evaluation of solving linear inverse problems for VE DDPM for 1000 steps with increasing noise on FFHQ 256x256-1k validation dataset.
PROBLEM
METHOD
FID +
LPIPS
MSE
+
PSNR f
SSIM 1
By 0.01
DTMPD-D
32.3
0.203 + 20000
1.57g:03
+ 8800404
28.6 211
0.810
0,022
4x 'BICUBIC"
DPS-D
47.0
0.273
1.70g.03
5.002444
28.1 +
0.747
TIGM-DD
37.4
0.244
1 0.000
1.74F-03
+: #.6wec
28.1 2.0
0.755
4: 0.012
By 0.05
DTMPD-D
32.1
0.268
2.62E-03
1.17aca
26.2
0.733
4x 'DICURIC
DPS-D
105.9
0.590
0.936
6.18r-03
+: sanees
22.1 0.
0.404
: 0,040
SUPER-RESOLUTION
IGGM-DD
106.8
0.592
0.64
7.92r.03
+: 1otwea
21.0
0.353
::
Ty = 0.1
DTMPD-D
32.7
0.310
3.99E-03
+ 1.4444
24.3
0.679
'BICUBIC
DPS-D
114.0
0.569
0.044
8,47E-03
+ 4.63400
21.1 + 1.2
0.483 + 0.046
IMPPER-ESOLUTION
TIGM-DD
206.0
0.724
2.46F:02
16.1
0.176
By = 0.01
DTMPD-D
30.2
0. 114
0.029
2.84E-03
+ 2.50a.0a
26.5 + 2.8
0.907 + 0,520
'BOX'
DPS-D
23.9
0.093
1 0019
2.53r:03
4: 1.77w0
26.9 2.8
0.899 1: 0.017
INPAINTING
IIGM-DD
27.1
0.108
0.225
2.50n-03
4: 1.35wcd
26.7
0.877 1: 0.015
Jy = 0.05
DTMPD-D
33.6
0.186
+ 0096
3.24E-03
+ 5.19w40
25.8
4 2.
0.847 + 00000
'BOX' MASK INPAINTING
DPS-D
39.7
0.318
3.81E-03
192244
24.7
0.677
IGDM-DD
49.5
0.354
4 0.61
4.67E-03
4 1...100 1.30s-ca
23.5
2: 1.a
0.555 4 0.001
Gy 0.1
DTMPD-D
34.0
0.223
3.42E:03
2243.00
25.3
0.801 0.045
'BOX' MASK INPAINTING
DPS-D
59.1
0.467
0.53
7.78F-03
1: s.swec
21.4
1
0.476 1 0.110
TIGM--D
72.6
0.529
9.88r.-03
4 1-48
20.2
0.356 1
CIFAR10 64x64 For VE-SDE, our CIFAR-10 64x64 experiments compare TMPD to DPS and HIGMM, and also compare diffusion methods (DDPM) in Table 12 and score-based methods (discretized with Euler-
Maruyama) in Table 13 Various samples used produce the figures Tables 12 and 13 are shown in Fig 16
and 17
35

---

Published in Transactions on Machine Learning Research (09/2024)
Ground truth
Mensurement
TMPD-D Jound
DPS-D
IIDM-DD
Figure 8: 'Box' mask inpainting samples from the VP-SDE on FFHQ across a range observation noise.
Table 11:
Quantitative evaluation of solving linear inverse problems for VE DDPM for 100 steps with increasing noise on FFHQ 256x250-1k validation dataset.
PROBLEM
МETHOD
FID 1
LPIPS
MSE
PSNR 1
SSIM
Ty 0.01
DTMMPDDD
0.215
0.012
0.001
+ 0.001
28.9 + 2:2
0.821
+ 2004
4x 'BICUBIC' SUPER-RESOLUTION
TIGDM-D
46.1
0.252
0.002
27.4
0.788
By = 0.05
DTMMP-D
39.2
0.279
0,040
0.003
0.001
26.2
0.741
+ 00065
4x 'RICUBIC' SUPER-RESOLLTTION
TGDM-D
43.3
0.302
+ 0.01
0.003
:: 0.001
25.1 1:
0.722
Jy = 0.1
DTMPD-D
41.1
0.317
0.004
24.5 1.7
0.693
o0078
'BICUBIC' SUPERCRESSOLLTION
IIDM-D
44.7
0.344
+ 0.003
0.005
+ 0.002
23.4 + 1.6
0.672
0.070
Ty 0.01
DTMPD-D
31.4
0.124
0.003
26.5
0.906
'BOX' MASK INPAINTING
IIDM-D
23.7
0.069
0.121
0.002
0.001
28.0 2.8
0.925 + a.16
Ty 0.05
DTMPD-D
38.2
0.199
1 0.009
0.003
1: 0.003
26.0 1:
0.848 1: 3:044
'BOX' MASK INPAINTING
IIGDM-D
29.8
0.185
0.002
26.7 2.1
0.825
0.021
By = 0.1
DTMPD-D
39.7
0.241
0.046
0.004
=: 0.005
25.2
1: 2.1
0.799
1 0.088
'BOX' MASK INPAINTING
IIDM-DD
42.1
0.279
0.003
25.1
0.716
For IIGMM(-),, we are able to use the hyperparameter v = v1/(1 (1 + vz) as suggested by [Song et al. (2023). which is calculated by assuming the data distribution px(x)) standard normal and then calculating the posterior variance, for both VE DDIM and VE-SDE methods, but note some instability for the IIGDM VE-SDE method for small noise, shown in Fig 10
For DDPM we use the step-size constant suggested in Chung et al. (2022a) for inpainting. Ç. = =<<'/lly
Hmojel, where we tune ç' over the suggested range of 5" E [0.1, 1.0] in Chung et al. (2022a) across LPIPS, MSE, PSNR and SSIM, as shown in Fig. for each inverse problem (cach line in the Tables 12 and 13
36

---

Published in Transactions on Machine Learning Research (09/2024)
Table 12: Noisy observation inpainting and super-resolution for VE DDPM on CIFAR-10 1K validation set.
PROBLEM
МETHOD
FID
LPIPS
MSE +
PSNR
SSIM T
By 0.01
DTMPD-D
34.1
0.094
0014
0.006
0.00
23.6 23
0.782 0.057
'BOX' MASK
DPS-D
40.6
0.085
0.005
24.2
0.802
INPAINTING
IGDM-D
40.0
0.076
0.004
25.0 =.8
0.824
By = 0.01
DTMPD-D
39.0
0.271
0.028
16.5 2.1
0.582
'HALF' MASK INPAINTING
DPS-D
42.8
0.247
0.065
0.029
0.020
16.5 #.
0.595
6,077
IGDM-D
42.2
0.237
0.087
0.029
0.02)
16.4 +.8.1
0.610
6,079
B = 0.05
DTMPD-D
36.7
0.163
+ 0.070
0.007
0.011
22.6 +.
0.724
0,082
'BOX' MASK
DPS-D
101.5
0.224
# 0:077
0.008
0.004
21.8 2.2
0.681
6,057
INPAINTING
IIDM-D
99.9
0.225
0.006
0.007
0.00a
21.8 1.7
0.682
0.003
Jy = 0.05
DTMPD-D
40.9
0.320
0.078
0.029
0.020
16.4 +2
0.544
0,078
'HALF' MASK
DPS-D
105.5
0.362
0.078
0.032
0.021
15.7 2.
0.500
6.07
INPAINTING
IIDM-D
97.1
0.366
0.033
0.020
15.4
0.503
6.0/9
Jy 0.1
DTMPD-D
38.1
0.233
0.009
21.4
0.652
'BOX' MASK
DPS-D
80.4
0.313
0.012
19.7
0.596
INPAINTING
IGDM-D
140.5
0.378
086
0.013
0.000
18.8
0.550
6.113
Tg 0.1
DTMPD-D
40.8
0.359
0.086
0.030
0.029
16.1 2.8
0.495
0.057
'HALF' MASK
DPS-D
99.5
0.434
0/072
0.038
0.023
14.8 2.4
0.400
6.063
INPAINTING
IIDM-DD
138.8
0.476
0:078
0.039
+: 0.019
14.5 =
0.402
6.035
By = 0.01
DTMPD-D
32.7
0.126
1 0.055
0.004
i: o.a
24.4 2.9
0.828
6.073
2x 'NEAREST'
DPS-D
42.3
0.134
# 0.083
0.004
+ 0.002
24.8 22
0.839
6.053
SUPER-RESOLUTION
IIGM-D
34.9
0.110
0.002
0.004
0.000
24.8 2.5
0.839
0.036
Ty 0.01
DTMPD-D
38.6
0.295
0.082
0.011
0.00
20.2 2.
0.544
6.100
4x 'BICUBIC'
DPS-D
53.4
0.296
0.011
20.1
0.547
SUPER-RESOLUTION
IGDM-D
37.8
0.250
0.077
0.011
20.4
0.577
Ty = 0.05
DTMPD-D
34.0
0.204
0.006
23.1 2.3
0.762
2x 'NEAREST'
DPS-D
98.8
0.283
0.077
0.007
0.003
21.9 1.6
0.711
6,051
SUPER-RESOLUTION
IGDM-D
99.6
0.298
0.081
0.008
0.00G
21.1
0.677
6.022
T = 0.05
DTMPD-D
36.8
0.380
0.007
0.015
0.006
18.8 +
0.440
6,118
4x 'BICUBIC'
DPS-D
126.1
0.467
+ 0.072
0.017
0.007
18.2 1.7
0.434
6.036
IIDM-D
104.2
0.419
0.088
0.016
0.000)
18.1 1.6
0.454
6.117
T2 0.1
DTMPD-D
37.1
0.285
+ 0,300
0.008
0.000
21.5
0.657
6,112
2x 'NEAREST'
DPS-D
144.6
0.421
0.082
0.012
0.00c
19.4
0.571
6.101
SUPER-RESOLUTION
IIDM-DD
175.8
0.456
0.016
0.000
17.9
0.514
Ty = 0.1
DTMPD-D
36.1
0.439
0.019
17.5
0.353
4x 'BICUBIC'
DPS-D
155.1
0.535
0.022
16.8
0.360
SUPER-RESOLUTION
TGDM-D
196.2
0.539
0787
0.030
0.007
15.3
0.327
0.106
37

---

Published in Transactions on Machine Learning Research (09/2024)
Table 13: Noisy observation inpainting and super-resolution for the reverse VE-SDEs CIFAR-10 1k
validation set.
PROBLEM
METHOD
FID
LPIPS +
MSE
PSNR f
SSIM 1
8y = 0.01 'BOX' MASK
TMPD
40.0
0.102
0.047
0.005
0.00c
23.7 +81
0.773
0,009
DPS
103.6
0.637
0,074
0.114
0.002
9.9 19
0.050
0.005
INPAINTING
IIDM
78.9
0.094
0.036
0.005
0.00%
24.1 3.1
0.787
0.071
By = 0.01
TMPD
45.8
0.279
0.066
0.030
0,029
16.3 8.0
0.569
0,009
'HALF' MASK
DPS
110.8
0.638
0.072
0.117
0.008
9.8 1 2.0
0.045
0.007
INPAINTING
IIGM
50.5
0.264
0.068
0.027
0.000
16.7 3.1
0.585
0.077
By 0.05
TMPD
45.3
0.167
9.065
0.007
22.6 4
0.718
0.078
'BOX' MASK
DPS
103.4
0.638
0.115
9.8
0.047
INPAINTING
IIDM
81.9
0.160
0.006
22.8
0.720
B = 0.05
TMPD
51.1
0.319
0.029
16.2
0.532
'HALF' MASK
DPS
109.3
0.638
0.070
0.116
9.8
0.044
0.067
INPAINTING
IGDM
56.0
0.311
9.004
0.029
0./22
16.4
0.540
0,008
By = 0.1
TMPD
47.6
0.231
9.079
0.008
0.005
21.5 2a
0.650
0.087
'BOX' MASK
DPS
104.3
0.639
0.068
0.113
0,040
9.9 + 1.9
0.048
0.065
INPAINTING
IGDM
84.6
0.220
+ OU02
0.008
0.000
21.6
0.665
0.305
By = 0.1
TMPD
54.1
0.366
Q.980
0.032
0,020
15.8
0,480
0.085
'HALF' MASK
DPS
110.4
0.639
0.068
0.117
0.004
9.7 + 2.0
0.046
0.007
INPAINTING
IGDM
58.5
0.349
0.08
0.031
16.1
0.499
0.000
γ 0.01
TMPD
43.5
0.141
0.007
24.1
0.810
2x 'NEAREST'
DPS
118.7
0.641
0.117
9.7
0.048
SUPER- REEEERRRESOOT
IGDM
59.4
0.258
0.007
22.0
0.716
Y = 0.01
TMPD
49.6
0.439
0.053
0.020
0.008
17.3 1.8
0.345
0.115
4x 'BICUBIC'
DPS
128.3
0.642
0.071
0.122
0.009
9.6 2.0
0.046
0.005
DEPERRRESOLUTION
IIDM
52.9
0.561
0.053
0.042
0.015
14.1 +
0.167
0.085
y = 0.05
TMPD
47.5
0.213
0.087
0.007
0,012
22.8 2.6
0.744
0,100
2x 'NEAREST'
DPS
119.1
0.638
+ 0.000
0.118
0.009
9.7 + 2.0
0.049
0.005
SUPER-RESOLUTION
IIDM
60.0
0.326
0.097
0.009
0.004
20.7 4
0.619
0.104
y 0.05
TMPD
51.2
0.379
0.091
0.016
0.07M
18.6
0.428
0.117
4x 'BICUBIC'
DPS
128.0
0.642
0.088
0.122
9.6 2.0
0.043
0.007
SUPER-RESOLUTION
HIGM
53.8
0.548
0.038
14.5
0.186
B = 0.1
TMPD
51.6
0.292
0.009
21.3
0.646
2x 'NEAREST'
DPS
120.5
0.644
0.074
0.121
9.7 +
0.048
0.068
SUPER-RESOLUTION
TIDM
61.9
0.386
0.100
0.012 0.005
19.5 1.5
0.523
0.117
yy = 0.1
TMPD
49.6
0.439
0.050
0.020
0.008
17.3 1.8
0.345
0.115
4x 'BICUBIC'
DPS
128.3
0.642
0.071
0.122
0.000
9.6 + 2.0
0.046
0.068
SUPER-RESOLUTEE RESOLUTION
HIDM
52.9
0.561
+ o.083
0.042
0.005
14.1 1.2
0.167
0.085
38

---

Published in Transactions on Machine Learning Research (09/2024)
servaatio
Obua vallons
10-1
Exc0
4xc00
10
10-4
10P
10
10a
dbs_scale
dos_scele
Observation:
2x10
10
Figure 9: DPS scale hyperparameter search across LPIPS, MSE, PSNR and SSIM for CIFAR-10 4x bicubic interpolation super-resolution with 0.05. Plotted are the mean values + 2 standard deviations over 128 sample/bates /bach size, calculated over 10 values of ç'. We chose an optimal value of 0.15 for the DPS scale hyperparameter in this case.
39

---

Published in Transactions on Machine Learning Research (09/2024)
Ground truth
Meusurement
TMPD-D (ours)
TMPD (ours)
DPS-D (Chung 2022)
DPS
TGDM-D (Song et al., 2023)
TIGDM
Figure 10: Inpainting samples from the VP-SDE CIFAR-10. The observation model was 'box' mask with
Gaussian (ay 0.05) noisc.
40

---

Published in Transactions on Machine Learning Research (09/2024)
Ground truth
Mensurement
TMPD-D (ours)
TMPD (ours)
DPS-D (Chung 2022)
DPS
IGGM-D (Song et al., 2023)
TIGDM
Figure 11: 2x nearest-neighhour super-resolution samples from the VP-SDE on CIFAR-10. The observation model was Gaussian (ay = 0.05) noisc.
41

---

Published in Transactions on Machine Learning Research (09/2024)
61
08
175
275
0.5
130
130
LPIPS
SSIM
05-
100
100
04-
-75
-75
03
30:10 noise std, oy
3::00"
noise std, oy
Figure 12: 4x bicubic super-resolution FID vs LPIPS (left) and SSIM (right) using the VE-SDE on FFHQ-IK 1k validation datasct for incrcasing observation noise.
D
07-
PPPSS
FID
SSIM
FID
0.-
0.2
noise std, Gy
noise std, G
Figure 13: 'box' mask inpainting FID vs LPIPS (left) and SSIM (right) using the VE-SDE on FFHQ-Ik validation dataset for increasing observation noise.
42

---

Published in Transactions on Machine Learning Research (09/2024)
Groun Meascament
TMPD
DPS DPS-Doorrnn..
TIDMM
Figure 14: Whilst DTMPD-D with the VE-SDE remains robust to increasing noise, DPS-D and HIGM- D do not. This illustrated by samples from the different guidance methods for a 4x super-resolution (256 256 + 64 64) problem with increasing Gaussian observation noise distorting a ground truth image. The top row measurement has By 0.01, the middle row By 0.05 and the bottom σy 0.1. For the full results, see Table 10
Ground truth
Measurement
TMPD-D (ours)
DPS-D (Ching 2022)
HIGDM-D (Song
Figure 15: 4x bicubic super-resolution samples from the VE-SDE FFHQ. The observation model
Gaussian (ay 0.05) noise.
43

---

Published in Transactions on Machine Learning Research (09/2024)
Ground truth
Meusurement
TMPD-D (ours)
TMPD (ours)
DPS-D (Chung 2022)
DPS
TIGM-D (Song et al., 2023)
FIGDM
Figure 16: Inpainting samples from the VE-SDE 10. The observation model was 'box' mask with
Gaussian (ay 0.05) noisc.
44

---

Published in Transactions on Machine Learning Research (09/2024)
Ground truth
Meusurement
TMPD-D (ours)
TMPD (ours)
DPS-D (Chung 2022)
DPS
TGDM-D (ong et al., 2023)
FIGDM
Figure 17: 2x nearest-neighhour super-resolution samples from the VE-SDE CIFAR-10. The observation model was Gaussian (ay = 0.05) noisc.
45


Reviews:
[{"review_id": "njcuazynse", "reviewer": "Reviewer_ymX8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1. The paper is well-written and makes the method straightforward to understand\n2. The authors address the computational drawbacks of this method in Section 3, which is nice.\n3. The empirical results look better than the baselines, but I have some queries about this (see below). \n\n**Weaknesses**: See the Requested Changes section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method to approximate the diffusion posterior more expressively than prior works. This approximation can solve inverse problems by leveraging pre-trained diffusion models as priors. The authors demonstrate empirical results on toy datasets and the FFHQ image dataset for super-resolution and inpainting tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**General Comments/Requested Changes:**\n\nI don't have any major criticisms of this work (a primary concern of computational efficiency due to higher order gradients of the score function has already been addressed by the authors in the main text). I have some queries and comments on improving the paper further.\n\n**On the technical background**: The authors state: “There are two main diffusion modeling paradigms, namely, score-based models (SGMs) (Song et al., 2020) and denoising diffusion probabilistic models (DDPM) (Ho et al., 2020)”. This seems misleading. A lot of prior work (including the DDPM [Ho et al.] and the SGM [Song et al.] papers) shows that these are not separate paradigms but are equivalent depending on the score/denoiser parameterization. Please rephrase this introductory statement appropriately, as you later mentioned that these are closely related.\n\nThis section generally seems dated, and references to more recent diffusions like EDM/EDM2 [Karras et al.] or interpolants [Albergo et al., Lipman et al.] are missing. It would be great if the authors could update this section appropriately.\n\nNit: Equations 1 and 2 can be converted to single-line equations for better readability. However, this is only my preference, and I leave this to the author's discretion.\n\n**On Section 4**: While I appreciate the theoretical results, I don't think they add much value to the main text since the diffusion posterior is much more complicated than a simple Gaussian for any practical scenario I can imagine. In this context, it's best to move this section to the Appendix for better readability and faster access to the experimental section, where the empirical results on actual datasets are good.\n\n**On empirical results**: \n1. It's interesting that DPS and Pi-GDM don't quite work well with the VE-SDE process. Can the authors elaborate on whether the hyperparameters were tuned for the best performance for the DPS and Pi-GDM baselines for VE-SDE? If yes, is there any intuition on why these methods might not work in VE-SDE?\n2. For the experimental setup, what is the number of sampling steps used for Pi-GDM? I’m guessing this is 1000 since the authors set the eta parameter in the DDIM sampler to 1.0. However, the optimal setting for Pi-GDM is to set the number of sampling steps to 100 using DDIM. It has been observed that Pi-GDM performs worse for 1000 steps (See Red-Diff [Mardani et al.]). I'm curious how the proposed method scales for lesser sampling budgets (like 100 steps) compared to Pi-GDM. Can the authors include results either for super-resolution or inpainting for this setting?\n3. Are the results in Table 2 based on the diagonal variant of TMPD or the full covariance matrix? Based on the discussion in Section 3.4, these results appear for DTMPD. If yes, please change the method name in Table 2. If not, please add wall clock time estimates since, in that case, Pi-GDM would be significantly faster.\n\n**On related works**: Discussion on some recent related works like Red-Diff [1], D-Flow [3], Pi-GDM for flows [2], inverse samplers for latent diffusion [4,5], and fast samplers for inverse problems [6] is missing. Please note that the mentioned list of works is not exhaustive, and I leave it to the authors to improve the discussion in this section.\n\n[1] A Variational Perspective on Solving Inverse Problems with Diffusion Models - Mardani et al.\n\n[2] Training-free Linear Image Inverses via Flows - Pokle et al.\n\n[3] D-Flow: Differentiating through Flows for Controlled Generation - Hamu et al.\n\n[4] Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models - Rout et al.\n\n[5] Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion - Rout et al.\n\n[6] Fast Samplers for Inverse Problems in Iterative Refinement Models - Pandey et al.", "broader_impact_concerns": "Have been addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YVeLcCRBuc", "reviewer": "Reviewer_QhaZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly written, although with a number of grammatical mistakes. I would suggest the authors run the text through grammarly or writefull or similar. I'm also a bit confused by the \"inverse problem\" intro/motivation. In principle the p(y|x) they consider here is only the linear case, which is not a very interesting inverse problem. \n\nI didn't check the proofs carefully but the technical contributions appear sound, well motivated and backed by both theory and empirical results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider conditional sampling from diffusion models (DMs). They build on recent work that uses Tweedie's formula to approximate the reverse process, by estimating a full or diagonal covariance matrix. This enables exact sampling in the Gaussian case, new theoretical guarantees, and promising empirical results. The main downside is computational cost, although with approximations this is reasonable in the important inpainting and superresolution settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Releasing code should be a requirement of publication since the methodology is reasonably involved so having a working implementation will be important for those wanting to try the approach out. \n\nI don't know if it would be practical to compare to but the state-of-the-art work AFAIK in this space is https://arxiv.org/abs/2306.17775. It should at least be cited/discussed. \n\nMinor comments: \nI found the sentence \"Optimally, we would like to sample from the reverse SDE to the forward SDE started in...\" hard to grok. \nIn eq 9 I think f should be indexed by t.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u3cXVMVA9q", "reviewer": "Reviewer_uMGN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The submission is well-motivated: existing methods are extremely sensitive to hyperparameter tuning in terms of their reconstruction quality.\n- Providing theoretical guarantees is an important but often underestimated aspect of guidance methods.\n- Assumptions and limitations are presented clearly.\n\nWeaknesses:\n\n- Scalability of the second moment estimate is the biggest limitation of the proposed method (as it is discussed in the text).\n- Discussion of the theoretical results and their implications could be improved.\n- Missing comparison with SNIPS (Kawar et al., 2021), which is cited in the text.\n\nI will expand on these points below, and I am looking forward to discussing with the authors to clarify my questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission focuses on solving inverse problems with diffusion models, where computing the conditional score $\\nabla_{x_t} \\log p_t(x_t | y)$ in the reverse SDE is, in general, intractable. Building upon previous works, the submission proposes to also match the second moment of the Tweedie estimate in order to resolve known issues of hyperparameter tuning of existing alternatives (this method is called TMPD). \n\nThe submission discusses the computational complexity of this algorithm, its practical implementation and limitations, and theoretical results on when TMPD provides good approximations to the true posterior distribution.\n\nExperiments are conducted on both synthetic data and two real-world image datasets, where TMPD is shown to outperform two commonly used methods: DPS and PGDM (read $\\Pi$GDM).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Equation 9**\n\nI would suggest to expand the discussion paragraph after the equation. In particular, all approximations used to obtain Eq. (9) should be clearly stated and made precise.\n\n**Approximation based on properties of $H$**\n\nCould the authors expand on what other strategies could be employed in a general case where the operator $H$ may not be sparse?\n\n**Proposition 3**\n\nCould the authors expand on their claim \"if there are no errors in the initial condition\". What does this mean in practice? Is this equivalent to saying that the measurement should be noiseless?\n\n**Theorem 1**\n\n- The symbol $\\Phi$ is never defined, what is it referring to?\n- All equations should be numbered, including within Theorems.\n- I am not sure I fully understand the claim in the Theorem: is it bounding the total variation at the **end** of the reverse process (i.e., $t=T$), or **at any** time $t$? The text reads \"where $q_t$\", but the equation contains $q_T$ and $T$, not $t$. \n- I am not sure I fully understand the relation between $t$ and $T$ in the bound. Could the authors expand on an intuitive discussion of the claim in the Theorem? For example, what happens at the same time $t$ of the reverse process when starting from different $T$'s?\n\n**Remark 1**\n\nIn its current formulation, this is too strong of a claim. If the extensions are trivial, they should be included in the current manuscript. If not, more evidence needs to support the claim that all of these (ambitious) extensions can be attained. \n\n**Paragraph after Remark 1**\n\n- Clarify what the \"limit\" being taken is.\n- I am confused by the next sentence. \"The $\\Phi$ ... converges to 1 ... and therefore $M \\to 1$\". But $\\exp(1) = e > 1$, what am I misunderstanding here? Should it be \"the $\\Phi$ in Eq. (12)\" converges to 0?\n\n**Related work**\n\n> The mean is chosen the same way as our method.\n\nThis sentence should be rephrased, as DPS introduced ideas of using Tweedie's estimate in inverse problems before the current submission.\n\n**GMM experiment**\n\nIn the text, it is stated a few times \"non-linear SDE\" and \"non-linear problems in high dimension\", but I thought the submission was about linear inverse problems? In the Introduction, \"... a known linear observation map $H$\". What non-linearity are the authors referring to here?\n\n**Comparison with SNIPS**\n\nContinuing on my previous point of confusion, if this submission is only about linear inverse problems, an important comparison with SNIPS (Kawar et al., 2021) is missing. This comparison is important because SNIPS introduced a closed-form solution and efficient approximation of the **exact** reverse conditional likelihood for inverse linear problems (if the SVD of $H$ can be computed).\n\nCould the authors expand on their motivation not to compare with SNIPS and the advantages/limitations of TMPD compared to SNIPS? For example, could ideas of TMPD be extended to non-linear measurement models? This point should be made clearer, especially given the significant computational limitations of the submission (e.g., image results are shown on 32x32 pixels and 256x256 pixels images, whereas problems of practical relevance can be of the order of 1024x1024 pixels).\n\n---\n\n**Minor comments**\n\n- Typo in \"reasonance\" imaging\n- Typo in \"is devoted developing\"\n- Sec. 2, Technical background: \"In the SDE paradigm ... \". So far, the paradigms have been introduced as \"SGM\" and \"DDPM\". But \"SDE\" is only introduced afterwards \"In SGMs, a stochastic different equation (SDE) ...\". I would flip the order to clarify which paradigm uses SDEs.\n- Please conform cross reference formats: references to equations should be capitalized and have numbers in parentheses (e.g., \"equation 2\" -> \"Equation (2)\").\n- Please rephrase \"Optimally, we would like to sample from the reverse ... to the forward SDE started in $p_{\\text{data}}(\\cdot | y)$. This sentence reads as if the forward SDE is started at the measurement $y$, whereas its the reverse.\n- Typo in \"while $m_{0|t}$ and $C_{0|t}$ gives\"\n- Typo in \"PiDGM-D is not able to produce high quality ...\"", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work develops a method to better tackle conditional sampling in diffusion models when approximating the inverse/reverse process.  The method takes advantage of Tweedie's formula to enable exact sampling under Gaussian assumptions and provide new theoretical insights.  The authors demonstrate the method using a GMM example and image inpainting and superresolution problems.  Overall, the reviewers all found that the paper was well written (modulo some typos and grammar issues), interesting to the community, novel and sound.  The main concern before the rebuttal period seemed to be regarding discussion of recently related work, but the authors appear to have sufficiently addressed this in their response.  The remaining concern, highlighted by all reviewers, is that the proposed method is computationally expensive and thus the practical usefulness of the approach might be limited.   They suggested that developing efficient approximations would make the work more compelling and impactful.  Nonetheless, the reviewers all recommended to accept the paper.
Decision: Accept (poster)