Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models
Authors: Litu Rout
Negin Raoof
Giannis Daras
Constantine Caramanis
Alex Dimakis
Sanjay Shakkottai
Venue: NeurIPS 2023 poster (2023)
Keywords: Inverse Problems
Posterior Sampling
Latent Diffusion Model
Stable Diffusion
Sample Recovery
TL;DR: We present the first framework to solve inverse problems with a pre-trained latent diffusion model, which is the backbone of large-scale foundational generative models such as Stable Diffusion.

Abstract: We present the first framework to solve linear inverse problems leveraging pre-trained \textit{latent} diffusion models.  Previously proposed algorithms (such as DPS and DDRM) only apply to \textit{pixel-space} diffusion models.  We theoretically analyze our algorithm showing provable sample recovery in a linear model setting. The algorithmic insight obtained from our analysis extends to more general settings often considered in practice. Experimentally, we outperform previously proposed posterior sampling algorithms in a wide variety of problems including random inpainting, block inpainting, denoising, deblurring, destriping, and super-resolution.

Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models
Litu Rout
Negin Raoof
Giannis Daras
Constantine Caramanis
Alexandros G. Dimakis
Sanjay Shakkottai
The University of Texas at Austin"
Abstract
We present the first framework to solve linear inverse problems leveraging pre- trained latent diffusion models. Previously proposed algorithms (such DPS and DDRM) only apply to pixel-space diffusion models. We theoretically analyze our algorithm showing provable sample recovery in linear model setting. The algo- rithmic insight obtained from our analysis extends to more general settings often considered in practice. Experimentally, we outperform previously proposed poste- rior sampling algorithms in wide variety of problems including random inpainting, block inpainting, denoising, deblurring, destriping, and super-resolution.
Introduction
We study the use of pre-trained latent diffusion models to solve linear inverse problems such as denoising, inpainting, compressed sensing and super-resolution. There are two classes of approaches for inverse problems: supervised methods where a restoration model is trained to solve the task at hand [37, 39, 56, 31J, and unsupervised methods that the prior learned by generative model to guide the restoration process [52, 40, 5, 33, 11, 26]; see also the survey of Ongie et al. [36] and references therein.
The second family of unsupervised methods has gained popularity because: (i)
foundation generative models have become widely available, (ii) unsupervised methods do not require any training to solve inverse problems and leverage the massive data and compute investment of pre-trained models and (iii) generative models sample from the posterior-distriuttin. mitigating methods such as bias in the reconstructions [35, 24] and
certain pitfalls of
regression the mean [23, 22].
Diffusion models have emerged as a powerful new approach to generative modeling [47, 48, 49, 20, 29, 18, 54]. This family of generative models works by first corrupting the data distribution po (o(z)) using an Stochastic Differential Equation (SDE), dæ = f(x t)df + g(f))d,, and then by learning the score-furee funceffnniion Vz, log Pe (xx,), all levels f, using Denoising Score Matching (DSM) [21, 53]. The seminal result of Anderson [1] shows that we can reverse the corruption process, i.e., start with noise and then sample from the data distribution, by rumning another Itō SDE. The SDE that corrupts the data is often termed as Forward SDE and its reverse as Reverse SDE [49]. The latter depends on the score-function V., m, log pt (xx) that we learn through DSM. In [8, 9], the authors provided non-asymptotic analysis for the sampling of diffusion models when the score-function is only learned approximately.
The success diffusion models sparked the interest to investigate how we can use them to solve inverse problems. Song et al. [49] showed that given measurements y Ax + n, we can
"Email: Mamill:(lina
htt.:ttaiiitteexccce...o dimakis @ austin.utexas.cdu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

00p
Figure 1: Overall pipeline of our proposed framework from left to right. Given an image (left) and user defined mask (center), our algorithm inpaints the masked region (right). The known part the images are unaltered (see Appendix C for web demo and image sources).
provably sample from the distribution po(ol)y) by running modified Reverse SDE that depends on the unconditional score Va, log (xx;) and the term Va, log p xx)) The latter term captures how much the current iterate explains the measurements and is intractable even for lincar inverse problems without assumptions on the distribution p=(x)) [11, 14]. To deal with the intractability of the problem, series of approximation algorithms have been developed [22, 11, 2, 13, 26, 10, 6, 46, 12, 27] for solving (linear and non-linear) inverse problems with diffusion models. These algorithms pre-trained diffusion models as flexible priors for the data distribution to effectively solve problems such as inpainting, deblurring, super-resolutte
among others.
Recently, diffusion models have been generalized to learn to invert non-Markovian and non-linear corruption processes [16, 15, 3]. One instance of this generalization the family of Latent Diffusion Models (LDMs) [41]. LDMs project the data into some latent space, 2o E(zo), perform the

---

diffusion in the latent space and use decoder, D(zo), to move back to the pixel space. LDMs power state-of state-of-the-art foundation models such as Stable Diffusion [41] and have enabled a wide-
range of applications across many data modalities including images [41], video [4], audio [30] and medical domain distributions (e-g., for MRI and proteins) [38, 51]. Unfortunately, none of the existing algorithms for solving inverse problems works with Latent Diffusion Models. Hence, to use a foundation model, such as Stable Diffusion, for some inverse problem, one needs to perform
finetuning for each task of interest.
In this paper, we present the first framework to solve general inverse problems with pre-trained latent diffusion models. Our main idea is to extend DPS by adding an extra gradient update step to guide the diffusion process to sample latents for which the decoding poodding-nnoding map not lossy. By harnessing the power of available foundation models, we are able to outperform previous approaches without finetuning across a wide range of problems (see Figure and 2).
Our contributions are as follows:
(i) We show how to use Latent Diffusion Models models (such as Stable Diffusion) to solve linear inverse problem when the degradation operator is known.
(ii) We theoretically analyze our algorithm and show provable sample recovery in a linear model setting with two-step diffusion processes.
(iii) We achieve a new state-of-the-ar for solving inverse problems with latent diffusion models, outperforming previous approaches for inpainting, block inpainting, denoising, deblurring, destriping, and super-resolution."
2 Background and Method
Bold lower-case æ. bold upper-case X, and normal lower case x denote vector, matrix, and a scalar variable, respectively. We denote by @ element-wise multiplication. D(x) represents diagonal matrix with entries We use E() for the encoder and (-) for the decoder. Edp is pushforward measure of i.c., for every x E the sample {xx) is sample from Etp. We use arrows in Section to distinguish random variables of the forward (-+ and the reverse process (+))
Notation:
The standard diffusion modeling framework involves training a network, 8o (2 t). to learn the score-function, Va, log (((;)) at all levels f, of a stochastic process described by an Itô SDE:
dx = f(x, t) + g(t)dw,
where u is the standard Wiener process. To generate samples from the trained model, one can run the (unconditional) Reverse SDE, where the score-function approximated by the trained neural network. Given measurements y Azo + n, can sample from the distribution po( ly) by running the conditional Reverse SDE given by:
dæ (f(x,t) g" (t) (V æ. log pr (xx) + V log loyp(y))x:)) dt + g(t)dw.
As mentioned, Va, log p(y) an.) ) is intractable for general inverse problems. One of the most effective approximation methods is the DPS algorithm proposed by Chung al. [11]. DPS assumes that: p(y|z,) = (y) 2. == E(zo|<z) = (Ny = AE[z0 x E o.
(3)
Essentially, DPS substitutes the unknown clean image æo with its conditional expectation given the noisy input, E[z, lz.]. Under this approximation, the term v(3/)x becomes tractable.
The theoretical properties of the DPS algorithm are not well understood. In this paper, we analyze DPS in a linear model setting where the data distribution lives in a low-dimensional subspace, and show that DPS actually samples from p(zol y) (Section A. Then, we provide an algorithu (Section 2.1) and its analysis to sample from p( yy) using latent diffusion models (Section 3.2). Importantly, analysis suggests that our algorithm enjoys the same theoretical guarantees while avoiding the curse ambient dimension observed in pixel-space diffusion models including DPS. Using experiments (Section 4), show that our algorithm allows to powerful foundation models and solve linear inverse problems, outperforming previous unsupervised approaches without the need for finetuning.
"The source code is available at: //tttt://github http:///thhuu..com//ttttttttt/// and a web application for image inpainting is available at: /ttps://uggghggacccco///oa http:///w.c.../pppcc////////////

---

2.1 Method
In Latent Diffusion Models, the diffusion occurs in the latent space. Specifically, we train model s0(2;,)) t) to predict the score >x, log (e((),)), of a diffusion process:
d= = f(z,t)(ee) + g(t)dw,
(4)
where Zn ( for some encoder function E() IR + TR During sampling, we start with 27: we run the Reverse Diffusion Process and then we obtain clean image by passing 2o ~ (o(2o) 2r)
through a decoder D I Rd.
Although Latent Diffusion Models underlie some the most powerful foundation models for image generation, existing algorithms for solving inverse problems with diffusion models do not apply for LDMs. The most natural extension of the DPS idea would be to approximate p(y) (2)) with:
p z.) x p = D (z))),
(5)
Le., to approximate the unknown clean image Zo with the decoded version of the conditional expectation the clean latent 2y given the noisy latent ž₁. However, as we show experimentally in Section 4, this idea does not work. The failure of the "vanilla" extension of the DPS algorithm for latent diffusion models should not come as a surprise. The fundamental reason is that the encoder is a many-to-one mapping. Simply put, there are many latents 2g that correspond encoded versions of images that explain the measurements. Taking the gradient of the density given by (5) could be pulling 2, towards any of these latents 2o. potentially different directions. On the other hand, the score-function is pulling ž, towards a specific Zo that corresponds to the best denoised version of z;.
To address this problem, we propose an extra term that penalizes latents that are not fixed-points of the composition of the decoder- fundcionn with the intractable V log p(y| with:
Specifically, we approximate the
21, logp(y/z.) = Vz, 21 log ply uo - D (B|zo|z.[) +7+ V -EP()//)a)))) (6)
DPS vanilla extension
"goodness" of #o
We refer to this approximation as Goodness Modified Latent DPS (GML-DPS). Intuitively, we guide the diffusion process towards latents such that: i) they explain the measurements when passed through the decoder, and ii) they are fixed points of the decoder-encoder composition. The latter is useful to make sure that the generated sample remains on the manifold of real data. However, it does not penalize the reverse SDE for generating other latents žo long as (())) lies the manifold of natural images. Even in the linear case (see Section 3), this can lead to inconsistency at the boundary of the mask in the pixel space. The linear theory in Section 3 suggests that can circumvent this problem by introducing the following gluing objective. In words, the gluing objective penalizes decoded images having discontinuity the boundary of the mask.
V 21 Logp(y z,)= = V z. log p(y æ = D (B(z)lz1))
DPS vanilla extension
+ Y Vz, z1
E | 2.] - E(AT + A A)D(E//0)=)))
"gluing" of xo
The gluing objective is critical for our algorithm as it ensures that the denoising update, measurement- matching update, and the gluing update point to the same optima in the latent space. We refer to this approximation (7) as Posterior Sampling with Latent Diffusion (PSLD). In the next Section 3, we provide an analysis of these gradient updates, along with the associated algorithms.
Remark 2.1. Consider the optimization problem of projecting onto the measurements:
min |10
xol1
subject to Az, y.
In the linear setting, the optimal solution given by 2 -AAA (#g. AT (AAA)) Now further suppose that the measurement rows are orthogonal, i.e. AA" I, This condition holds for some natural linear inverse problems like inpainting. Suppose that want to update the latent vector 2, such that E[zol xx] E(z);; this ensures that the gradients

---

Algorithm 1: DPS
Algorithm 2: PSLD
Input: T, C (ās)
Input: T. 9. (m:) bwd (ā:) D.A.
Ør ~o
1 Zy ~ _/100 ()
for
T to
á + d))
3
4- #0( i)
áb Ta, (2; + (1 a.)8)
4
&o ya,
(#i + (1 a:)8)
10,
5
10,
x/-1 + yuae ai 12 + -- ô
الr 22 2 M 2o
T-
= t # - m/ v. 11y AD (20
1-01
V., 11 A(
8
end
x Vx 1 8(1" a A)D(E))1
return abo
end return ((2))
resulting from the two terms in (7) both point to the same optima in the latent space. Equivalently,
we want to solve the following minimization problem: ming, || |zy] E Exx))]l Substituting
11
(xa)) ((A", y + ( A Ato)) = E(A" y AA A))o). and B D ( z0 |ze)), we can thus interpret the gluing objective in (7) as a one step of gradient descent of this loss || |z|] {(x)) 1
with respect to 2, Note that, if there
no latent space, gluing would be equivalent
projection on the measurements, but now because of the encoder and decoder, it is not.
3 Theoretical Results
As discussed in Section 2, diffusion models consist of two stochastic processes: the forward and reverse processes, each governed by Itō SDEs. For implementation purposes, these SDEs are discretized over finite number of (time) steps, and the diffusion takes place using transition kernel. The forward process starts from Ty ~ p(z)) and gradually adds noise, i.e., H s+1 = VI 3r L + VB,e where 3e [0. 1] and Br 2 Bi-1 for 0, D..... T
The reverse process is initialized with E ~ N (0, Ia) and generates T t-1 = ne v t) + VBre. In the last step, He (z) 1. 1) is displayed without the noise.
In this section, we consider the diffusion discretized to two steps (f zi) )) and a Gaussian transition kernel that arises from the Ornstein- Uhlennbbk (OU) process. We choose this setup because it captures essential components of complex diffusion processes without raising unnecessary complications in the analysis. We provide principled analysis of Algorithm and Algorithm 2 in linear model setting with this two-step diffusion process under assumptions that guarantee exact reconstruction is possible in principle. A main result of our work is prove that in this setting we can solve inverse problems perfectly. As we show, this requires some novel algorithmic ideas that are suggested by our theory. In Section 4, we then show that these algorithmic ideas are much more general, and apply to large- scale real-world applications of diffusion models that use multiple steps (126, æi. -.. x ). where T 1000), and moreover do not satisfy the recoverability assumptions. We provide post-processing details of Algorithm 2 in Appendix C.I. All proofs are given in Appendix B.
3.1 Problem Setup
The goal is show that posterior sampling algorithms (such as DPS) can provably solve inverse problems in a perfectly recoverable setting. To show exact recovery, we analyze two-step diffusion processes in a linear model setting similar to [42, 7], where the images (w) IRd) reside in linear subspace of the form z = Swh, S E dxl wh E RR' and ay O. Here, S is a tall thin matrix with rank(S)= = - d that lifts any latent vector wh ~ A (0, Ii) to the image space with ambient dimension d. Given the measurements y = A + n, A E IRixd n E R', the goal is to sample from Po ( y) using a pre- pre-trained latent diffusion model. In the inpainting task, the measurement operator A is such that A A is diagonal matrix D(m), where m is the masking vector with elements set to 1 where data is observed and 0 where data is masked (see Appendix B for further details). Recall that in latent diffusion models, the diffusion takes place in the latent space of a pre-trained Variational Autoencoder (VAE). Following the common practice [41], we consider a setting where the latent vector of the VAE is k-dimensional and the latent distribution is standard Gaussian A (0, Ix). Our analysis shows that the proposed Algorithm provably solves inverse problems under the following assumptions.
5

---

Assumption 3.1. The columns of the data generating model S are orthonormal, i.e., S = I. Assumption 3.2. The measurement operator A satisfies (AS))((AS) > O.
These assumptions have previously appeared, e.g., [42]. While Assumption 3.1 is mild and can be relaxed at the expense of (standard) mathematical complications, Assumption 3.2 indicates that (AS)² (AS) is a positive definite matrix. The latter ensures that there is enough energy left in the measurements for perfect reconstruction. More precisely, any subset of coordinates exactly determines the remaining (d 1) coordinates of zy. The underlying assumption is that there exists solution and it is unique [42]. Thus, the theoretical question becomes how close the recovered sample this groundtruth sample from the true posterior. Alternatively, one may consider other types of posteriors and prove that the generated samples are close to this posterior in distribution. However, this does not guarantee that the exact groundtruth sample is recovered. Therefore, motivated by prior works [42, 71, we analyze posterior sampling in a two-step diffusion model and answer a fundamental question: Can a pre-trained latent diffusion model provably solve inverse problems in a perfectly recoverable setting?
3.2 Posterior Sampling using Latent Diff'usion Model
GML-DPS based (6), and PSLD based (7), displayed in Algorithm 2. We consider the case where the latent distribution of the VAE is in the same space as the latent distribution of the data generating model, i.e., k L, and normalize 7V 1 (as this is immaterial in the linear setting). In Proposition 3.3, we provide analytical solutions for the encoder and the decoder of the VAE.
In this section, analyze approximations:
Proposition 3.3 (Variational Autoencoder). Suppose Assumption 3.1 holds. For encoder & Rd + RR and a decoder D I2 + Rd, denote by C (ó. w) the training objective of VAE:
arg min L (b.w) = Bai-u [I (p): )):w) - ] + AKL (Ety, Ik)), 0:0
then the combination of E(0h; +) St To and D w( = S a minimizer of C (b,w).
Using the encoder E(bi; o) = St vo. we can use the analytical solution 0" of the LDM obtained in Theorem A.1. To verify that 0" recovers the true subspace (zb), we compose the decoder D w() Sio with the generator of the LDM, i.e., to = D (0+ x D (I+²T) St. Since b N~ I&) and S is the data generating model, this shows that 2y is sample from p p(z)). Thus we have the following.
Theorem 3.4 (Generative Modeling using Diffusson in Latent Space). Suppose Assumption 3.1 holds. Let the optimal solution of the latent diffusion model be
arg min Ez,
|1 (z)
2) zb) - po ( E20 2)}1"}
For fixed variance B > if no \ (z) 03 (z): 2). then the closed-form solution is 0- BI, which after normalization by vI-B and composition with the decoder D (Zo:w)
S. recovers the true subspace of p (.%)
With this optimal 0", we can now prove exact sample recovery using GML-DPS (6). Theorem 3.5 (Posterior Sampling using Goodness Modified Latent DPS). Let Assumptions 3.1 and 3.2 hold. Let Jj, Vj 1, 1,....? ,T denote the singular values of (AS)² T (AS), and let
0" arg min Ezg & [| (Zi(Z),
ps ( (2), 2))
Given a partially known image To ~ p(Z)), any fixed variance E (0, 1), then with the (unique) step size ni! = 1/2o,, = 1, 1,2...., ,,, the GML-DPS Algorithm (6) samples from the true posterior p(zōly) and exactly recovers the groundtruth sample, i.e., bng Tb
Theorem 3.5 shows that GML-DPS (6) recovers the true sample using LDM. This approach, however, requires the step size rj to be chosen coordinate-wist a specific manner. Also, multiple natural images could have the same measurements in the pixel space. This is reasonable concern for

---

LDMs due to one-to- me--to-many mappings of the decoder. Note that the goodness objective (Section 2.1) cannot help in this scenario because assigns uniform probability to many of these latents ž for which V= || b ]
O. These challenges motivate the gluing objective in Theorem 3.6. This crucial for two reasons. First, we show that it helps recover the true sample even when the step size 17 is chosen arbitrarily. Second, it assigns all the probability mass to the desired (unique) solution in the pixel space.
Theorem 3.6 (Posterior Sampling using Diffusion in Latent Space), Let Assumptions 3.1 and 3.2 hold. Let og Vj = 1....., denote the singular values of (AS)" (AS) and let
0t arg min R 7 || (Z)(E): () =b) no ( (Fo: 2))1
Given a partially known image To ~ p(Z)), any fixed variance 3 E (0,1), and any positive step sizes vg. =- 1,2, 1,2,..... the PSLD Algorithm 2 sumples from the true posterior pactorpppppp p(æoly) and exactly recovers the groundtruth sample, i.e., ty = TÚ--
The important distinction between Theorem 3.5 and Theorem 3.6 is that the former requires the exact step size while the latter works for any finite step size. Combining denoising, measurement- consistency (with scalar 17), and gluing updates, we have
So = - ~ |AD
v V
[T) - ATi (Ia - AT A)D( (1T))
When is chosen arbitrarily, then the third term guides the reverse SDE towards the optimal solution zh. When the reverse SDE generates the exact same groundtruth sample, i... ( (xx)) = Ty. then the third term becomes zero. For all other samples, it penalizes the reverse SDE. Thus, it forces the reverse SDE to recover the true underlying sample irrespective of the value of 17.
We draw the following key insights from our Theorem 3.6: Curse of ambient dimension: In order to run posterior sampling using diffusion in the pixel space, the gradient of the measurement error needs to be computed in the d-dimensional ambient space. Therefore, DPS algorithm suffers from the curse of ambient dimension. On the other hand, our algorithm uses diffusion in the latent space, and therefore avoids the curse of ambient dimension. Large-scale foundation model: We propose posterior sampling algorithm which offers the provision to use large-scale foundation models, and it provably solves general linear inverse problems. Robustness to measurement step: The gluing objective makes our algorithm robust to the choice of step size Furthermore, it allows the same (scalar) step size across all the coordinates of Tzg.
4 Experimental Evaluation
We experiment with in-distribution and out-of- distribution datasets. For in-distribution, duct our experiments on a subset of the FFHQ dataset [25] (downscaled 256 x 256 denoted
con-
Table 1: Quantitative super-resolution (using mea- surement operator from [32]) results on FFHQ 256 validation samples [25, 11]. We use PSLD with
by FFHQ 256). For out-of-distribution, we use images from the web and ImageNet dataset [17] (resized to 256 x 2566, denoted by ImageNet 256). To make fair comparison, we use the same vali- dation subset and follow the same masking strat-
Stable Diffusion. Table shows LPIPS (L).
Method
PSLD (Ours)
DPS [11]
2x
0.185
0.220
egy as the baseline DPS [11]. is important to note that our main contribution is an algorithm
3x
0.220
0.247
4x
0.233
0.291
that can leverage any latent diffusion model. We
test our algorithm with two pre-trained latent diffusion models: (i) the Stable Diffusion model that is trained on multiple subsets of the LAION dataset [44, 45]; and (ii) the Latent Diffusion model (LDM-VQ-4) trained on the FFHQ 256 dataset [41]. The DPS model is similarly trained from scratch for IM steps using 49k FFHQ 256 images, which excludes the first IK images used as validation set.
Inverse Problems. We experiment with the following task-specific measurement operators from the baseline DPS [11]: (i) Box inpainting uses a mask of size 128x 128 at the center. (ii) Random inpainting chooses a drop probability uniformly at random between (0.2, 0.8) and applies this drop
3 Phttps: httppp///wwwkkeeeeoooooedsssssss
imaggesss-ssssss---ee---- only
7

---

Table 2: Quantitative inpainting results on FFHQ 256 validation set [25, 11]. We use Stable Diffusion
v-1.5 and the measurement operators as in DPS [11]. As shown, our PSLD model outperforms DPS since it able to leverage the power of the Stable Diffusion foundation model.
Inpaint (random)
Inpaint (box)
SR (4x)
Gaussian Deblur
Method
FID (L)
LPIPS (1)
FID (1)
LPIPS (U)
FID (4)
LPIPS (L)
FID (4)
LPIPS (L)
PSLD (Ours)
21.34
0.096
43.11
0.167
34.28
0.201
41.53
0.221
DPS [11]
33.48
0.212
35,14
0.216
39.35
0.214
44.05
0.257
DDRM [26]
69.71
0.587
42.93
0.204
62.15
0.294
74.92
0.332
MCG [13]
29.26
0.286
40.11
0,309
87.64
0,520
101.2
0,340
PnP-ADMM [6]
123.6
0.692
151.9
0,406
66.52
0.353
90.42
0,441
Score-SDE [50]
76.54
0.612
60.06
0.331
96.72
0,563
109.0
0,403
ADMM-TV
181.5
0.463
68.94
0.322
110.6
0.428
186.7
0.507
probability to all the pixels. (iii) Super-resolution downsamples images at 4x scale. (iv) Gaussian blur convolves images with a Gaussian blur kemnel. (v) Motion blur convolves images with a motion blur kernel. We also experiment with these additional operators from RePaint [32]: (vi) Super-resolution downsamples images at 2x, 3x, and 4x scale. (vii) Denoising has Gaussian noise with a 0.05. (viii) Destriping has vertical and horizontal stripes in the input images.
Evaluation. We compare the performance of our PSLD algorithm with the state-of-the-ard DPS algorithm [11] on random inpainting, box inpainting, denoising. Gaussian deblur, motion deblur, arbitrary masking, and super-resolution tasks. We show that PSLD outperforms DPS, both in- distribution and out-of-distributte
datasets, using the Stable Diffusion y- 1-5 model pre-trained on the LAION dataset. We also test PSLD with LDM-VQ trained on FFHQ 256, to compare with DPS trained on the same data distribution. Note that the LDM-v4 is a latent- based model released prior to Stable Diffusion. Therefore, it does not match the performance of Stable Diffusion in solving inverse problems. However, shows the general applicability of our framework to leverage an LDM in posterior sampling. Since Stable Diffusion v-1.5 is trained with image resolution of 512 512, apply the forward operator after upsampling inputs to 512 x 512, run posterior sampling at 512 x 512, and then downsample images to the original 256 x 256 resolution for fair comparison with DPS. We observed a similar performance while applying the masking operator at 256 x 256 and upscaling 512 512 before running PSLD. More implementation details are provided in Appendix C.I.
Metrics. We use the commonly used Learned Perceptual Image Patch Similarity (LPIPS), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Metric (SSIM), and Fréchet Inception Distance (FID) metrics for quantitative evaluation.
Results. Figure 2 shows the inpainting results on out-of disribsttitttoon samples. This experiment was performed on commercial platforms that use (to the best of our knowledge) Stable diffusion and additional proprietary models. This evaluation was performed on models deployed in May 2023 and may change as commercial providers improve their platforms.
The qualitative advantage of PSLD is clearly demonstrated in Figures 2, 3, 4, 15 and 16. In Figure 5, we compare PSLD and DPS in random inpainting task for varying percentage of dropped pixels. Quantitatively, PSLD outperforms DPS in commonly used metrics: LPIPS, PSNR, and SSIM.
In our PSLD algorithm, Table 6 compares the quantitative results of PSLD with related works on random inpainting, box inpainting, super-resolution, and Gaussian deblur tasks. PSLD significantly outperforms previous approaches on the relatively easier random inpainting task, and it is better or comparable on harder tasks. Table draws comparison between PSLD and the strongest baseline (among the compared methods) on out-of-distribution images. Table shows the super-resolution results using nearest- neighbor kernels from [32] on FFHQ 256 validation dataset. Observe that PSLD outperforms -ftt--of-the-ar methods across diverse tasks and standard evaluation metrics.
Stable Diffusion vl.5 model and (zero-shot) test inverse problems.
In Table 3, compare PSLD (using (Q-4) and DPS random and box inpainting tasks with the same operating resolution (256 x 256) and training distributions (FFHQ 256). Although the LDM model exceeds DPS performance in box inpainting. it is comparable in random inpainting. As expected, using a more powerful pre-trained model such as Stable Diffusion is beneficial in
8

---

(a) Input
(b) Groundtruth
(c) Comm. Serv.
(d) Comm. Serv.
(c) PSLD (Ours)
Figure 2: Inpainting results in general domain images from the web (see Appendix C for image sources). Our model compared to state-of-art commercial inpainting services that leverage the same foundation model (Stable Diffusion v-1.5).
Table 3: Quantitative inpainting results on FFHQ 256 validation set [25, 11]. We use the latent diffiasion (LDM-VQ-4) trained on FFHQ 256. Note that in this experiment PSLD and DPS use diffusion models trained on the same dataset. As shown, PSLD with LDM-VQ-4 as diffusion model outperforms DPS in box inpainting and has comparable performance in random inpainting.
Inpaint (random)
Inpaint (hox)
Method
PSNR (f)
SSIM (t)
LPIPS (4)
PSNR (1)
SSIM (f)
I.PIPS (4)
PSLD (Ours)
30.31
0,851
0.221
24.22
0.819
0.158
DPS [11]
29.49
0.844
0.212
23.39
0.798
0.214
Table 4: Quantitative results of random inpainting and denoising FFHQ 256 [25, 11] using Stable Diffusion v-1.5. Note that DPS trained on FFHQ 256. The results show that our method PSLD generalizes well disto--distriuution samples even without finetuning.
Random inpaint denoise 0.00
Random inpaint + denoise a = 0.05
Method
PSNR (t)
SSIM (t)
LPIPS (1)
PSNR (t)
SSIM (t)
LPIPS (4)
PSLD (Ours)
34.02
0.951
0.083
33.71
0.943
0.096
DPS [11]
31.41
0.884
0.171
29.49
0.844
0.212
reconstruction-see Table This highlights the significance our PSLD algorithm that has the provision to incorporate powerful foundation model with extra training costs for solving inverse problems. Importantly, PSLD uses latent-based diffusion, and thus avoids the curse ambient dimension (Theorem 3.6), while still achieving comparable results to the of-te-of-thh-at method DPS [11] that has been trained on the same dataset. Additional experimental evaluation is provided in Appendix C.
5 Conclusion
In this paper, we leverage latent diffusion models to solve general linear inverse problems. While previously proposed approaches only apply to pixel-space diffusion models, our algorithm allows us to use the image prior learned by latent-based foundation generative models. We provide principled analysis of our algorithm linear two-step diffusion setting, and use insights from this analysis to design modified objective (goodness and gluing). This leads to our algorithm Posterior

---

e
Ce
e
Creundouta
DPS (9)
FSLD (Oura)
ta
de Gozzadrads
Ecl DPS (1)
PSLD (0un))
Figure 3: Left panel: Random Inpainting images from FFHQ 256 [25] using PSLD with Stable Diffusion v-1.5. Notice the text in the top and the facial expression the bottom Right panel: Block (128 128) inpainting, using LDM- -VQ model trained on FFHQ [25]. Notice the glasses the top row and eyes in the bottom row.
(a) Input
(b) Groundtruth
(c) DPS [11]
(d) PSLD (Ours)
Figure Inpainting (random and box) results on out-of-distribution samples, 256 256 (see Appendix C for image sources). We use PSLD with Stable Diffusson v-1.5 generative foundation model.
0.30
0.90
0.25
PSLD
LPIPS
PSNR
30
SSIM
0.85
0.20
0.15
28
DPS
0.80
DPS
PSLD
PSLD
0.10
26
0.75
20
10
60
80
20
40
60
80
20
40
60
80
Percentage dropped pixels
Percentage dropped pixels
Percentage dropped pixels
Figure 5: Comparing DPS and PSLD performance in random inpainting FFHQ 256 [25, 11], as the percentage of masked pixels increases. PSLD with Stable Diffusion outperforms DPS.
Sampling with Latent Diffusson (PSLD) that experimentally outperforms state-of-art baselines a wide variety tasks including random inpainting, block inpainting, denoising, destriping, and super-resolution.
Limitations. Our evaluation is based on Stable Diffusion which was trained on the LAION dataset. Biases in this dataset and foundation model will be implicitly affecting our algorithm. Our method can work with any LDM and we expect new foundation models trained on better datasets like [19] to mitigate these issues. Second, we have not explored how to use latent-based foundation models to solve non-linear inverse problems. Our method builds on the DPS approximation (which performs well on non-linear inverse problems), and hence we believe our method can also be similarly extended.
10

---

Acknowledgements
This research has been supported by NSF Grants 2019844, 2112471, AF 1901292, CNS 2148141, Tripods CCF 1934932, the Texas Advanced Computing Center (TACC) and research gifts by Western Digital, Wireless Networking and Communications Group (WNCG) Industrial Affiliates Program, UT Austin Machine Learning Lab (MLL), Cisco and the Stanly P. Finch Centennial Professorship in Engineering. Litu Rout has been supported by the Ju-Nam and Pearl Chew Endowed Presidential Fellowship in Engineering, Giannis Daras has been supported by the Onassis Fellowship (Scholarship ID: ZS 012-1/2022 212-1/2022-2023), the Bodossaki Fellowship and the Leventis Fellowship. We thank the HuggingFace team for providing us GPU support for the demo of our work.
References
[1] Brian D.O. Anderson. "Reverse-time diffusion equation models". In: Stochastic Processes and their Applications 12.3 (1982), pp. 313-326 (page 1).
[2] Marius Arvinte, Ajil Jalal, Giannis Daras, Eric Price, Alex Dimakis, and Jonathan I Tamir. "SSingle-hoot Adaptation using Score- Models for MRI Reconstruction". In: International
Society Magnetic Resonance in Medicine, Annual Meeting. 2022 (page 2).
[3] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. "Cold Diffusion: Inverting arbitrary image transforms without noise". In: arXiv preprint arXiv:2208. 200.092200999 (2022) (page 2).
[4] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. "Align your latents: High-resolution video synthesis with latent diffusion models". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 22563-22575 (page 3).
[5] Ashish Bora, Ajil Jalal, Eric Price, and Alexandros Dimakis. "Compressed sensing using generative models". " In: International Conference on Machine Learning. PMLR. 2017, pp. 537- 546 (page 1).
[6] Stanley H Chan, Xiran Wang, and Omar A Elgendy. "Plug-and-play ADMM for image restora- tion: Fixed-poin convergence and applications" In: IEEE Transactions on Computational Imaging 3.1 (2016), pp. 84-98 (pages 2, 8, 24, 27).
[7] Minshuo Chen, Kaixuan Huang. Tuo Zhao, and Mengdi Wang. "Score Approximation, Esti- mation and Distribution Recovery of Diffusion Models Low-Dimensional Data". In: arXiv preprint arXivv2302.07 0000000779//4 194 (2023) (pages 5, 6).
[8] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. "Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions". In: arXiv preprint arXiv:2209.1 1215 (2022) (page 1).
[9] Sitan Chen, Giannis Daras, and Alexandros G Dimakis. Crestoration-Degraaattii Beyond Linear Diffussons: A Non- Non-Asynon Aoomptosytii Analysis For DDIM-Type Samplers". In: arXiv preprint arXiv:2303.03388 (2023) (page 1).
[10] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. "Ilvr: Conditioning method for denoising diffusion probabilistic models". In: arXiv preprint arXiv:2108 02938 (2021) (page 2).
[11] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong
Chul Ye. "Diffusion Posterior Sampling for General Noisy Inverse Problems". In: The Eleventh International Conference on Learning Representations. 2023. URL: https: /ttp////pper.e..eee// (pages 1-3, 7-10, 15, 18, 20, 22, 24-31).
[12] Hyungjin Chung, Jeongsol Kim, and Jong Chul Ye. "Direct Diffusion Bridge using Data Consistency for Inverse Problems". In: arXiv preprint arXiv:2305. 022.335.999900 19809 (2023) (page 2).
[13] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. "Improving Diffusion Models for Inverse Problems using Manifold Constraints". In: Advances in Neural Information Processing Systems. Ed. by Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun (pages 2, 8, 24, 27,
Cho. 2022. URL: https: https:/ /owp://iee/e/eeeeee/mmoeme met:/forumiddnnJJjooo0 31).
[14] Giannis Daras, Yuval Dagan, Alexandros G Dimakis, and Constantinos Daskalakis. "Score- guided intermediate layer optimization: Fast langevin mixing for inverse problem"". In: arXiv preprint arXiv:2206. 2414004000 (2022) (page 2).

---

[15] Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G Dimakis, and Peyman Milanfar. "Soft diffusion: Score matching for general corruptions", In: arXiv preprint (XXix:229.054444 (2022) (page 2).
[16] Mauricio Delbracio and Peyman Milanfar. "Inversion by direct iteration: An alternative to denoising diffusion for image restoration". In: arXiv preprint 1rXiv:2303.14335 (2023)
(page 2).
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. "Imagenet: A large- scale hierarchical image database". In: 2009 IEEE conference on computer vision and pattern recognition. leee. 2009, pp. 248-255 (pages 21, 22, 27-29).
[18] Prafulla Dhariwal and Alexander Nichol. "Diffusion models beat gans on image synthesis". In: Advances in Neural Information Processing Systems 34 (2021), pp. 8780-8788 8794 (page 1).
[19] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. "DataComp: In search of the next generation of multimodal datasets". In: arXiv preprint arXiv:2304. 14108 (2023) (page 10).
[20] Jonathan Ho, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models". In: Advances in Neural Information Processing Systems 33 (2020), pp. 6840-6851 (page 1).
[21] Aapo Hyvärinen and Peter Dayan. "Fstimation of non-normalized statistical models by score matching." In: Journal of Machine Learning Research 6.4 (2005) (page 1).
[22] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. "Robust compressed sensing mri with deep generative priors". In: Advances in Neural Informa-
tion Processing Systems 34 (2021), pp. 14938- -1954 (pages 1, 2).
[23] Ajil Jalal, Sushrut Karmalkar, Alexandros G Dimakis, and Eric Price. "Instance-optimal compressed sensing via posterior sampling". In: arXiv preprint arXiv:2106. 11438 (2021)
(page l).
[24] Ajil Jalal, Sushrut Karmalkar, Jessica Hoffmaann, Alex Dimakis, and Eric Price. "Faimess for Image Generation with Uncertain Sensitive Attributes", In: Proceedings of the 38th Inter- national Conference on Machine Learning. Ed. by Marina Meila and Tong Zhang. Vol. 139, Proceedings of Machine Learning Research. PMLR, 18-24 Jul 2021, pp. 4721-4732. URL: https://proceediiii
mlr. press / j ala121b. html (page 1).
[25] Tero Karras, Samuli Laine, and Timo Aila. "A style-based generator architecture for generative adversarial networks". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, pp. 4401-4410 (pages 7-10, 24-28,37 31).
[26] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. "Denoising Diffusion Restora- tion Models". In: Advances in Neural Information Processing Systems (pages 1, 2, 8, 24, 27, 31).
[27] Bahjat Kawar, Noam Elata, Tomer Michaeli, and Michael Elad, "GSURE- -Bseed Diffussonn Model Training with Corrupted Data". In: arXiv preprint arXiv:2305.13122 (2023) (page 2).
(28) Bahjat Kawar, Gregory Vaksman, and Michael Elad. "SNIPS: Solving noisy inverse problems stochastically". In: Advances in Neural Information Processing Systems 34 (2021), pp. 21757-
21769 (page 31).
[29] Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. "Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation".. In: International Conference on Machine Learning. PMLR, 2022, pp. 11201-
11 11228 (page 1).
[30] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. "Audioldm: Text-to-audio generation with latent diffusion models". In arXiv preprint arXiv:2301. 12503 (2023) (page 3).
[31] Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. "Coherent Semantic Attention for Image Inpainting". In: 20/9 IEEE/CVF International Conference on Computer Vision (ICCV) (Oct. 2019). DOI: 10. 1109/ /it//.cc...// 2019. 00427.
URL: http: //dx doi. www.oogg///..... 1109 ICCV.
2019. 00427 (page 1).
[32] Andreas Lugmayr, Martin Danelljan, Andres Romero. Fisher Yu, Radu Timofte, and Luc Van Gool. "Repaint: Inpainting using denoising diffusion probabilistic models". In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2022, pp. 11461- 1 11471 (pages 7, 8, 30).
12

---

[33] Gary Mataev, Peyman Milanfar, and Michael Elad. "DeepRED: Deep image prior powered by RED". In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2019, pp. 0-0 (page 1).
[34] Xiangming Meng and Yoshiyuki Kabashima. "Diffusion model based posterior sampling for noisy linear inverse problems". In: arXiv preprint arXiv:2211.. 123443 (2022) (pages 27, 31). [35] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. "Pulse: Self- supervised photo upsampling via latent space exploration of generative models". In: Proceed- ings of the ieee/cvf conference on computer vision and pattern recognition. 2020, pp. 2437- 2445 (page 1).
[36] Gregory Ongie, Ajil Jalal, Christopher A Metzler, Richard G Baraniuk, Alexandros G Dimakis, and Rebecca Willett. "Deep learning techniques for inverse problems in imaging". In: IEEE Journal on Selected Areas in Information Theory 1.1 (2020), pp. 39-56 (page 1).
[37] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. "Con- text encoders: Feature learning by inpainting". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. Pp. 2536-2544 (page 1).
[38] Walter HL Pinaya, Petru-Daniel Tudosiu, Jessica Dafflon, Pedro F Da Costa, Virginia Fernan- dez, Parashkev Nachev, Sebastien Ourselin, and M Jorge Cardoso. "Brain imaging generation with latent diffusion models". In: Deep Generative Models: Second MICCAI Workshop, DGM4MICCAI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 22,
2022, Proceedings. Springer. 2022. pp. 117-126 (page 3).
[39] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. "Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation".
In: arXiv preprint arXiv:2008. 00951 (2020) (page 1).
[40] Yaniv Romano, Michael Elad, and Peyman Milanfar. "The little engine that could: Regulariza- tion by denoising (RED)". In: SIAM Journal on Imaging Sciences 10.4 (2017), pp- 1804-1844 (pages 31).
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. "High-resolution image synthesis with latent diffusion models". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 10684-10695 (pages 2, 3, 5,5,7, 7).
[42] Litu Rout, Advait Parulekar, Constantine Caramanis, and Sanjay Shakkottai. "A Theoretical Justification for Image Inpainting using Denoising Diffusion Probabilistic Models". In: arXiv (2023) (pages 5, 6, 15).
preprint arXiv:2302.00
[43] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-lmage Diffusion Models. 2022.
arxiv: 2111. 058266 [cs. CV] (page 31).
[44] Christoph Schuhmann. Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 4A1ON-400M: Open Dataset of CI.I--illered 400 Million Image-Text Pairs. 2021. arXiv: 2111 02114 (page 7).
[45] Christoph Schuhmann et al. LAION-5B: An open large-scale dataset for training next genera- tion image-text models. 2022. arXiv: 2210. 08402 [cs. Cv] (page 7).
[46] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. seruudoinneere-guuddee diffu- sion models for inverse problems". In: International Conference on Learning Representations. 2023 (pages 2, 31).
[47] Yang Song and Stefano Ermon. "Generative modeling by estimating gradients of the data distribution". In: Advances in Neural Information Processing Systems 32 (2019) (page l). [48] Yang Song and Stefano Ermon. "Improved techniques for training score-based generative models". In: Advances in neural information processing systems 33 (2020), pp. 12438-12448 (page 1).
[49] Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. "Score- -Bsee Generative Modeling through Stochastic Differential Equations". In: International Conference on Learning Representationss 2021 (page 1).
[50] Yang Song, Jascha Sohl- -icckiicst,,n, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. "Score-B -asee Generative Modeling through Stochastic Differential Equations". In: International Conference on Learning Representations (pages 8, 24, 27).
13

---

[51] Yu Takagi and Shinji Nishimoto. "High-resolution" image reconstruction with latent diffusion models from human brain activity".. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 14453-14463 (page 3).
[52] Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. -adggaandplla priors for model based reconstruction". In: 2013 JEEE Global Conference on Signal and Information Processing. IEEE. 2013, pp. 945-948 (page 1).
[53] Pascal Vincent. "A connection between score matching and denoising autoencoders". In: Neural computation 23.7 (2011), pp. 1661-1674 (page 1).
[54] Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pelle- grini. Yasumass Onoe, Sarah Laszlo, David Fleet, Radu Soricut, et al. "Imagen editor and editbench: Advancing and evaluating text-guided image inpainting" In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 18359-18369 (pages 1, 21).
[55] Yinhuai Wang, Jiwen Yu, and Jian Zhang. "Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model". In: The Eleventh International Conference on Learning Repre- sentations. 2023. URL:
NggNNNN (page 31).
[56] Jiahui Yu, Zhe Lin, Jimei Yang. Xiaohui Shen, Xin Lu, and Thomas Huang. "Free-Form Image Inpainting With Gated Convolution". In: 2019 IEEE/CVE International Conference on Computer Vision (ICCV) (Oct. 2019). DOI: 10. 1109/ iicc 2019. 00457.
URL: http:
//dx. doi wwooo/////// 10. 1101/1109/0/00 ICCV, 2019 00457 (page 1).
14

---

A Additional Theoretical Results
We elaborate on the structure of the measurement matrix A E Rixd In our setting, we are considering linear inverse problems. Thus, this matrix pixel selector and consists of subset of the rows from the d x d identity matrix (the rows that are present correspond to the indices of the selected pixels from the image To E Id). Given this structure, immediately follows that A AA is d x d matrix that has the interpretation of pixel selection mask. Specifically, AA" a x diagonal matrix ((m), where the elements of m are set where data (pixel) is observed and 0 where data (pixel) is masked. Without the loss of generality, we suppose that the first coordinates are known.
Notation and Measurement Matrix.
A.1 Posterior Sampling using Pixel-space Diffusion Model
We first consider the reverse process, starting with & ~ N (0, Ia). and borrow result from [42] to show that the sample &o generated by the reverse process is a valid image from p p(æ)).
Theorem A.1 (Generative Modeling using Diffusion Pixel Space, [42]). Suppose Assumption 3.1 holds. Let
0- arg min E Eag
|| ( (2d; 2). nr (ari (Zd, 2))
For a fixed variance O, if μe ( vb. 2)) = Owi Lio 2), then the closed-forn solution 0 is v1 3SS", which after normalization by 1/v11 B recovers the true subspace of p (zo).
Though this establishes that ănn generated by the reverse process is valid image from pl (())) it not necessarily a sample from the posterior p(z0ly) (y) that satisfies the measurements. To accomplish this we perform one additional step of gradient descent for every step of the reverse process. This gives us Algorithm 1, the DPS algorithm. The next theorem shows that the reverse SDE guided by these measurements recovers the true underlying sample".
Theorem A.2 (Posterior Sampling using Diffusion in Pixel Space), Suppose Assumption and Assumption 3.2 hold. Let denote by oj. Vj = .,, the singular values of (AS) (AS) and
0- - aar min E L0
| ( (zb, (). ne (zi (mb; 2?
Given partially known image To p(z)). a fixed variance B > O, there exists step size < 1/20, for all the coordinates of 2y such that Algorithm samples from the true posterior py and exactly recovers the groundtruth sample, i.e., Do Z-.
B Technical Proofs
This section contains proofs of all the theorems and propositions presented in the main body of the paper. For clarity, we restate the theorems more formally with precise mathematical details.
B.1 Proof of Theorem A.2
Theorem B.1 (Posterior Sampling using Diffuusion in Pixel Space). Suppose Assumption 3.1 and Assumption 3.2 hold. Let us denote by a = = the singular values of (AS) i.e. (AS)* = UEVT = UD E R V E B and
0" = arg min E [
Do A ) xb) - (
TO: (Z0 2)
0
'While the DPS Algorithm [11] uses scalar step size Çi at each step, this does not suffice for exact recovery. However, by generalizing allow different step per coordinate, can show sample recovery. Thus, in this section, we denote ? to be the step size at step and coordinate j. & Also note that the step index i is vacuous in this section, consider two-step diffusion process (i.e., i is always "I').
15

---

Suppose Db p(zb). Given measurements Azi and fixed variance 3 E (0, 1), there exists (c? 1/ for all the coordinates of x6 such that Algorithm I samples from the true posterior p(ar y) and exactly recovers the groundtruth sample, ie., To T.g
matrix step size Ç ((C)
Proof. Our goal is to show that xn 2y. where To returned by Algorithm 1. Recall that the reverse process starts with .1 ~ NN (0, Ia) and generates the following:
To = 0 <V ( 01/
= 0' <V 4; |ASS in 1112
SS . 2 (ASS*)² (ASO v)
55 2Ç88T A" (ASST y)
S 2ÇS8" ASS by + "ESSTATTy
= SST b 2ÇSST A" ASS + 2ÇSS Amd
= -SS 2ÇSS" A" ASS E + 2CSS" AT ASZI
Now, use the singular value decomposition of (AS)* (AS) with left singular vectors in U Rixk right singular vectors in V E R and singular values = [o....., in E D Thus, the above expression becomes
Fo SS 1 2<SUEV' S
= SST 3
= SS 2 - SUS S + SUSEV"
@ SST 2(SU))(C.)U"''
(8) SSTT 1 2481DOGIUTT000037377288877777713
S 2501))(()""""33""""000000""3""2"
SS b 3077
SS 5 2SU "
where (i) is due to Assumption 3.1 and (ii) uses Assumption 3.2. By choosing Ci as half the inverse of the non-zero singular values of (AS)² (AS), i.e., Çi 1/20, Vi I,..., k, we obtain
To SS - -UU SUU z
SS - +S To.
which completes the statement of the theorem.
B.2 Proof of Proposition 3.3
Proposition B.2 (Variational Autoencoder). Suppose Assumption 3.1 holds. For an encoder & Rd k and decoder D 1 + Rd, denote by L (db, w) the training objective of VAE:
arg min L (b. w) = Emg- ~p [|D(E )):w) - ]b|12] + AKL (O.
then the combination of E(00; 6) = ST To and (( = Si a minimizer of C (d,w).
"We use the term 'step size' in more general way than is normally used. In this case, the step size is pre-conditioning' positive definite matrix, whose eigenvalue magnitudes correspond the scalar step sizes per coordinate along an appropriately rotated basis. This general form needed and with carefully selected (unique) eigenvalues; otherwise the DPS algorithm fails to converge to the groundtruth sample. We will later see that for our PSLD Algorithm in Theorem 3.6, we can revert to the commonly used notion of step size (a single scalar), any finite step size (including single scalar common across all coordinates) suffices for proving recovery.
16

---

Proof. To show that the encoder E(zb: Ø) S and the decoder w() = S minimize the VAE training objective L (Ø, w), we begin with the first part of the loss, which is also called reconstruction error feveron (0, w). Substituting ( Φ) S nó and D (o:ww = S2 we have
Grecon (0, w) = Ra mp
DE T6: k):w) -
|| का. w) =
||S5 1
B
Using the fact that øy lives in a linear subspace, we arrive at
Crecon (Ø,w) = = BxOnD
[SS - Sz [
[||S
Szb [|] =0,
E ((
where (i) is due to Assumption 3.1. Now, we analyze the distribution loss. Note that the KL-
divergence between two Gaussian distributions with moments (MI, 01) and (#2, 02) is given by
+ of (m1- - p2)2
KL
02)) log
02
203
2
Since (aro) 8 x00 S
Szo Zo. the distribution loss becomes:
Ldist (Ø) = KL (Etp, I2) = KL (N (O, 1,) =0.
B.3 Proof of Theorem 3.4
Theorem B.3 (Generative Modeling using Diffusion Latent Space). Suppose Assumption 3. holds. Let the optimal solution of the latent diffusion model be
0- arg min Rgg 2 |1/2 (Z)
26) - 110 2O: 2)
For fixed variance B > O, if nD 2 E 2)) = oz (z E) then the closed-form solution is 0" = /1 BI, which after normalization by -T and composition with the decoder 'D (Zo; =
(z,,
S.6 recovers the true subspace of p (
Proof. In latent diffusion models, the training is performed in the latent space of pre-trained VAE. If the VAE is chosen from Proposition 3.3, then the training objective becomes:
minE
(( nn
( (8(x)), +)) ||]
20.
Esg z
no ( ( 2))1
Ez,- z
= E
|| ozi (Z), 1]|
E
0 (z)
3 / v )
k
2 0.¹ / - 3 7 vB vB))² 2
E
17

---

where o denotes the tth row of matrix O. The solution of this regression problem is given by?
0:
20 - 3 -+ (20VI-B+EVD)
(2oVi-
Ezoo. Zu,i (2oVI- - - E e//3]
E
(2oVI- 3 3 + vVB)
11 +3++
Bxris
20,i 3 +
(E(Z)) 1- - 3 + evB) (E(0)) + CVB) T7
Exzg,e (S(zo): (E(Z0)i c evB)]
E ma.c
(E(SZ)) V1- 3 + E B) (E(SZ)) 3 E vB/) v3) '"] Exaie (8(SZ0): (E(SZO\ 3 + + EV3)
20.e
= (ST - vV/ (S" 1-
- 3 + "]
Ezo.c [(ST (ST EV [3]
Using Assumption 3.1, the above expression simplifies to
3 + + (20VII 1-ovi-A0 - B + <//B =+ 0-= E ( (20VI- 3
Ezo,e (2o)s
2o z0VI- - 1 + +EVB)
= E 20.e
(1- -
(1-}3)z02) =0 + 206" VB(1 - 3) + E v 3(1 - B) + BEE
-1 Ezo,e (20)) (zo - + +
= =[(1= B) E [2000]] + E Izne" T
(3(1-3)
- 3) E [ezØ] VBL - +B B E [EE") TT
E
((00) (20V1- - B + eV/)
x Eza,c
= =[(u-9))
B)
+
Ble V - 3) + E zu
E
E
E [zo]
vB(1 3) + +.x
20
x Ez,,c
(zo) (20V - B + </3)]
where the last step uses the fact that zu and independent Gaussian random vectors with zero mean and unit covariance. Simplifying further, we arrive at
3) + 3Tk] Exz.e ((0)) (20V1- B (+EEE)))))))))) EVDr 0: [(1
(ZoVi-B+eV))
E Ezo.c
(zo); 20 (20V1- B + +ev3]]
Ezo (zo) -
+ Eza.c (zo)ie 13
= Ezo
(20): - 3 + Ezg [(z0)i] E. [e] VB.
The final step follows from independence of z0 and e. Since 2o and are also N (0, Ix), we get
0 = Ez (zo)) 1- J.....,,, o, 11-,,0,000 B,0,....0 T
where the 4th coordinate is v1 - B and zero everywhere else. Therefore, stacking all the rows together, we get 0* /1 gIg. which after normalization by 1/v1- - 3 gives the desired result.
Next, we show that 0- recovers the true subspace of T p (m)). When composed with the
decoder of VAE, the generator of the LDM gives to D (0 Σ D (Ix S².. Since S N (0, IR), this completes the statement of the theorem.
B.4 Proof of Theorem 3.5
Recall that the the latent-space GML-DPS (6) algorithm (based on the pixel-space DPS algorithm [1I]) has three key steps. In the first step, it uses the normalized closed-form solution obtained in
For ease of notation, we drop the forward arrow in the rest of this proof.
18

---

Theorem 3.4 to perform one step of denoising by the reverse SDE. In the second step, it runs one step of gradient descent satisfy the measurements in the pixel space. Finally, it takes one step of gradient descent on the goodness objective, which acts as regularizer to ensure that the reconstructed image lies on the data manifold.
This can be formalized as:
2 = 01 - n |A ( v122
(8)
So = arg min _ E(D )))
In practice, solving (9) can be difficult, and can be approximated via gradient descent. In our analysis however, we analyze the exact system of equations above, as (9) has closed-form solution in the linear setting.
Theorem B.4 (Posterior Sampling using Goodness Modified Latent DPS). Suppose Assumptions 3.1 and Assumption 3.2 hold. Denote by {oz}ー=1 the singular values of (AS)"(AS) i.e.
(AS))(AA) UEUT =UD(o)U", U kxk, and ler
0" arg min Bz,
||
| 2)
no (Z) 14
2)]1
Suppose Eo p (.). Given measurements = Azi and any fixed variance B (0, 1), then with the (unique) step size (1/2) D(.) Th f 1/20, the GML-DPS algorithm (6) samples from the true posterior pl and exactly recovers the groundtruth sample, i.e. to x
Proof. We start with the measurement consistency update (8) and then show that the solution obtained from (8) is already a minimizer of (9). Therefore, we have
2 0' V (AA((Z((E))) y 2
-1 v [IAD(II y
1 17 V [AStT) 21/
S 1) V (ASE)) 21/
( S V |ASS v
S STST AT (ASE) 9)
S 2nST AT ASE 2nS A
= S 2nST AT AS + +2ns AT Az
S 2TST AT AS -2nST AT ASEi
where (i) is due to Assumption 3.1. By Assumption 3.2. (AS) (AAS) is a positive definite matrix and can be written USUT:
2/ b 2 + + zे
5 UU' UU"
Σ =U
Σ
+2U
= Σ 1 2UD(m @ U b 1 DU @ U
Since n/} 1/20,, the above expression further simplifies to
- + z.
Next, we show that 2 is already minimizer of (9). This is direct consequence of the encoder- decoder architecture of the VAE: 8(D( = S S = % Hence, || E(D(
and
19

---

f
consequently 2 = ~
30 E(D(2)))
z6. Thus, the reconstructed sample becomes
-o D( Szi F-.
-
[ - E
= for all Z it is evident that the goodness objective cannot rectify the error incurred in the measurement update (8). For this reason, GML-DPS algorithm (6) requires the exact step size to sample from the posterior.
Furthermore, as
Beyond the linear setting, we also refer to Table 5 for experiments supporting this result.
B.5 Proof of Theorem 3.6
Different from GML-DPS, PSLD Algorithm 2 replaces the goodness objective (6) with the gluing objective (7), which can be formalized as:
V |AD( - 2/1 = 00 2
(10)
S/ arg min
1/40 - E(A" AZ + A A" A)DE
(11)
We again remind that solving the minimization problem (11) is hard in general, and can be approxi- mated by gradient descent as typically followed in practice [11]. However, in a linear model setting, (1I) has a closed-form solution which we derive prove exact recovery.
Theorem B.5 (Posterior Sampling using Diffusion in Latent Space), Let Assumptions 3.1 and 3.2 hold. Let og Vj- = 1, ,....,, r denote the singular values of (AS) (AA)) and let
0 arg min Ez, 2 [| (zi 2))= ps (zl (Zo. m)-
Suppose 2o ~ p(o)). Given measurements y = Aw any fixed variance 3 E (0, 1), and any positive step sizes vi, = 1,2,...., the PSLD Algorithm 2 samples from the true posterior p(zoly) and exactly recovers the groundiruth sample, i.e., 35 = zo
Proof. Following the proof in Appendix B.4, we have
1/1 2 1/V v. |IA
y
I - V AD(()) y 2
5 V AASEi - y1/2
= S 21SST AT (AS - y)
2 2T/ST AT ASti + + A ASZ0
Σ -2/ST AT ASZ - AT ASE
We use the above expression to derive a closed-form solution to the minimization problem (11):
0
V
(AAA (Ia A" A))))22)
ASE - A)S 2
V
1/4
-
AT
V
17
S
ASE S - AT
V
1
1/0
ST ASE - ST + ST AT AS
2 (I ST + St A AS) -5
5t AT ASE S + S A- AS
= 2S AS
AS (ST AT AS - AT
where the last step is due Assumption 3.1. Thus, we have
2/ = arg min
1/4 E(A A + (Ia - AT 2 2b.
20

---

which produces t0p = D = D = Sz =
It is worth highlighting that PSL.D exactly recovers the groundtruth sample inrespective of the choice of the step size whereas GML--DPS requires the step size to be exactly r = (1/2)UD(m.)U'.
C
Additional Experiments
C.1 Implementation Details
For inpainting tasks, we note that the PSLD sampler generates missing parts (by design of our gluing objective) that are consistent with the known portions of the image, i.e., æn AA Azo (Ia A)D ()). This is different from the DPS sampler, which generates the whole image which may not match the observations exactly. In other words, in the last of step of algorithm, the observations are glued onto the corresponding parts of the generated image, leaving the unmasked portions untouched [54]. This sometimes creates edge effects which processing the glued image through the encoder and decoder of the SD model, i.e. running one last step of our algorithm. Figure 2 illustrates that gluing the observations in commercial services still leads to visually inconsistent results (e.g. head in top row) unlike our method.
then removed by post-
For all other tasks, such as motion deblur, Gaussian deblur, and super-resolution, this last step is not needed, as there is no box inpainting, i.e., kng = D ( Furthermore, we use the same measurement operator A and its transpose A² as provided by the DPS code repository However, since Stable Diffusion v1.5 generates images of size 512 x 312 resolution and DPS operates at 256 x 256, we adjust the size of the kernels used in PSLD to ensure that both the methods use the same amount of information while sampling from the posterior. During evaluation, we downsample PSLD generated images from 512 x 512 to 256 x 256 to compare with DPS at the same resolution.
We run Algorithm with Stable Diffusion version 1.5 as the foundation model". We use fixed 1 and y 0.1. Since we study posterior sampling of images without conditioning on tex/ inputs, we pass an empty string to the Stable Diffusion foundation model, which accepts texts as input argument. For better performance, we recommend using the latest pretrained weights.
PSLD (Stable Diffusion-V1.5
PSLD (LDM-VQ-4 ): This is the same sampling algorithm as before but with a different latent diffusion model, LDM-VQ-4"0 which contains pretrained weights for FFHQ 256 and large-scale text-to-image generative ! We keep the hyperparameters same (1) and 0.1). For each task, we provide with these two available in the same repository.
details in our codebase 13, Although we have tested our framework wwwfisiionmmodeeee one may experiment with other antent-diffusin modees
DPS: We use the original source code provided by the authors
OOD images are sourced online:
1. Figure the original images are generated by Stable Diffusion v-2.1 15
2. Figure 2 first row: Walking example from the web.
3. Figure 2 second row, Obama-Biden image from the web.
4. Figure 2 third row, Fisherman from ImageNet 256 [17].
5. Figure 4 first row: Racoon image from the web.
6. Figure 4 second row: Fisherman from ImageNet 256 [17].
7. Figure 15: Celebrity face from the web.
http:p//gtttaaooo.ee/////////aaa http:////.....aaaoooooo./oooaaaa Aaffuusiooa/mmesuurmmntt....
10 hht.pp///ggtaaao.ococ/o///tttttt
15 httpp//whhggggaoocoooeee///lllld
21

---

PSLD Image Inpainting
irgitringggyyy Toterfe Dinarin
vilend andu doferm Lnicia seeeedd ports
Ouinghendtinn In
wwrggenttagaaaiee g
Yourprono Decin enotyrto potaroo
Inainst
Figure 6: Results from the web application of our PSLD algorithm, 512 x 512. The original image (1) is generated by Stable Diffusion v-2.1 with the prompt." "A dinner date between a robot couple during sunset".
C.2 Additional Experimental Evaluation
Here, we provide additional results to support our theoretical claims on various inverse problems.
Figures 6, 7, 8, and 9 show the inpainting results of user defined masks obtained from PSLD inpainting web demo. Note that the foundation model used in this demo a generic model. For better performance on specific images, recommend finetuning the foundation model this class and then running posterior sampling using our web demo: https: hatpss::///ggnnaaaaa co/ /sspaccss/ PSLD/PSLD.
Figure 10 and 11 illustrate super-resolution (4x) of in-distribution samples from the validation set of FFHQ 256. Observe that the samples generated by DPS far from the groundtruth sample. On the other hand, the samples generated by PSLD closely capture the perceptual quality of the groundtruth sample. In other words, one may identify (b) and (c) images of two different individuals, whereas (b) and (d) of the same individual. We attribute this photorealism of our method to the power of Stable Diffusion foundation model and the ability to use the knowledge of the VAE encoder-decoder in the gluing objective.
In addition, we test out-of-distributie samples from ImageNet [17] validation set. Figure 12 and Figure 13 show the results motion deblur and Gaussian deblur, respectively. By leveraging the foundation model Stable Diffusion v1.5, our PSLD method clearly outperforms DPS [11] in the general domain. Further, Figures 14, 15, and 16 show reconstruction of general domain samples for random inpainting, super-resolution and destriping tasks, respectively. In all these tasks, the samples generated by PSLD are closer to the groundtruth sample than the ones generated by DPS. Figure 17 shows the results image colorization. Table 5 and Table 6 show the quantitative results. Table 7 draws comparison between the latent-DPS and PSLD algorithms, and shows that the PSLD objective enhances the reconstruction performance.
In Table we compare the runtime and NFES of PSLD with prior works. PSLD-SD (trained on LAION-5B) takes 776 s to generate 512x512 12 images. To compare with other methods that generate 256x256 images, we divide our runtime by All the other methods use diffusion models trained on FFHQ and produce 256x256 images.
22

---

PSLD Image Inpainting
hurdeoolllfioe orpe kig XX
Deel
Figure 7: Results from the web application of our PSLD algorithm, 512 x 512. The original image (1) is generated by Stable Diffusion v-2.1 with the prompt," "A panda wearing a spiderman costume".
PSLD Image Inpainting
Hum Mumbrrtti of iuaiondtas kup 301
41
Ohing phmmeiaaa leg 201
Figure 8: Results from the web application of PSLD algorithm, 512 512. The original image (1) is generated by Stable Diffusion v-2.1 with the prompt," "A teddy bear showing stop sign at the traffic".
23

---

PSLD Image Inpainting
Cabut
Rumber of ionooioi 200
Capu
Rangtaate 30-11
Gainghemets tng
sanghemees grarlng-
Figure 9: Results from the web application of our PSLD algorithm, 512 x 512. The original image (1) is generated by Stable Diffusion v-2.1 with the prompt,", "A cute dog playing with toy teddy bear on the lawn".
Table 5: Quantitative random inpainting results FFHQ 256 validation set [25, 11]. We use Stable
Diffusion (v1.5) trained on LAION.
Inpaint (random)
SR (4x)
Gaussian Deblur
Method
PSNR (1)
SSIM (f)
PSNR (1)
SSIM (t)
PSNR (t)
SSIM (1)
PSLD (Ours)
33.71
0.943
30.73
0.867
30.10
0.843
GML-DPS (Ours)
29.49
0.844
29.77
0.860
29.21
0.820
DPS [11]
25.23
0.851
25.67
0.852
24.25
0.811
DDRM [26]
9.19
0.319
25.36
0.835
23.36
0.767
MCG [13]
21.57
0.751
20.05
0.559
6.72
0.051
PnP-ADMM [6]
8.41
0.325
26.55
0.865
24.93
0.812
Score-SDE [50]
13.52
0.437
17.62
0.617
7.12
0.109
ADMM-TV
22.03
0.784
23.86
0.803
22.37
0.801
D Additional Discussion
Curse of ambient dimension: DPS [11] suffers from the curse of ambient dimension because in this method, gradients are computed in the pixel space with dimension However, latent- based methods such as PSLD compute gradients in the latent dimension k, and hence the computation more efficient. Furthermore, applying the chain rule on VAE and running diffusion in the latent space less expensive than running diffusion in pixel space directly. In practice, the computational complexity of Stable Diffusion model (~ 4GB) is higher (roughly 6 times) than the computational complexity of the encoder-decode model (~ 700MB. B). Therefore, applying the chain rule in the encoder-decoder and running diffusion in the latent space is less expensive than applying diffusion models in the pixel space directly.
24

---

(a) Input
(b) Groundtruth
(c) DPS [11]
(d) PSLD (Ours)
Figure 10: Super-resolution results on images from FFHQ 256 [25, 11] (in distribution).
25

---

(a) Input
(b) Groundtruth
(c) [11]
(d) PSLD (Ours)
Figure 11: Super-resolution results FFHQ 256 [25, 11] (in distribution).
26

---

(a) Input
(b) Groundtruth
(c) DPS [11]
(d) PSLD (Ours)
Figure 12: Motion deblur results on ImageNet 256 [17] (out-of--istrriuutio)))
Table 6: Additional quantitative results on FFHQ 256 validation set [25, 11].
(4x)
Gaussian Deblur
Method
FID (L)
LPIPS (L)
FID (L)
LPIPS (L)
PSLD (Ours)
34.28
0.201
41.53
0.221
DPS [11]
39.35
0.214
44.05
0.257
DDRM [26]
62.15
0,294
74.92
0,332
MCG [13]
87.64
0.520
101.2
0.340
PnP-ADMM [6]
66.52
0.353
90.42
0,441
Score-SDE [50]
96.72
0.563
109.0
0,403
ADMM-TV
110.6
0.428
186.7
0.507
SR (4x)
Gaussian Deblur
Method
PSNR (1)
SSIM (t)
PSNR (t)
SSIM (1)
PSLD (Ours)
30.73
0.867
30.10
0.843
GML-DPS (Ours)
29.77
0.860
29.21
0.820
DMPS [34]
27.63
25.41
DPS [11]
25.67
0.852
24.25
0.811
DDRM [26]
25.36
0.835
23.36
0.767
MCG [13]
20.05
0.559
6.72
0.051
PnP-ADMM [6]
26.55
0.865
24.93
0,812
Score-SDr [50]
17.62
0.617
7.12
0,109
ADMM-TV
23.86
0.803
22.37
0.801
27

---

(a) Input
(b) Groundtruth
(c) DPS [11]
(d) PSLD (Ours)
Figure 13: Gaussian deblur results ImageNet 256 [17] (out-of-distribution))
Table 7: Latent-DPS and PSLD methods evaluated on FFHQ 256 validation set [25, 11]. We use the latent diffussio (LDM-VQ-4) trained FFHQ 256. Latent-DPS is special of PSLD algorithm when 7 = 0.
Inpaint (box)
Method
PSNR (t)
SSIM (t)
LPIPS (L)
PSLD
24.22
0.819
0.158
latent-DPS
17.58
0.780
0.21
28

---

(a) Input
(b) Groundtruth
(c) DPS [11]
(d) PSLD (Ours)
Figure 14: Random inpainting results ImageNet 256 [17] -out-f--distibbution))
29

---

(a) Input
(b) Groundtruth
(c) DPS [11]
(d) PSLD (Ours)
Figure 15: Super-resolution (using nearest neighbor kernel from [32]) results on out-of-distribution samples from the web, 256 x 256 (see Table for LPIPS of these images).
Input
(b) Groundtruth
DPS [11]
PSLD (Ours)
Figure 16: Destriping results on out-of-distribune samples from the web, 256 x 256. (Top row) Horizontal destriping: LPIPS PSLD=0.244 and DPS [11]=0.613. (Bottom row) Vertical destriping:
LPIPS of PSLD-0.255, DPS [11]-0.597.
30

---

(a) Input
(b) Groundtruth
(c) DPS [11]
(d) PSLD (Ours)
Figure 17: Additional colorization results on images from FFHQ 256 [25, 11j. PSLD generates
photo-realistic color, whereas DPS [11] generates overly saturated images.
Table 8: Runtime (top) and NFEs (bottom) of different posterior sampling algorithms. Runtimes are computed for the super-resolution task.
Method
Runtime (s)
PSLD-LDM
187.00
PSLD-LDM (LAION-400M)
190.00
PSLD-SD (LAION-5B)
194.25
DMPS [34]
67.02
DPS [11]
180.00
DDNM+ [55]
18.5
DDRM [26]
2.15
MCG [13]
193.71
Method
NFEs
PSLD (Ours)
100 to 1000
DPS [11]
1000
DDRM [26]
20
RED [40]
500
IIGM [46]
20 to 100
Palette [43]
1000
Regression
1
SNIPS [28]
1000
31


Reviews:
[{"review_id": "SikCjpZCr9", "reviewer": "Reviewer_vbWT", "summary": "This paper investigates the use of diffusion models for solving inverse problems. While many recent papers have explored diffusion models for inverse problems, to the best of my knowledge, this study is the first to propose a method for inverse problems with latent diffusion. One advantage of latent diffusion is its lower computational demand.\n\nThe main challenge in solving inverse problems with diffusion models lies in computing the intractable likelihood term $\\nabla \\log p_{x_t}( y | x_t )$, where $x_t$ represents a noisy point in pixel-space. Recently, the DPS algorithm was introduced to approximate this term as $\\nabla \\log p_{x_t}( y | \\mathbb{E}[x_0| x_t] )$. The first contribution of this paper is to extend the DPS framework to a latent diffusion model, approximating the intractable likelihood $\\nabla \\log p_{x_t}( y | z_t )$, where $z_t$ denotes a noisy point in latent space, with $\\nabla \\log p_{z_t}( y | \\mathbb{E}[z_0| z_t] )$ along with an additional term that enforces $\\mathbb{E}[z_0| z_t]$ to be a fixed point of the autoencoder (GML-DPS). Furthermore, the latter term is modified to enforce consistency in the measurements (PSLD). The empirical performance of the PSLD algorithm is compared with the standard DPS in high-dimensional reconstruction tasks, and improvements over DPS are demonstrated.\n\nThese three algorithms (DPS, GML-DPS, PSLD) are theoretically studied in the case of a data distribution corresponding to a Gaussian supported on a low-dimensional subspace, perfect knowledge of the subspace, zero noise in the measurements, and a measurement matrix that is bijective over the subspace. Specifically, this implies that the inverse problem can be exactly solved given the measurements. Finally, the sampling processes are two-step diffusion processes. The authors demonstrate that all three algorithms precisely recover the target signal, and PSLD exhibits robustness to variations in the specification of the step sizes.", "questions": "Some typos and questions:\n\n- In Section 2, the notation can be confusing. To clarify, it is suggested to use $x_0^*$ to denote the target signal and distinguish it from the first sample of the backward diffusion ($x_0$).\n\n- For Equations (3), should it be $\\hat{x}_0 = \\mathbb{E}[x_0 | x_t]$?\n\n- In Algorithm 1: line 1 what is $\\mathcal{T}$?\n\n- Equations (7) should include a $\\log$ term for the DPS vanilla extension. Additionally, as shown in the supplementary material, $\\mathcal{A}^\\top$ should be added to (7) and line 8 of Algorithm 2.\n\n- In Algorithm 2, it appears that the unknown target $x_0^*$ is needed, but in reality, the algorithm only requires the measurements $y$. It is advisable to modify the algorithm to reflect this (as in Algorithm 1).\n\n- Line 142 of the Theoretical section introduces the noisy inverse problem as $y = A x_0 + \\sigma_y n$.  However, it should be noted that exact recovery is not possible in this case and it seems that the theoretical results are proven under the assumption of $\\sigma_y = 0$, where the problem can be solved exactly. The authors should explicitly address these points to ensure clarity.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Solving inverse problems with (large) latent diffusion models is a relatively underexplored area, and this paper fills a gap in the existing literature. In particular, the authors demonstrate how their methods can be used with SOTA latent-based foundation models such as stable diffusion. The empirical results presented are promising and demonstrate the potential of this approach. \n\nAdditionally, the authors offer some theoretical guarantees, albeit in a limited setting. These guarantees are valuable and noteworthy, considering the field's scarcity of rigorous results, even in simple toy examples.", "weaknesses": "\nThe main weaknesses of the paper lie in the experimental evaluations and results. Specifically, all experiments are conducted on subsets of the FFHQ dataset and images sourced from the internet. The authors evaluate the PSLD algorithm on two latent models: LDM-VQ-4, trained on FFHQ, and Stable Diffusion, trained on a significantly larger dataset (LAION). However, the comparisons are made between PSLD on these two latent models  (proposed) and DPS (baseline) on a standard diffusion model trained *exclusively* on FFHQ.\n\nThe authors demonstrate improved performance of PSLD on the (latent) Stable Diffusion model compared to DPS on the standard diffusion model. However, when employing the latent model LDM-VQ-4 trained solely on FFHQ, the improvements are marginal (Table 3).\n\nAs a result, it becomes challenging to ascertain whether the observed improvements stem from algorithmic enhancements or simply the utilization of superior models trained on larger datasets, which may also account for the enhanced out-of-distribution performance.\n\nFurthermore, the experiments do not shed light on the improvement attributable to the addition of the \"gluing\" term in comparison to the DPS's vanilla extension term. One might hypothesize that the \"gluing\" term could be employed in DPS (by removing the encoder-decoder), potentially enhancing the algorithm's performance.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer vbWT\n\nDear Reviewer vbWT,\n\n\nThank you for highlighting the fact that our study proposes the **first method** for inverse problems with latent diffusion models. We also thank you for your comment that the **theoretical results are valuable and noteworthy** and the **empirical results are promising**.\n\nBelow, we respond to your remaining comments and questions.\n\n(1) **The main weakness is that all experiments are conducted on subsets of FFHQ and images from the internet.** To make a fair comparison with the baselines, we use the same experimental setting and evaluate on the same subsets of FFHQ as the baselines. In addition, we show improved performance on out-of-distribution images from the internet. We view this as a strength because we use the same generative model for both the datasets, whereas prior works (including DPS) require dataset specific generative models.\n\n(2) **We only show marginal improvement of PSLD (LDM-VQ-4) over DPS in Table 3.**\nOur goal is to develop a framework to leverage the power of pretrained latent models, such as Stable Diffusion. In this table, our focus is not to maximize the margin that we beat the previous state-of-the-art DPS. The fact that we can still obtain results comparable to (or better than) the state-of-the-art DPS using an LDM (see Table 3) indicates that we do not loose a lot of information by shifting the diffusion process to the lower-dimensional latent space. Latent diffusions are much faster, and also our method\nunlocks the potential of using pre-trained LDMs as pointed out by Reviewer yVXi. Besides, Table 2 (also the newly added results) shows that PSLD can be better than DPS by simply scaling these LDMs, which is a common practice adopted by commercial service providers.\n\n(3) **Is our Improvement due to better algorithm or better generative models due to larger training dataset?**\nWe would like to clarify that the significant gain of PSLD over DPS (both in-distribution and out-of-distribution) is partly due to the fact that Stable Diffusion has been trained on a significantly larger dataset. However, it is important to note that none of the existing posterior sampling algorithms could leverage this pre-trained latent diffusion model for inverse problems. Another important aspect is the ability of PSLD to use the same generative model for several downstream tasks on FFHQ, ImageNet, and random images sourced from the web. On the contrary, prior works (including DPS) require  dataset specific generative models, which limits their application to general domain images.\n\n(4) **Is our improvement attributable to gluing term, compared to the DPS vanilla extension?**\nAs we discuss in Section 2.1, the vanilla extension of DPS fails due to many-to-one mapping of the encoder. Still, we believe this is not a fair comparison with DPS because the authors originally built DPS for a pixel-space diffusion model (not a latent one). Making DPS approximation work in the latent-space requires non-trivial extensions, which we discuss in Section 2.1, Section 3.2 (**Theorem 3.4**), and Section 3.3 (**Theorem 3.7 and 3.8**).  We also provide experimental results in Appendix B.2 (Table 5) and newly added results on super-resolution and Gaussian deblur tasks (please see the attached pdf).\n\n(5) **One may hypothesize that including gluing term in DPS may enhance its performance.**\nNaively gluing in pixel-space will create visible edges separating inpainted pixels from the observed ones. Still, the gluing term in pixel-space DPS will provide guidance through an extra gradient term $\\nabla_{\\mathbf{x}_i}||A^T\\mathbf{y} - \\hat{\\mathbf{x}}_0||_2^2$. This is an interesting direction for future research, but not within the scope of our paper since our goal is to solve inverse problems using latent diffusion models.\n\n(6) **Typos and questions.** We will correct all the typos in the revised version, thank you for your careful reading.\nFor equation (3), it should be $\\hat{x}_0 = \\mathbb{E}[x_0|x_t]$. In Algorithm 1: $\\mathcal{T}$ in line 1 should be the standard normal distribution $\\mathcal{N}$. Since our algorithm only requires $\\mathbf{y}$, we will modify the **Input** in Algorithm 2 similar to DPS as suggested by the reviewer. **(Line 142)** As correctly pointed out by the reviewer, exact recovery is possible when $\\sigma_y=0$ (the setting we consider), but not when $\\sigma_y>0$. We will clarify this in the revision.\n\n\n### Concluding Remark\n\nPlease let us know if the clarifications and additions suitably address your concerns. We are happy to address any remaining points during the discussion phase."}, {"review_id": "AiSzo0vBCv", "reviewer": "Reviewer_yVXi", "summary": "While the preivous methods have focused on solving linear inverse problems based on diffusion models, the paper presents a first extension to latent diffusion models (LDM). The core idea is developed upon the existing DPS method, which forms an approximation to p(y|xt) by using the denoising score estimate. The key challenges of the extension to LDM are (1) the non-bijective mapping between encoder and decoder, and (2) the inconsistency at the boundary of mask (for inpainting problems). To address these challenges, the authors propose an additional gluing term that penalizes the discontinuity at the mask boundary.\n\nThe main practical contribution of this paper is that if unlocks the potential of large pre-trained LDM (e.g. stable diffusion), and hence the proposed based based on more powerful pre-trained models enjoys superior performance compared to its counterpart with ambient / pixel space settings. \n\nIn addition, the paper also presents a theoretical analysis in a toy setting (linear two-step diffusion models) to motivate the proposed objective. ", "questions": "- Theorem 3.8: what is the value for gamma_i in Alg 2? \n\n- Theorem 3.8 and its surrounding text: can you elaborate more on why it is robust to different values of step size eta? In the equation between line 221 and line 222 on page 7, apparently if I choose different value of eta, the resulting z0 would be different (as long as the grad norm after eta is non-zero). Am I missing anything here? In addition, in this equation, should there be a multiplicative factor \\gamma before the last term, or is it set to 1 for the theorem and the statement to be correct? \n\n- Page 7. line 228 \"DPS algorithm suffers from the curse of ambient dimension\". Can you elaborate on the issue? Is it mainly the efficiency issue or the approximation accuracy issue?\n\n- Would be helpful to accomplish the theoretical analysis with toy data empirical verification. \n\n- (If possible), would be helpful to provide results where the base diffusion models for DPS and PSLD have similar performance. ", "limitations": "Yes. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- While the extension of DPS to LDMs seems natural, I appreciate the practical impact of the proposed methodology given the popularity and superior power of existing pre-trained LDMs. \n\n- If I understand it correctly, doing conditional inference in latent space would also be more efficient than doing inference in the ambient space (after controlling the model complexity though in practice I wonder if that comparison can be made) \n\n- The proposed method is also claimed to be robustness of the choice of step-size, at least in toy settings, though I am not sure how this would extend to general case. \n\n", "weaknesses": "- The theoretical analysis seems to follow reference 30 closely, takes up a large portion of the paper. However, I am not sure how much insights can be carried over to the general cases. In my opinion, the proposed method stays well-motivated without this analysis. Hence. I believe either discussing the extension to general cases, or trimming this part would make a better paper. \n\n- The evaluation part can be misleading given that pre-trained LDMs are \"trained on much more data compared to the one used by DPS\" (ref: appendix, caption above table 5). I think this information should be disclosed in the main paper instead of the appendix. In addition, it would be helpful to also disclose the unconditional generation performance of all backbone diffusion models, as well as the size of the models, the number of training data, etc. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer yVXi\nDear Reviewer yVXi,\n\nThank you for the review and for pointing out the fact that our study proposes the **first method** for inverse problems with latent diffusion models, and **unlocks the potential** of large **pre-trained LDMs** for sample recovery.\n\nBelow, we respond to your remaining comments and questions.\n\n(1) **Methodology stays well-motivated, trimming theoretical analysis would make a better paper.**\nWe will trim this section of the paper as per the reviewer's comment. Note that the main purpose of the theoretical analysis is to give **intuition** (as rightly pointed out by Reviewer vbWT) on why gluing term is critical to the empirical success of PSLD. The two-step process in a linear model setting **serves this purpose without unnecessary mathematical complications**. To elaborate, (a) vanilla extension of DPS fails due to many-to-one mapping of the VAE encoder, (b) GML-DPS fails due to infinitely many fixed-points of the linear system, and (c) PSLD works due to its inherent nature to find the stable fixed-point. The theoretical analysis gave us the intuition for the gluing objective that led to contraction of the distance from optimal solution and thereby strong empirical performance.\n\n(2) **\"Stable Diffusion is trained on much more data compared to the DM used by DPS\". Move this information from Table 5 in appendix to the main body of the paper.**\nAs suggested, we will move this statement to the main body of the paper. However, we would like to highlight that the focus of the paper is to **unlock** the potential of large pre-trained LDMs (e.g. Stable Diffusion), not to beat DPS.\n\n(3) **Helpful to discuss unconditional generation performance of the backbone diffusion models, dataset, size etc.**\nSince most of the large pre-trained LDMs (e.g. Stable Diffusion) are maintained by commercial service providers, they keep updating their pre-trained weights, datasets, and training iterations.  It is hard to compare these generative models with the pixel-space diffusion model used by DPS. Nevertheless, we report the metrics on FFHQ from the original papers and the github source code.\n\n\n| Model    | Dataset|Weights|Iterations |Image Size|\n|:---------|:-------|:------|:----------|:---------|\n|SD-v-1.5  |LAION-5B|4.00GB |840K       | 512x512  |    \n|PSLD-LDM  |FFHQ    |2.40GB |635K       | 256x256  |\n|DPS-DM    |FFHQ    |358MB  |1M         | 256x256  |\n\n\n\nThe FID score of an earlier version of LDM is 4.98 [1]. We could not find the unconditional generation performance of DPS in terms of FID anywhere as the authors did not evaluate it for generative modeling. The goal of their paper was to study **posterior sampling** using a pre-trained diffusion model. Also, sampling from diffusion models takes much more time than sampling from other generative models, such as GANs, which could be another reason for lack of evaluation in generative modeling unless that is the main contribution. \n\n\n\n(5) **Theorem 3.8: What is the value of $\\gamma_i$ in Alg 2?**\nWe set $\\gamma_i=1$ as it is immaterial in a linear model setting.\n\n(4) **Intuition behind Theorem 3.8 and its surrounding text.**\nThe intuition is that, after every step of denoising and measurement update, we would like to solve an optimization problem to make sure that the decoded sample resides on the manifold of natural images. This is achieved by the gluing update. In practice, one step of gluing update suffices instead of exactly solving the optimization problem at every step. However, in a perfectly recoverable linear model setting, the optimization problem can be exactly solved in one step. This makes the gluing term robust against the choice of step size $\\eta$.\n\nAs we discuss in line 184, the multiplicative factor $\\gamma$ is immaterial in a linear model setting. Hence, we set it to $\\gamma=1$. Empirically, we observe that $\\gamma=0.1$ and $\\eta=1$ are robust to several downstream tasks given in Appendix B (please also see newly added experiments in the attached PDF).\n\n\n(6) **Elaborate on \"DPS suffers from the curse of ambient dimension\".**\nDPS suffers from curse of ambient dimension because in this method, gradients are computed in the pixel space with dimension $d$. However, latent-based methods such as PSLD compute gradients in the latent dimension $k$, and hence the computation is more efficient. Furthermore, applying the chain rule on VAE and running diffusion in the latent space is less expensive than running diffusion in pixel space directly.\n\n(7) **(If possible) Compare PSLD and DPS with the same diffusion model.**\nSince our goal is to solve inverse problems using LDMs, we use large pre-trained LDMs (e.g., Stable Diffusion) as our generative prior. In practice, training these LDMs require a lot of computational investment and their performance is usually better than pixel-space diffusion models. Although we would be happy to compare their performance with the same diffusion model, we are not aware of any pixel-space diffusion model that is comparable to latent-space diffusion models.\n\n\n### Concluding Remark\n\nPlease let us know if the clarifications and additions suitably address your concerns. We are happy to address any remaining points during the discussion phase.\n\n### Reference\n[1] Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. \"High-resolution image synthesis with latent diffusion models.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695. 2022.\n"}, {"review_id": "MLGDZ62VYd", "reviewer": "Reviewer_S3JW", "summary": "This paper focuses on solving inverse problems using diffusion based probabilistic models without **retraining**. The authors build on \"Diffusion posterior sampling\" which basically builds a diffusion model for the posterior using only the score of the prior distribution. Indeed, the score of the posterior $\\nabla \\log p_t(x_t | y)$ is equal to $\\nabla \\log p(y|x_t) + \\nabla \\log p_t(x_t)$ where the second term is given by the prior diffusion model. The first term is however intractable but can be approximated by noting that $p(y | x_t) = \\int p(y|x_0) p(x_0 | x_t) dx_0$. This integral is then approximated by simply taking the mean a posteriori i.e. $p(y | x_t) \\approx p(y | E(x_0 | x_t))$ which  itself can be approximated using Tweedie's formula and the prior score. The original DPS paper only considers the diffusion in the pixel space and this paper extends the methodology to latent diffusion models. While the direct extension is straightforward, the authors claim that it does not work in practice and they provide a rather intuitive modified posterior diffusion model that ensures consistency at the borders of the mask. \n\n", "questions": "- What do the authors mean by the *curse of ambient dimension*, line 226-228? In DPS the gradient is indeed computed in the $d$-dimensional space but this is also the case of PSLD by the chain rule (and in fact the computational cost of PSLD is slightly larger).\n\n- In Figure 2 right panel, an inpainting example is considered. In this case p(y|x_0) is a dirac delta so I am not sure how the authors manage to take the gradient. Do the authors approximate the dirac delta with a Gaussian with small variance? Was the same courtesy applied to DPS?  \n\n- It is quite surprising that in Algorithm 2 (but also Algo1) nothing depends on the measurement variance. Why is it not taken into account? \n\n- While I could guess what it respresents, $x_0 *$ is not defined anywhere. ", "limitations": "see below.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tackles an important problem and provides a sound methodology for diffusion posterior sampling for latent space diffusion models. Furthermore, the numerical experiments suggest that this method outperforms other existing methods. ", "weaknesses": "The fact that the authors show recovery in a 2 step diffusion model in a very basic setting proves nothing. First, the results hold only when the optimal solution is normalized. In realistic settings and with multi step diffusion one does not know in advance how to rescale (if even rescaling is the right to do?) the drift so that the diffusion model matches the target distribution. Next, the authors prove that DPS \"provably\" recovers the posterior but this to me is also meaningless since one can easily show numerically in a noisy inpainting setting with Gaussian mixtures (the posterior is tractable in this case) that DPS **does not** in fact sample the posterior. It samples outside the support of the posterior even in the simplest settings. I believe that the authors should provide a numerical experiment where the target posterior is known (again, Gaussian mixtures) and show that their method does sample from the posterior. To me there is no reason why it should exactly sample from the posterior and not outside the support like DPS. Indeed, this issue is not related to the quality of the generative model but more to the fact that DPS (and truthfully many of the difussion posterior sampling methods) are not exact samplers since there is no correction like in MCMC algorithms. I believe that the authors should drop the \"provably\" in the title as it is misleading. \n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer S3JW\nDear Reviewer S3JW,\n\nThank you for the review and for pointing out the **importance of our work in leveraging the power of latent-based diffusion models** in solving inverse problems with state-of-the-art performance.  \n\n\nBelow, we provide answers to your remaining comments and questions.\n\n\n(1) **Recovery in two-step process is a basic setting.**\nThe main purpose of the theoretical analysis is to give **intuition** (as rightly pointed out by Reviewer vbWT) on why gluing term is critical to the empirical success of PSLD. The two-step process in a linear model setting **serves this purpose without unnecessary mathematical complications**. To elaborate, (a) vanilla extension of DPS fails due to many-to-one mapping of the VAE encoder, (b) GML-DPS fails due to infinitely many fixed-points of the linear system, and (c) PSLD works due to its inherent nature to find the stable fixed-point. The theoretical analysis gave us the intuition for the gluing objective that led to contraction of the distance from optimal solution and thereby strong empirical performance.  Besides, the main contribution of the paper is to solve inverse problems using latent diffusion models, unlocking the potential of large foundation models (e.g. Stable Diffusion) as rightly pointed out by Reviewer  vbWT, yVXi and xKok. We substantiate our contribution with large-scale experiments typically considered in practice.\n\n(2) **One does not know in advance how to scale.**\nWe would like to point out that the scaling factor is a hyper-parameter that is usually tuned in practice. Our theory provides intuitions on how to better tune this factor and rule out some unwanted experiments. Besides, we believe this is an issue of the generative model, not posterior sampling. As long as we have a pre-trained generative model that can sample from the data distribution, our method can leverage this model for posterior sampling.    \n\n\n(3) **DPS does not sample the posterior in noisy setting.** The setting we consider is noiseless and exactly recoverable, where we prove that DPS samples the posterior under valid assumptions. We will clarify this noiseless setting in the revised version.\n\n\n(4) **Many diffusion posterior sampling algorithms are not exact samplers.**\nMany diffusion posterior samplers are not exact samplers, but their empirical performance is significantly better than any of the MCMC algorithms with strong theoretical guarantees. To the best of our knowledge, these MCMC algorithms with strong theoretical results are only shown to work in toy experimental setting. **Provably in title.** To remove ambiguity, we will change the title to *\"Solving **Linear** Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models\"*.\n\n(5) **What do the authors mean by the curse of ambient dimension?**\nAs we discuss in Section 3, PSLD computes the gradients in the latent space with dimension $k$ (wrt $\\mathbf{z}_i$), whereas DPS computes the gradients in the pixel space with dimension $d$ (wrt $\\mathbf{x}_i$). In practice, the computational complexity of Stable Diffusion model ($\\sim 4.00GB$) is higher (roughly 6 times) than the computational complexity of the encoder-decoder model ($\\sim 700 MB$). Therefore, applying the chain rule in the encoder-decoder and running diffusion in the latent space is less expensive than applying diffusion models in the pixel space directly.  \n\n(6) **How to take gradients of $P(y|x_0)$ in block inpainting?**\nAs correctly pointed out by the reviewer, we use Gaussian approximation of the dirac delta. The same idea is applied in DPS approximation for handling block inpainting.\n\n(7) **It is quite surprising that in Algorithm 2 (but also Algo1) nothing depends on the measurement variance. Why is it not taken into account?**\nWe believe there is a misunderstanding. Recall that this is a maximum likelihood estimation problem. Since $\\mathbf{y} = AD(\\mathbf{z})+\\sigma_y\\mathbf{n} $, where $\\mathbf{n}\\sim \\mathcal{N}(\\mathbf{0},\\mathbf{I})$, we have $\\mathbf{y} \\sim P(y|z) \\propto \\exp{\\left(-\\frac{1}{2\\sigma_y^2} || \\mathbf{y} - AD(\\mathbf{z})||_2^2\\right)}$. Taking the gradient of the log-likelihood, we have $\\eta\\nabla_z || \\mathbf{y} - AD(\\mathbf{z})||_2^2$, where $\\eta$ absorbs the scaling factor $\\frac{1}{2\\sigma_y^2}$.\n\n\n(8) **Definition of $x_0^\\*$.**\nThank you for pointing out the typo. We will correct it in the revision. We denote by $x_0^*$ the true underlying sample as you could have rightly guessed.\n\n\n\n### Concluding Remark\n\nPlease let us know if the clarifications and additions suitably address your concerns. We are happy to address any remaining points during the discussion phase.\n"}, {"review_id": "RQEjtqSNNX", "reviewer": "Reviewer_xKok", "summary": "This paper proposes a method to deal with linear inverse problems using pre-trained latent diffusion models (LDM) as a prior. The main idea is to extend the original diffusion posterior sampling (DPS) to the case of LDM by approximating the gradient term of the intractable likelihood.  Two approximations (GLM-DPS), and (PSLD) are proposed for such approximation, and it turns out that PSLD achieves better performances. This paper also theoretically analyzes the DPS and PSLD in a toy linear setting. Experiment results show superior performances compared with the original DPS in a variety of tasks, such as random/block inpainting, denoising, super-resolution.", "questions": "Three additional questions:\n\n1. How do you add the additive Gaussian noise with standard deviation \\sigma_y? \n\n\n2. Can I understand that the improved performance of PSLD over DPS comes solely from the improvement of diffusion models in the latent space over that in the pixel space? In other words, if DPS is implemented with more powerful diffusion models, is it possible to outperform PSLD? In this sense, how to ensure a fair comparison between PSLD and DPS? \n\n3. Are the results of PSLD sensitive to the choice of hyper-parameters? Compared to DPS which has only one scaling hyperparameter, there are two scaling hyperparameters, namely, \\eta and \\gamma. It is suggested to evaluate the sensitivity of these hyperparameters.", "limitations": "Please refer to the weakness and questions parts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is the first work, to my best knowledge, that LDM is used to address the linear inverse problems. \n\n2. Compared to the original DPS, the proposed PSLD achieves apparently better performances in most cases. \n\n3. Some theoretical analysis is also provided, although it is based on some toy linear model.", "weaknesses": "1.  There is a lack of analysis of the complexity, or running time, of the proposed PSLD method. Adding details of the running time of PSLD and other comparison methods is suggested. \n\n2. Part 3 contains some theoretical analysis using a simple toy linear model. For example, Theorem 3.4 states that DPS exactly recovers the groundtruth sample. It is known that DPS uses a Laplace approximation to approximate the gradient term of the likelihood. As a result, how can it exactly recover the posterior samples with the crude approximation? This is also the case with Theorem 3.7 and Theorem 3.8.  To what extent can the results of Theorem 3.4, 3.7, and 3.8 be generalized to the real diffusion models?\n\n3. In the experiments part, while PSLD shows superior performances over the original DPS, there is a lack of comparison with other latest diffusion-model-based algorithms that were available long before NeurIPS submission, e.g., \n\n[A] Wang, Yinhuai, Jiwen Yu, and Jian Zhang. \"Zero-shot image restoration using denoising diffusion null-space model.\" arXiv preprint arXiv:2212.00490 (2022).\n\n[B] Meng, Xiangming, and Yoshiyuki Kabashima. \"Diffusion model based posterior sampling for noisy linear inverse problems.\" arXiv preprint arXiv:2211.12343 (2022).\n\n[C] Song, Jiaming, Arash Vahdat, Morteza Mardani, and Jan Kautz. \"Pseudoinverse-guided diffusion models for inverse problems.\" In International Conference on Learning Representations. 2022.\n\n4. Most previous image restoration methods using diffusion models also consider colorization, can the authors also add some results in the case of colorization? Besides, only Table 2 and Table 5 show quantitative comparisons with other methods but only for inpainting. It is suggested to add quantitative comparisons with other methods in other tasks such as super-resolution, denoising, colorization, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer xKok\n\nDear Reviewer xKok,\n\nThank you for the review and for pointing out that **our study achieves state-of-the-art performance** in addressing inverse problems with latent diffusion models, and **unleashes the capacity of large-scale pre-trained LDMs** for sample recovery.\n\n\nBelow, we provide answers to your remaining comments and questions.\n\n(1) **NFEs of different algorithms.**\nSince the runtime of diffusion based posterior samplers depends heavily on the number of Neural Function Evaluations (NFEs), we add comparison of different algorithms in terms NFEs.\n\n|    |PSLD (ours)|DPS |DDRM|RED [2]|$\\Pi$GDM [1]|Palette [3]|Regression|SNIPS [4]|\n|:---|:----------|:--------|:--|:---|:--|:------|:---------|:----|\n|NFEs|100 to 1000|1000|20 |500|20 to 100|1000   |1         |1000 |\n\nThe computational complexity of our method PSLD is similar to that of DPS. For a relatively smaller number of diffusion steps (say 100), PSLD significantly outperforms DPS. The DPS generated samples do not look like realistic images for fewer diffusion steps [1]. \n\n\n**Runtime of different algorithms for Super-resolution task.**\n\n|Method        | Runtime (s)|\n|:-------------|:-----------|\n|DMPS [6]|67.02  |\n|DPS     |180.00 |\n|DDNM+[5]|18.5   |\n|DDRM    |2.15   |\n|MCG     |193.71 |\n|PSLD-LDM|187.00 |\n|PSLD-LDM (LAION-400M)^|190.00 |\n|PSLD-SD (LAION-5B)* |194.25  |\n\n\n*PSLD-SD (trained on LAION-5B) takes 776 s to generate 512x512 images. To compare with other methods, we divide its runtime by 4. ^PSLD-LDM (LAION-400M) uses a diffusion model trained on LAION-400M dataset. All the other methods use diffusion models trained on FFHQ and produce 256x256 images.\n\n\n(2) **Exact recovery is not possible by DPS.**\nAlthough DPS uses Laplace approximation for the gradient of the likelihood, this approximation is not necessary in a recoverable linear model setting as it can be exactly solved. The same argument also holds for Theorem 3.7 and Theorem 3.8.\n\n(3) **PSLD is better than DPS, but no comparison with other latest diffusion-model-based algorithms available before NeurIPS deadline.**\nThanks for bringing these recent studies to our attention. Unfortunately, none of the suggested papers are **latent diffusion-model-based algorithms**. Nevertheless, we have compared with these methods, please see our answer to (7) and the attached PDF.\n\n\n(4) **How do you add Gaussian noise with standard deviation σ.**\nWe sample $\\hat{n} \\sim \\mathcal{N}(0,I)$ and then scale this sample by $\\sigma$, i.e., $n = \\sigma * \\hat{n}~$ to generate the IID Gaussian noise $n \\sim \\mathcal{N}(0,\\sigma I).$\n\n(5) **If DPS is implemented with a powerful diffusion model, can it outperform PSLD?**\nOur goal is to develop a framework to leverage the power of pre-trained latent diffusion models, such as Stable Diffusion. Our focus is not to maximize the margin that we beat the previous state-of-the-art DPS, but to unlock the potential of large pre-trained LDMs. Needless to say, the more powerful the prior the better it is for posterior sampling (e.g., classical priors to GANs and GANs to Diffusion). But previous posterior sampling algorithms (including DPS, DMPS and DDRM) only apply to pixel-space diffusion models, which leaves a gap to leverage the power of pre-trained LDMs. In this paper, we bridge this gap in literature as pointed out by Reviewer vbWT.\n\n(6) **Robustness of hyper-parameters.**\nThe values of {$\\gamma = 0.1$ and $\\eta = 1$} are fixed for most of the practical tasks as mentioned in the appendix, justifying their robustness.\n\n(7) **Additional quantitative Results.**\nAs suggested by the reviewer, we have added the following new results:\n\n\n\n| Method | SR(4X) |            | Gaussian Deblur |             | \n|:----------|:--------|:--------|:--------------------|:--------|\n|               |PSNR  |SSIM|PSNR|SSIM\n|PSLD (Ours)|30.73|0.867|30.10|0.843\n|GML-DPS (Ours)|29.77|0.860|29.21|0.820\n|DMPS[6]|27.63|-|25.41|-\n|DPS|25.67|0.852|24.25|0.811\n|DDRM|25.36|0.835|23.36|0.0.767\n|MCG|20.05|0.559|6.72|0.051\n|PnP-ADMM|26.55|0.865|24.93|0.812\n|Score-SDE|17.62|0.617|7.12|0.109\n|ADMM-TV |23.86|0.803|22.37|0.801\n\n\n\n\n| Method | SR(4X) |            | Gaussian Deblur |             | \n|:----------|:--------|:--------|:--------------------|:--------|\n|               |FID       |LPIPS |FID                        |LPIPS\n|PSLD (Ours)|34.28|0.201|41.53|0.221\n|DPS|39.35|0.214|44.05|0.257\n|DDRM|62.15|0.294|74.92|0.332\n|MCG|87.64|0.520|101.2|0.340\n|PnP-ADMM|66.52|0.353|90.42|0.441\n|Score-SDE|96.72|0.563|109.0|0.403\n|ADMM-TV |110.6|0.420|186.7|0.507\n\n\n\n\n### Concluding Remark\n\nPlease let us know if the clarifications and additions suitably address your concerns. We are happy to address any remaining points during the discussion phase.\n\n\n\n### Reference\n[1] Song, Jiaming, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guided diffusion models for inverse problems. In: International Conference on Learning Representations. 2023. url: https://openreview.net/forum?id=9_gsMA8MRKQ.\n\n[2] Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by\ndenoising (RED). arXiv preprint arXiv:1611.02862, November 2016.\n\n[3] Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D. and Norouzi, M., 2022, July. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings (pp. 1-10).\n\n[4] Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving noisy inverse problems\nstochastically. arXiv preprint arXiv:2105.14951, May 2021.\n\n[5] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model. In: The Eleventh International Conference on Learning Representations. 2023. url: https://openreview.net/forum?id=mRieQgMtNTQ.\n\n[6] Meng, X. and Kabashima, Y., 2022. Diffusion model based posterior sampling for noisy linear inverse problems. arXiv preprint arXiv:2211.12343.\n\n"}]

Author rebuttal:
### Response to all reviewers
Dear Reviewers,

We thank you for carefully reading our paper and providing us with valuable feedback. Below, we summarize the reviews and newly added experiments to substantiate our contributions.

(1) We are encouraged by the **unanimous comment** by all reviewers that our study proposes the **first method for solving inverse problems with LDMs**, which **unlocks** the potential of large pre-trained LDMs, such as Stable Diffusion.

(2) We thank Reviewer vbWT for highlighting the fact that the **theoretical result is valuable and noteworthy**, considering the field's scarcity of rigorous results, even in toy examples. We also thank all the reviewers for pointing out the fact that our **experimental results are promising**. 

(3) Regarding constructive feedback, we have **favorably addressed all the questions** raised by the reviewers. In particular, we thank Reviewer vbWT, yVXi and xKok for suggesting relevant experiments with quantifiable metrics, which helped **strengthen our contributions**. We have added the following experimental results in the attached PDF:
    
    (a) Quantitative results on Super-resolution (4X): Table 1 
    (b) Quantitative results on Gaussian Deblur: Table 1 
    (c) Qualitative results on colorization: Figure 1
    (d) Quantitative comparison of runtimes of different algorithms: Table 2
    (e) Quantitative comparison of NFEs of different algorithms: Table 2 
    (f) Quantitative comparison of unconditional generation performance: Table 2 
    (g) Overall pipeline of our proposed framework for arbitrary mask: Figure 2

(4) We thank Reviewer xKok for bringing related works to our attention. We have compared with these methods and cited accordingly in the revised version. 



### Concluding Remark

Please let us know if the clarifications and additions suitably address your concerns. We are happy to address any remaining points during the discussion phase.



### Reference

[1] Song, Jiaming, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guided diffusion models for inverse problems. In: International Conference on Learning Representations. 2023. url: https://openreview.net/forum?id=9_gsMA8MRKQ.

[2] Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by
denoising (RED). arXiv preprint arXiv:1611.02862, November 2016.

[3] Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D. and Norouzi, M., 2022, July. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings (pp. 1-10).

[4] Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving noisy inverse problems
stochastically. arXiv preprint arXiv:2105.14951, May 2021.

[5] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model. In: The Eleventh International Conference on Learning Representations. 2023. url: https://openreview.net/forum?id=mRieQgMtNTQ.

[6] Meng, X. and Kabashima, Y., 2022. Diffusion model based posterior sampling for noisy linear inverse problems. arXiv preprint arXiv:2211.12343.



Meta-review: This paper proposes a new method to deal with linear inverse problems using pre-trained latent diffusion models (LDM). It lowers computational demand for large-scale problems, and unlocks the potential of large pre-trained LDM for inverse problem solving.
Most of the reviewers acknowledge the novelty of the method, along with the theoretical justifications.

Please incorporate all the reviewers' feedback into the final version, especially the new experimental comparison with other methods.
Decision: Accept (poster)