SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \overleftarrow {z _ {0} ^ {\prime}} = \boldsymbol {\theta} ^ {*} \overleftarrow {z _ {1}} - \boldsymbol {\eta} \nabla_ {\overleftarrow {z _ {1}}} \left\| \mathcal {A D} (\overleftarrow {z _ {0}} (\overleftarrow {z _ {1}})) - \boldsymbol {y} \right\| _ {2} ^ {2} \\ = \boldsymbol {I} _ {k} \overleftarrow {\boldsymbol {z}} _ {1} - \eta \nabla_ {\hat {\boldsymbol {z}} _ {1}} \| \mathcal {A D} (\boldsymbol {I} _ {k} \overleftarrow {\boldsymbol {z}} _ {1}) - \boldsymbol {y} \| _ {2} ^ {2} \\ = \overleftarrow {z _ {1}} - \eta \nabla_ {\overleftarrow {z} _ {1}} \| \mathcal {A S} \overleftarrow {z _ {1}}) - \boldsymbol {y} \| _ {2} ^ {2} \\ = \overleftarrow {z _ {1}} - \eta \nabla_ {\overleftarrow {z} _ {1}} \left\| \mathcal {A S} \overleftarrow {z _ {1}}\right) - \boldsymbol {y} \| _ {2} ^ {2} \\ \stackrel {(i)} {=} \overleftarrow {z _ {1}} - \eta \nabla_ {\overleftarrow {z _ {1}}} \| \mathcal {A} \mathcal {S} \overleftarrow {z _ {1}} - \boldsymbol {y} \| _ {2} ^ {2} \\ = \overleftarrow {z _ {1}} - 2 \eta \mathcal {S} ^ {T} \mathcal {A} ^ {T} (\mathcal {A} \overleftarrow {z _ {1}} - \boldsymbol {y}) \\ = \overleftarrow {z} _ {1} - 2 \eta \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overleftarrow {z} _ {1} + 2 \eta \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {y} \\ = \overleftarrow {z _ {1}} - 2 \eta \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overleftarrow {z _ {1}} + 2 \eta \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \overrightarrow {x _ {0}} \\ = \overleftarrow {z _ {1}} - 2 \eta \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A S} \overleftarrow {z _ {1}} + 2 \eta \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A S} \overrightarrow {z _ {0}}, \\ \end{array}
$$
where (i) is due to Assumption 3.1. By Assumption 3.2,  $(\mathcal{A}\mathcal{S})^T (\mathcal{A}\mathcal{S})$  is a positive definite matrix and can be written as  $U\Sigma U^{T}$ :
$$
\begin{array}{l} \overleftarrow {z _ {0} ^ {\prime}} = \overleftarrow {z _ {1}} - 2 \boldsymbol {\eta} \mathbf {U} \Sigma \mathbf {U} ^ {T} \overleftarrow {z _ {1}} + 2 \boldsymbol {\eta} \mathbf {U} \Sigma \mathbf {U} ^ {T} \overrightarrow {z _ {0} ^ {\prime}} \\ = \overleftarrow {z _ {1}} - 2 \boldsymbol {U} \boldsymbol {D} (\boldsymbol {\eta} _ {i}) \boldsymbol {U} ^ {T} \boldsymbol {U} \boldsymbol {\Sigma} \boldsymbol {U} ^ {T} \overleftarrow {z _ {1}} + 2 \boldsymbol {U} \boldsymbol {D} (\boldsymbol {\eta} _ {i}) \boldsymbol {U} ^ {T} \boldsymbol {U} \boldsymbol {\Sigma} \boldsymbol {U} ^ {T} \overrightarrow {z _ {0}} \\ = \overleftarrow {z _ {1}} - 2 \boldsymbol {U} \boldsymbol {D} (\boldsymbol {\eta} _ {i}) \Sigma \boldsymbol {U} ^ {T} \overleftarrow {z _ {1}} + 2 \boldsymbol {U} \boldsymbol {D} (\boldsymbol {\eta} _ {i}) \Sigma \boldsymbol {U} ^ {T} \overrightarrow {z _ {0}} \\ = \overleftarrow {z _ {1}} - 2 U D (\boldsymbol {\eta} _ {i}) D (\boldsymbol {\sigma}) U ^ {T} \overleftarrow {z _ {1}} + 2 U D (\boldsymbol {\eta} _ {i}) D (\boldsymbol {\sigma}) U ^ {T} \overrightarrow {z _ {0}} \\ = \overleftarrow {\mathbf {z}} _ {1} - 2 \mathbf {U} \mathbf {D} (\boldsymbol {\eta} _ {i} \odot \boldsymbol {\sigma}) \mathbf {U} ^ {T} \overleftarrow {\mathbf {z}} _ {1} + 2 \mathbf {U} \mathbf {D} (\boldsymbol {\eta} _ {i} \odot \boldsymbol {\sigma}) \mathbf {U} ^ {T} \overrightarrow {\mathbf {z}} _ {0}. \\ \end{array}
$$
Since  $\eta_j^i = 1 / 2\sigma_j$ , the above expression further simplifies to
$$
\stackrel {\leftarrow} {\boldsymbol {z}} _ {0} ^ {\prime} = \stackrel {\leftarrow} {\boldsymbol {\bar {z}}} _ {1} - \boldsymbol {U} \boldsymbol {U} ^ {T} \stackrel {\leftarrow} {\boldsymbol {\bar {z}}} _ {1} + \boldsymbol {U} \boldsymbol {U} ^ {T} \vec {\boldsymbol {z}} _ {0} = \vec {\boldsymbol {z}} _ {0} ^ {\prime}.
$$
Next, we show that  $\overleftarrow{z_0'}$  is already a minimizer of (9). This is a direct consequence of the encoder-decoder architecture of the VAE:  $\mathcal{E}(\mathcal{D}(\overleftarrow{z_0'})) = \mathcal{S}^T\mathcal{S}\overleftarrow{z_0'} = \overleftarrow{z_0'}$ . Hence,  $\left\| \overleftarrow{z_0'} - \mathcal{E}(\mathcal{D}(\overleftarrow{z_0'}))\right\|^2 = 0$ , and
consequently  $\overleftarrow{z_0} = \overleftarrow{z_0'} - \gamma \nabla_{\overleftarrow{z_0'}} \left\| \overleftarrow{z_0'} - \mathcal{E}(\mathcal{D}(\overleftarrow{z_0'})) \right\|^2 = \overrightarrow{z_0}$ . Thus, the reconstructed sample becomes  $\overleftarrow{\pmb{x}_0} = \mathcal{D}(\overleftarrow{\pmb{z}_0}) = \mathcal{S}\overrightarrow{\pmb{z}_0} = \overrightarrow{\pmb{x}_0}$ .
Furthermore, as  $\left\| \overleftarrow{z_0'} - \mathcal{E}(\mathcal{D}(\overleftarrow{z_0'})) \right\|^2 = 0$  for all  $\overleftarrow{z_0'}$ , it is evident that the goodness objective cannot rectify the error incurred in the measurement update (8). For this reason, GML-DPS algorithm (6) requires the exact step size to sample from the posterior.
Beyond the linear setting, we also refer to Table 5 for experiments supporting this result.
B.5 Proof of Theorem 3.6
Different from GML-DPS, PSLD Algorithm 2 replaces the goodness objective (6) with the gluing objective (7), which can be formalized as:
$$
\overleftarrow {z _ {0} ^ {\prime}} = \boldsymbol {\theta} ^ {*} \overleftarrow {z _ {1}} - \eta \nabla_ {\overleftarrow {z} _ {1}} \| \mathcal {A D} (\overleftarrow {z} _ {0} (\overleftarrow {z} _ {1})) - \boldsymbol {y} \| _ {2} ^ {2}; \tag {10}
$$
$$
\left. \overleftarrow {\boldsymbol {z}} _ {0} = \arg \min  _ {\overleftarrow {\boldsymbol {z}} _ {0} ^ {\prime}} \left| \left| \overleftarrow {\boldsymbol {z}} _ {0} ^ {\prime} - \mathcal {E} \left(\mathcal {A} ^ {T} \mathcal {A} \overrightarrow {\boldsymbol {z}} _ {0} + \left(\boldsymbol {I} _ {d} - \mathcal {A} ^ {T} \mathcal {A}\right) \mathcal {D} \left(\overleftarrow {\boldsymbol {z}} _ {0} ^ {\prime}\right)\right) \right| \right| _ {2} ^ {2}. \right. \tag {11}
$$
We again remind that solving the minimization problem (11) is hard in general, and can be approximated by gradient descent as typically followed in practice [11]. However, in a linear model setting, (11) has a closed-form solution which we derive to prove exact recovery.
Theorem B.5 (Posterior Sampling using Diffusion in Latent Space). Let Assumptions 3.1 and 3.2 hold. Let  $\sigma_{j},\forall j = 1,\ldots ,r$  denote the singular values of  $(\mathcal{AS})^T (\mathcal{AS})$  and let
$$
\boldsymbol {\theta} ^ {*} = \arg \min  _ {\boldsymbol {\theta}} \mathbb {E} _ {\vec {z} _ {0}, \vec {\epsilon}} \left[ \left\| \tilde {\mu} _ {1} \left(\overrightarrow {z} _ {1} ^ {\prime} (\vec {z} _ {0} ^ {\prime}, \vec {\epsilon}), \vec {z} _ {0} ^ {\prime}\right) - \mu_ {\theta} \left(\overrightarrow {z} _ {1} ^ {\prime} (\vec {z} _ {0} ^ {\prime}, \vec {\epsilon})\right) \right\| ^ {2} \right].
$$
Suppose  $\overrightarrow{\pmb{x}_0} \sim p(\overrightarrow{\pmb{x}_0})$ . Given measurements  $\pmb{y} = \mathcal{A}\overrightarrow{\pmb{x}_0}$ , any fixed variance  $\beta \in (0,1)$ , and any positive step sizes  $\eta_i^j$ ,  $j = 1,2,\ldots,r$ , the PSLD Algorithm 2 samples from the true posterior  $p(\overrightarrow{\pmb{x}_0} | y)$  and exactly recovers the groundtruth sample, i.e.,  $\overleftarrow{\pmb{x}_0} = \overrightarrow{\pmb{x}_0}$ .
Proof. Following the proof in Appendix B.4, we have
$$
\begin{array}{l} \overleftarrow {z _ {0} ^ {\prime}} = \boldsymbol {\theta} ^ {*} \overleftarrow {z _ {1}} - \eta \nabla_ {\overleftarrow {z _ {1}}} \left\| \mathcal {A D} (\overleftarrow {z _ {0}} (\overleftarrow {z _ {1}})) - \boldsymbol {y} \right\| _ {2} ^ {2} \\ = I _ {k} \overleftarrow {\mathbf {z}} _ {1} - \eta \nabla_ {\underline {{\mathbf {z}}} _ {1}} \| \mathcal {A D} (\overleftarrow {\mathbf {z}} _ {1}) - \mathbf {y} \| _ {2} ^ {2} \\ = \overleftarrow {z _ {1}} - \eta \nabla_ {\overleftarrow {z} _ {1}} \| \mathcal {A S} \overleftarrow {z _ {1}} - \boldsymbol {y} \| _ {2} ^ {2} \\ = \overleftarrow {\mathbf {z}} _ {1} - 2 \eta \mathcal {S} ^ {T} \mathcal {A} ^ {T} (\mathcal {A S} \overleftarrow {\mathbf {z}} _ {1} - \boldsymbol {y}) \\ = \overleftarrow {z _ {1}} - 2 \eta \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overleftarrow {z _ {1}} + 2 \eta \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overrightarrow {z _ {0}} \\ = \overleftarrow {z _ {1}} - 2 \eta \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A S} \overleftarrow {z _ {1}} + 2 \eta \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A S} \overrightarrow {z _ {0}}. \\ \end{array}
$$
We use the above expression to derive a closed-form solution to the minimization problem (11):
$$
\begin{array}{l} \mathbf {0} = \nabla_ {\overleftarrow {\boldsymbol {z} _ {0} ^ {\prime}}} ^ {\leftarrow} \left\| \overleftarrow {\boldsymbol {z} _ {0} ^ {\prime}} - \mathcal {S} ^ {T} (\mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overrightarrow {\boldsymbol {z} _ {0}} + (\boldsymbol {I} _ {d} - \mathcal {A} ^ {T} \mathcal {A}) \mathcal {S} \overleftarrow {\boldsymbol {z} _ {0} ^ {\prime}}) \right\| _ {2} ^ {2} \\ = \nabla_ {\boldsymbol {z} _ {0} ^ {\prime}} ^ {\leftarrow} \left\| \overleftarrow {\boldsymbol {z} _ {0} ^ {\prime}} - \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overrightarrow {\boldsymbol {z} _ {0}} - \mathcal {S} ^ {T} (\boldsymbol {I} _ {d} - \mathcal {A} ^ {T} \mathcal {A}) \mathcal {S} \overleftarrow {\boldsymbol {z} _ {0} ^ {\prime}}) \right\| _ {2} ^ {2} \\ = \nabla_ {\overrightarrow {z _ {0} ^ {\prime}}} ^ {\leftarrow} \left\| \overleftarrow {z _ {0} ^ {\prime}} - \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overrightarrow {z _ {0}} - \mathcal {S} ^ {T} \mathcal {S} \overleftarrow {z _ {0} ^ {\prime}} - \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overleftarrow {z _ {0} ^ {\prime}}\right) \| _ {2} ^ {2} \\ = \nabla_ {\overrightarrow {z _ {0} ^ {\prime}}} ^ {\leftarrow} \left\| \overleftarrow {z _ {0} ^ {\prime}} - \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overrightarrow {z _ {0}} - \mathcal {S} ^ {T} \mathcal {S} \overleftarrow {z _ {0} ^ {\prime}} + \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overleftarrow {z _ {0} ^ {\prime}}\right) \| _ {2} ^ {2} \\ = 2 \left(\boldsymbol {I} _ {k} - \mathcal {S} ^ {T} \mathcal {S} + \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S}\right) \left(\overleftarrow {\boldsymbol {z} _ {0} ^ {\prime}} - \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overrightarrow {\boldsymbol {z} _ {0}} - \mathcal {S} ^ {T} \mathcal {S} \overleftarrow {\boldsymbol {z} _ {0} ^ {\prime}} + \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overleftarrow {\boldsymbol {z} _ {0} ^ {\prime}}\right) \\ = 2 \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \left(\mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overleftarrow {\boldsymbol {z} _ {0} ^ {\prime}} - \mathcal {S} ^ {T} \mathcal {A} ^ {T} \mathcal {A} \mathcal {S} \overrightarrow {\boldsymbol {z} _ {0}}\right), \\ \end{array}
$$
where the last step is due to Assumption 3.1. Thus, we have
$$
\overleftarrow {\pmb {z} _ {0}} = \arg \min _ {\overleftarrow {\pmb {z} _ {0} ^ {\prime}}} \left| \left| \overleftarrow {\pmb {z} _ {0} ^ {\prime}} - \mathcal {E} (\mathcal {A} ^ {T} \mathcal {A} \overrightarrow {\pmb {z} _ {0}} + (\pmb {I} _ {d} - \mathcal {A} ^ {T} \mathcal {A}) \mathcal {D} (\overleftarrow {\pmb {z} _ {0} ^ {\prime}})) \right| \right| _ {2} ^ {2} = \overrightarrow {\pmb {z} _ {0}},
$$
which produces  $\overrightarrow{\pmb{x}_0} = \mathcal{D}(\overrightarrow{\pmb{z}_0}) = \mathcal{D}(\overrightarrow{\pmb{z}_0}) = \mathcal{S}\overrightarrow{\pmb{z}_0} = \overrightarrow{\pmb{x}_0}$
It is worth highlighting that PSLD exactly recovers the groundtruth sample irrespective of the choice of the step size  $\eta$ , whereas GML-DPS requires the step size to be exactly  $\pmb{\eta} = (1/2)\pmb{U}\pmb{D}(\pmb{\eta}_i)\pmb{U}^T$ .
For inpainting tasks, we note that the PSLD sampler generates missing parts (by design of our gluing objective) that are consistent with the known portions of the image, i.e.,  $\overleftarrow{\pmb{x}}_0 = \mathcal{A}^T\mathcal{A}\overrightarrow{\pmb{x}_0} + (\pmb{I}_d - \mathcal{A}^T\mathcal{A})\mathcal{D}(\overline{\pmb{z}_0})$ . This is different from the DPS sampler, which generates the whole image which may not match the observations exactly. In other words, in the last step of our algorithm, the observations are glued onto the corresponding parts of the generated image, leaving the unmasked portions untouched [54]. This sometimes creates edge effects which are then removed by post-processing the glued image through the encoder and decoder of the SD model, i.e. running one last step of our algorithm. Figure 2 illustrates that gluing the observations in commercial services still leads to visually inconsistent results (e.g. head in top row) unlike our method.
For all other tasks, such as motion deblur, Gaussian deblur, and super-resolution, this last step is not needed, as there is no box inpainting, i.e.,  $\overleftarrow{x_0} = \mathcal{D}(\overleftarrow{z_0})$ . Furthermore, we use the same measurement operator  $\mathcal{A}$  and its transpose  $\mathcal{A}^T$  as provided by the DPS code repository. However, since Stable Diffusion v1.5 generates images of size  $512 \times 512$  resolution and DPS operates at  $256 \times 256$ , we adjust the size of the kernels used in PSLD to ensure that both the methods use the same amount of information while sampling from the posterior. During evaluation, we downsample PSLD generated images from  $512 \times 512$  to  $256 \times 256$  to compare with DPS at the same resolution.
PSLD (Stable Diffusion-V1.5): We run Algorithm 2 with Stable Diffusion version 1.5 as the foundation model<sup>9</sup>. We use a fixed  $\eta = 1$  and  $\gamma = 0.1$ . Since we study posterior sampling of images without conditioning on text inputs, we pass an empty string to the Stable Diffusion foundation model, which accepts texts as an input argument. For better performance, we recommend using the latest pretrained weights.
PSLD (LDM-VQ-4): This is the same sampling algorithm as before but with a different latent diffusion model, LDM-VQ-4 $^{10}$ , which contains pretrained weights for FFHQ 256 $^{11}$  and large-scale text-to-image generative model $^{12}$ . We keep the hyperparameters same ( $\eta = 1$  and  $\gamma = 0.1$ ). For each task, we provide hyper-parameter details in our codebase $^{13}$ . Although we have tested our framework with these two latent-diffusion-models, one may experiment with other latent-diffusion-models available in the same repository.
DPS: We use the original source code provided by the authors<sup>14</sup>.
OD images are sourced online:
1. Figure 1: the original images are generated by Stable Diffusion v-2.1<sup>15</sup>.
2. Figure 2 first row: Walking example from the web.
3. Figure 2 second row, Obama-Biden image from the web.
4. Figure 2 third row, Fisherman from ImageNet 256 [17].
5. Figure 4 first row: Racoon image from the web.
6. Figure 4 second row: Fisherman from ImageNet 256 [17].
7. Figure 15: Celebrity face from the web.
$^{8}$ https://github.com/DPS2022/diffusion-posterior-sampling/blob/main/guided_diffusion/measurements.py  
 $^{9}$ https://huggingface.co/runwayml/stable-diffusion-v1-5  
 $^{10}$ https://github.com/CompVis-latent-diffusion  
 $^{11}$ https://ommer-lab.com/files/latent-diffusion/ffhq.zip  
 $^{12}$ https://ommer-lab.com/files/latent-diffusion/nitro/txt2img-f8-large/model.ckpt  
 $^{13}$ https://github.com/LituRout/PSLD  
 $^{14}$ https://github.com/DPS2022/diffusion-posterior-sampling  
 $^{15}$ https://huggingface.co/spaces/stabilityai/stable-diffusion
PSLD Image Inpainting
Given an image (square size preferred) and a user defined mask, click on Inpaint to generate missing parts.
Image inpainting by Posterior Sampling with Latent Diffusion (PSLD)
Figure 6: Results from the web application of our PSLD algorithm,  $512 \times 512$ . The original image (1) is generated by Stable Diffusion v-2.1 with the prompt, "A dinner date between a robot couple during sunset".
C.2 Additional Experimental Evaluation
Here, we provide additional results to support our theoretical claims on various inverse problems.
Figures 6, 7, 8, and 9 show the inpainting results of user defined masks obtained from our PSLD inpainting web demo. Note that the foundation model used in this demo is a generic model. For better performance on specific images, we recommend finetuning the foundation model on this class and then running posterior sampling using our web demo: https://huggingface.co/spaces/PSLD/PSLD.
Figure 10 and 11 illustrate super-resolution  $(4\times)$  of in-distribution samples from the validation set of FFHQ 256. Observe that the samples generated by DPS are far from the groundtruth sample. On the other hand, the samples generated by PSLD closely capture the perceptual quality of the groundtruth sample. In other words, one may identify (b) and (c) as images of two different individuals, whereas (b) and (d) of the same individual. We attribute this photorealism of our method to the power of Stable Diffusion foundation model and the ability to use the knowledge of the VAE encoder-decoder in the gluing objective.
In addition, we test on out-of-distribution samples from ImageNet [17] validation set. Figure 12 and Figure 13 show the results in motion deblur and Gaussian deblur, respectively. By leveraging the foundation model Stable Diffusion v1.5, our PSLD method clearly outperforms DPS [11] in the general domain. Further, Figures 14, 15, and 16 show reconstruction of general domain samples for random inpainting, super-resolution, and destriping tasks, respectively. In all these tasks, the samples generated by PSLD are closer to the groundtruth sample than the ones generated by DPS. Figure 17 shows the results on image colorization. Table 5 and Table 6 show the quantitative results. Table 7 draws a comparison between the latent-DPS and PSLD algorithms, and shows that the PSLD objective enhances the reconstruction performance.
In Table 8, we compare the runtime and NFEs of PSLD with prior works. PSLD-SD (trained on LAION-5B) takes  $776\mathrm{s}$  to generate  $512\times 512$  images. To compare with other methods that generate  $256\times 256$  images, we divide our runtime by 4. All the other methods use diffusion models trained on FFHQ and produce  $256\times 256$  images.
Figure 7: Results from the web application of our PSLD algorithm,  $512 \times 512$ . The original image (1) is generated by Stable Diffusion v-2.1 with the prompt, "A panda wearing a spiderman costume".
PSLD Image Inpainting
Image inpainting by Posterior Sampling with Latent Diffusion (PSLD)
Given an image (square size preferred) and a user defined mask, click on Inpaint to generate missing parts.
Figure 8: Results from the web application of our PSLD algorithm,  $512 \times 512$ . The original image (1) is generated by Stable Diffusion v-2.1 with the prompt, "A teddy bear showing stop sign at the traffic".
Figure 9: Results from the web application of our PSLD algorithm,  $512 \times 512$ . The original image (1) is generated by Stable Diffusion v-2.1 with the prompt, "A cute dog playing with a toy teddy bear on the lawn".
Table 5: Quantitative random inpainting results on FFHQ 256 validation set [25, 11]. We use Stable Diffusion (v1.5) trained on LAION.
Method	Inpaint (random)	SR (4×)	Gaussian Deblur
PSNR (↑)	SSIM (↑)	PSNR (↑)	SSIM (↑)	PSNR (↑)	SSIM (↑)
PSLD (Ours)	33.71	0.943	30.73	0.867	30.10	0.843
GML-DPS (Ours)	29.49	0.844	29.77	0.860	29.21	0.820
DPS [11]	25.23	0.851	25.67	0.852	24.25	0.811
DDRM [26]	9.19	0.319	25.36	0.835	23.36	0.767
MCG [13]	21.57	0.751	20.05	0.559	6.72	0.051
PnP-ADMM [6]	8.41	0.325	26.55	0.865	24.93	0.812
Score-SDE [50]	13.52	0.437	17.62	0.617	7.12	0.109
ADMM-TV	22.03	0.784	23.86	0.803	22.37	0.801
D Additional Discussion
**Curse of ambient dimension:** DPS [11] suffers from the curse of ambient dimension because in this method, gradients are computed in the pixel space with dimension  $d$ . However, latent-based methods such as PSLD compute gradients in the latent dimension  $k$ , and hence the computation is more efficient. Furthermore, applying the chain rule on VAE and running diffusion in the latent space is less expensive than running diffusion in pixel space directly. In practice, the computational complexity of Stable Diffusion model ( $\sim 4GB$ ) is higher (roughly 6 times) than the computational complexity of the encoder-decoder model ( $\sim 700MB$ ). Therefore, applying the chain rule in the encoder-decoder and running diffusion in the latent space is less expensive than applying diffusion models in the pixel space directly.
(a) Input
(c) DPS [11]
Figure 10: Super-resolution results on images from FFHQ 256 [25, 11] (in distribution).
(a) Input
(c) DPS [11]
Figure 11: Super-resolution results on FFHQ 256 [25, 11] (in distribution).
(d)PSLD (Ours)
Figure 12: Motion deblur results on ImageNet 256 [17] (out-of-distribution).
Table 6: Additional quantitative results on FFHQ 256 validation set [25, 11].
Method	SR (4×)	Gaussian Deblur
FID (↓)	LPIPS (↓)	FID (↓)	LPIPS (↓)
PSLD (Ours)	34.28	0.201	41.53	0.221
DPS [11]	39.35	0.214	44.05	0.257
DDRM [26]	62.15	0.294	74.92	0.332
MCG [13]	87.64	0.520	101.2	0.340
PnP-ADMM [6]	66.52	0.353	90.42	0.441
Score-SDE [50]	96.72	0.563	109.0	0.403
ADMM-TV	110.6	0.428	186.7	0.507
Method	SR (4×)	Gaussian Deblur
PSNR (↑)	SSIM (↑)	PSNR (↑)	SSIM (↑)
PSLD (Ours)	30.73	0.867	30.10	0.843
GML-DPS (Ours)	29.77	0.860	29.21	0.820
DMPS [34]	27.63	-	25.41	-
DPS [11]	25.67	0.852	24.25	0.811
DDRM [26]	25.36	0.835	23.36	0.767
MCG [13]	20.05	0.559	6.72	0.051
PnP-ADMM [6]	26.55	0.865	24.93	0.812
Score-SDE [50]	17.62	0.617	7.12	0.109
ADMM-TV	23.86	0.803	22.37	0.801
(d)PSLD (Ours)
Figure 13: Gaussian deblur results on ImageNet 256 [17] (out-of-distribution).
Table 7: Latent-DPS and PSLD methods evaluated on FFHQ 256 validation set [25, 11]. We use the latent diffusion (LDM-VQ-4) trained on FFHQ 256. Latent-DPS is a special case of PSLD algorithm when  $\gamma = 0$ .
Method	Inpaint (box)
PSNR (↑)	SSIM (↑)	LPIPS (↓)
PSLD	24.22	0.819	0.158
latent-DPS	17.58	0.780	0.21
(d) PSLD (Ours)
Figure 14: Random inpainting results on ImageNet 256 [17] (out-of-distribution).
(d) PSLD (Ours)
Figure 15: Super-resolution (using nearest neighbor kernel from [32]) results on out-of-distribution samples from the web,  $256 \times 256$  (see Table 1 for LPIPS of these images).
(d)PSLD (Ours)
Figure 16: Destriping results on out-of-distribution samples from the web,  $256 \times 256$ . (Top row) Horizontal destriping: LPIPS of  $\mathrm{PSLD} = 0.244$  and DPS [11] = 0.613. (Bottom row) Vertical destriping: LPIPS of  $\mathrm{PSLD} = 0.255$ , DPS [11] = 0.597.
(c) DPS [11]
Figure 17: Additional colorization results on images from FFHQ 256 [25, 11]. PSLD generates photo-realistic color, whereas DPS [11] generates overly saturated images.
Table 8: Runtime (top) and NFEs (bottom) of different posterior sampling algorithms. Runtimes are computed for the super-resolution task.
Method	Runtime (s)
PSLD-LDM	187.00
PSLD-LDM (LAION-400M)	190.00
PSLD-SD (LAION-5B)	194.25
DMPS [34]	67.02
DPS [11]	180.00
DDNM+ [55]	18.5
DDRM [26]	2.15
MCG [13]	193.71
Method	NFEs
PSLD (Ours)	100 to 1000
DPS [11]	1000
DDRM [26]	20
RED [40]	500
IIGDM [46]	20 to 100
Palette [43]	1000
Regression	1
SNIPS [28]	1000