SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we analyze two approximations: GML-DPS based on (6), and PSLD based on (7), displayed in Algorithm 2. We consider the case where the latent distribution of the VAE is in the same space as the latent distribution of the data generating model, i.e.,  $k = l$ , and normalize  $\gamma_{i} = 1$  (as this is immaterial in the linear setting). In Proposition 3.3, we provide analytical solutions for the encoder and the decoder of the VAE.
Proposition 3.3 (Variational Autoencoder). Suppose Assumption 3.1 holds. For an encoder  $\mathcal{E}$  :  $\mathbb{R}^d\to \mathbb{R}^k$  and a decoder  $\mathcal{D}:\mathbb{R}^k\rightarrow \mathbb{R}^d$  , denote by  $\mathcal{L}(\phi ,\omega)$  the training objective of VAE:
$$
\arg \min _ {\phi , \omega} \mathcal {L} (\phi , \omega) := \mathbb {E} _ {\overrightarrow {\boldsymbol {x} _ {0}} \sim p} \left[ \left\| \mathcal {D} (\mathcal {E} (\overrightarrow {\boldsymbol {x} _ {0}}; \phi); \omega) - \overrightarrow {\boldsymbol {x} _ {0}} \right\| _ {2} ^ {2} \right] + \lambda K L \left(\mathcal {E} \sharp_ {p}, \mathcal {N} (\boldsymbol {0}, \boldsymbol {I} _ {k})\right),
$$
then the combination of  $\mathcal{E}(\overrightarrow{\pmb{x}_0};\phi) = \mathcal{S}^T\overrightarrow{\pmb{x}_0}$  and  $\mathcal{D}(\overleftarrow{z_0};\omega) = \mathcal{S}\overleftarrow{z_0}$  is a minimizer of  $\mathcal{L}(\phi ,\omega)$ .
Using the encoder  $\mathcal{E}(\overrightarrow{\pmb{x}_0};\phi) = \mathcal{S}^T\overrightarrow{\pmb{x}_0}$ , we can use the analytical solution  $\theta^{*}$  of the LDM obtained in Theorem A.1. To verify that  $\theta^{*}$  recovers the true subspace  $p(\overrightarrow{\pmb{x}_0})$ , we compose the decoder  $\mathcal{D}(\overleftarrow{z_0};\omega) = S\overleftarrow{z_0}$  with the generator of the LDM, i.e.,  $\overleftarrow{\pmb{x}_0} = \mathcal{D}(\pmb {\theta}^*\overleftarrow{z_1}) = \mathcal{D}(\pmb {I_k}\overleftarrow{z_1}) = \mathcal{S}\overleftarrow{z_1}$ . Since  $\overleftarrow{z_1}\sim \mathcal{N}(\mathbf{0},\mathbf{I}_k)$  and  $\mathcal{S}$  is the data generating model, this shows that  $\overleftarrow{\pmb{x}_0}$  is a sample from  $p(\overrightarrow{\pmb{x}_0})$ . Thus we have the following.
Theorem 3.4 (Generative Modeling using Diffusion in Latent Space). Suppose Assumption 3.1 holds. Let the optimal solution of the latent diffusion model be
$$
\boldsymbol {\theta} ^ {*} = \arg \min  _ {\boldsymbol {\theta}} \mathbb {E} _ {\overrightarrow {\boldsymbol {z} _ {0}}, \overrightarrow {\boldsymbol {\epsilon}}} \left[ \left\| \tilde {\mu} _ {1} \left(\overrightarrow {\boldsymbol {z} _ {1}} (\overrightarrow {\boldsymbol {z} _ {0}}, \overrightarrow {\boldsymbol {\epsilon}}), \overrightarrow {\boldsymbol {z} _ {0}}\right) - \mu_ {\theta} \left(\overrightarrow {\boldsymbol {z} _ {1}} \left(\overrightarrow {\boldsymbol {z} _ {0}}, \overrightarrow {\boldsymbol {\epsilon}}\right)\right) \right\| ^ {2} \right].
$$
For a fixed variance  $\beta > 0$ , if  $\mu_{\theta}\left(\overrightarrow{z_1}(\overrightarrow{z_0}, \overrightarrow{\epsilon})\right) := \theta \overrightarrow{z_1}(\overrightarrow{z_0}, \overrightarrow{\epsilon})$ , then the closed-form solution is  $\pmb{\theta}^* = \sqrt{1 - \beta} \pmb{I}_k$ , which after normalization by  $\frac{1}{\sqrt{1 - \beta}}$  and composition with the decoder  $\mathcal{D}\left(\overleftarrow{z_0}; \omega\right) = S \overleftarrow{z_0}$  recovers the true subspace of  $p\left(\overrightarrow{x_0}\right)$ .
With this optimal  $\theta^{*}$ , we can now prove exact sample recovery using GML-DPS (6).
Theorem 3.5 (Posterior Sampling using Goodness Modified Latent DPS). Let Assumptions 3.1 and 3.2 hold. Let  $\sigma_{j},\forall j = 1,\dots ,r$  denote the singular values of  $(\mathcal{AS})^T (\mathcal{AS})$  , and let
Given a partially known image  $\overrightarrow{\pmb{x}_0} \sim p(\overrightarrow{\pmb{x}_0})$ , any fixed variance  $\beta \in (0,1)$ , then with the (unique) step size  $\eta_i^j = 1/2\sigma_j$ ,  $j = 1,2,\ldots,r$ , the GML-DPS Algorithm (6) samples from the true posterior  $p(\overrightarrow{\pmb{x}_0} | y)$  and exactly recovers the groundtruth sample, i.e.,  $\overleftarrow{\pmb{x}_0} = \overrightarrow{\pmb{x}_0}$ .
Theorem 3.5 shows that GML-DPS (6) recovers the true sample using an LDM. This approach, however, requires the step size  $\eta$  to be chosen coordinate-wise in a specific manner. Also, multiple natural images could have the same measurements in the pixel space. This is a reasonable concern for
LDMs due to one-to-many mappings of the decoder. Note that the goodness objective (Section 2.1) cannot help in this scenario because it assigns uniform probability to many of these latents  $\overleftarrow{z}_1$  for which  $\nabla_{\overleftarrow{z}_1}\left|\left|\overleftarrow{z}_0(\overleftarrow{z}_1)\right| - \mathcal{E}(\mathcal{D}(\overleftarrow{z}_0(\overleftarrow{z}_1)))\right|\right|^2 = 0$ . These challenges motivate the gluing objective in Theorem 3.6. This is crucial for two reasons. First, we show that it helps recover the true sample even when the step size  $\eta$  is chosen arbitrarily. Second, it assigns all the probability mass to the desired (unique) solution in the pixel space.
Theorem 3.6 (Posterior Sampling using Diffusion in Latent Space). Let Assumptions 3.1 and 3.2 hold. Let  $\sigma_{j},\forall j = 1,\dots ,r$  denote the singular values of  $(\mathcal{AS})^T (\mathcal{AS})$  and let
$$
\boldsymbol {\theta} ^ {*} = \arg \min  _ {\boldsymbol {\theta}} \mathbb {E} _ {\overrightarrow {z _ {0}}, \overrightarrow {\boldsymbol {\epsilon}}} \left[ \left\| \tilde {\mu} _ {1} \left(\overrightarrow {z _ {1}} (\overrightarrow {z _ {0}}, \overrightarrow {\boldsymbol {\epsilon}}), \overrightarrow {z _ {0}}\right) - \mu_ {\theta} \left(\overrightarrow {z _ {1}} \left(\overrightarrow {z _ {0}}, \overrightarrow {\boldsymbol {\epsilon}}\right)\right) \right\| ^ {2} \right].
$$
Given a partially known image  $\overrightarrow{\pmb{x}_0} \sim p(\overrightarrow{\pmb{x}_0})$ , any fixed variance  $\beta \in (0,1)$ , and any positive step sizes  $\eta_i^j$ ,  $j = 1,2,\ldots,r$ , the PSLD Algorithm 2 samples from the true posterior  $p(\overrightarrow{\pmb{x}_0} | y)$  and exactly recovers the groundtruth sample, i.e.,  $\overleftarrow{\pmb{x}_0} = \overrightarrow{\pmb{x}_0}$ .
The important distinction between Theorem 3.5 and Theorem 3.6 is that the former requires the exact step size while the latter works for any finite step size. Combining denoising, measurement-consistency (with a scalar  $\eta$ ), and gluing updates, we have
$$
\overleftarrow {z _ {0}} = \boldsymbol {\theta} ^ {*} \overleftarrow {z _ {1}} - \eta \nabla_ {\underline {{z}} _ {1}} \left\| \mathcal {A D} (\overleftarrow {z _ {0}} (\overleftarrow {z _ {1}})) - \boldsymbol {y} \right\| _ {2} ^ {2} - \nabla_ {\underline {{z}} _ {1}} \left\| \overleftarrow {z _ {0}} (\overleftarrow {z _ {1}}) - \mathcal {E} (\mathcal {A} ^ {T} \mathcal {A} \overrightarrow {x _ {0}} + (\boldsymbol {I} _ {d} - \mathcal {A} ^ {T} \mathcal {A}) \mathcal {D} (\overleftarrow {z _ {0}} (\overleftarrow {z _ {1}}))) \right\| _ {2} ^ {2}.
$$
When  $\eta$  is chosen arbitrarily, then the third term guides the reverse SDE towards the optimal solution  $\overline{z_0}$ . When the reverse SDE generates the exact same groundtruth sample, i.e.,  $\mathcal{D}(\overline{z_1} (\overline{z_0})) = \overline{x}_0^{\prime}$ , then the third term becomes zero. For all other samples, it penalizes the reverse SDE. Thus, it forces the reverse SDE to recover the true underlying sample irrespective of the value of  $\eta$ .
We draw the following key insights from our Theorem 3.6: Curse of ambient dimension: In order to run posterior sampling using diffusion in the pixel space, the gradient of the measurement error needs to be computed in the  $d$ -dimensional ambient space. Therefore, DPS algorithm suffers from the curse of ambient dimension. On the other hand, our algorithm uses diffusion in the latent space, and therefore avoids the curse of ambient dimension. Large-scale foundation model: We propose a posterior sampling algorithm which offers the provision to use large-scale foundation models, and it provably solves general linear inverse problems. Robustness to measurement step: The gluing objective makes our algorithm robust to the choice of step size  $\eta$ . Furthermore, it allows the same (scalar) step size across all the coordinates of  $\vec{x_0}$ .
We experiment with in-distribution and out-of-distribution datasets. For in-distribution, we conduct our experiments on a subset of the FFHQ dataset [25] (downscaled to  $256 \times 256^3$ , denoted by FFHQ 256). For out-of-distribution, we use images from the web and ImageNet dataset [17] (resized to  $256 \times 256$ , denoted by ImageNet 256). To make a fair comparison, we use the same validation subset and follow the same masking strategy as the baseline DPS [11]. It is important to note that our main contribution is an algorithm that can leverage any latent diffusion model. We
Table 1: Quantitative super-resolution (using measurement operator from [32]) results on FFHQ 256 validation samples [25, 11]. We use PSLD with Stable Diffusion. Table shows LPIPS  $(\downarrow)$ .
Method	PSLD (Ours)	DPS [11]
2×	0.185	0.220
3×	0.220	0.247
4×	0.233	0.291
To test our algorithm with two pre-trained latent diffusion models: (i) the Stable Diffusion model that is trained on multiple subsets of the LAION dataset [44, 45]; and (ii) the Latent Diffusion model (LDM-VQ-4) trained on the FFHQ 256 dataset [41]. The DPS model is similarly trained from scratch for 1M steps using 49k FFHQ 256 images, which excludes the first 1K images used as validation set.
Inverse Problems. We experiment with the following task-specific measurement operators from the baseline DPS [11]: (i) Box inpainting uses a mask of size  $128 \times 128$  at the center. (ii) Random inpainting chooses a drop probability uniformly at random between  $(0.2, 0.8)$  and applies this drop
<sup>3</sup>https://www.kaggle.com/datasets/denislukovnikov/ffhq256-images-only
Table 2: Quantitative inpainting results on FFHQ 256 validation set [25, 11]. We use Stable Diffusion v-1.5 and the measurement operators as in DPS [11]. As shown, our PSLD model outperforms DPS since it is able to leverage the power of the Stable Diffusion foundation model.
Method	Inpaint (random)	Inpaint (box)	SR (4×)	Gaussian Deblur
FID (↓)	LPIPS (↓)	FID (↓)	LPIPS (↓)	FID (↓)	LPIPS (↓)	FID (↓)	LPIPS (↓)
PSLD (Ours)	21.34	0.096	43.11	0.167	34.28	0.201	41.53	0.221
DPS [11]	33.48	0.212	35.14	0.216	39.35	0.214	44.05	0.257
DDRM [26]	69.71	0.587	42.93	0.204	62.15	0.294	74.92	0.332
MCG [13]	29.26	0.286	40.11	0.309	87.64	0.520	101.2	0.340
PnP-ADMM [6]	123.6	0.692	151.9	0.406	66.52	0.353	90.42	0.441
Score-SDE [50]	76.54	0.612	60.06	0.331	96.72	0.563	109.0	0.403
ADMM-TV	181.5	0.463	68.94	0.322	110.6	0.428	186.7	0.507
probability to all the pixels. (iii) Super-resolution downsamples images at  $4\times$  scale. (iv) Gaussian blur convolves images with a Gaussian blur kernel. (v) Motion blur convolves images with a motion blur kernel. We also experiment with these additional operators from RePaint [32]: (vi) Super-resolution downsamples images at  $2\times$ ,  $3\times$ , and  $4\times$  scale. (vii) Denoising has Gaussian noise with  $\sigma = 0.05$ . (viii) Dstriping has vertical and horizontal stripes in the input images.
Evaluation. We compare the performance of our PSLD algorithm with the state-of-the-art DPS algorithm [11] on random inpainting, box inpainting, denoising, Gaussian deblur, motion deblur, arbitrary masking, and super-resolution tasks. We show that PSLD outperforms DPS, both in-distribution and out-of-distribution datasets, using the Stable Diffusion v-1.5 model pre-trained on the LAION dataset. We also test PSLD with LDM-VQ-4 trained on FFHQ 256, to compare with DPS trained on the same data distribution. Note that the LDM-v4 is a latent-based model released prior to Stable Diffusion. Therefore, it does not match the performance of Stable Diffusion in solving inverse problems. However, it shows the general applicability of our framework to leverage an LDM in posterior sampling. Since Stable Diffusion v-1.5 is trained with an image resolution of  $512 \times 512$ , we apply the forward operator after upsampling inputs to  $512 \times 512$ , run posterior sampling at  $512 \times 512$  and then downsample images to the original  $256 \times 256$  resolution for a fair comparison with DPS. We observed a similar performance while applying the masking operator at  $256 \times 256$  and upscaling to  $512 \times 512$  before running PSLD. More implementation details are provided in Appendix C.1.
Metrics. We use the commonly used Learned Perceptual Image Patch Similarity (LPIPS), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Metric (SSIM), and Fréchet Inception Distance<sup>4</sup> (FID) metrics for quantitative evaluation.
Results. Figure 2 shows the inpainting results on out-of-distribution samples. This experiment was performed on commercial platforms that use (to the best of our knowledge) Stable diffusion and additional proprietary models. This evaluation was performed on models deployed in May 2023 and may change as commercial providers improve their platforms.
The qualitative advantage of PSLD is clearly demonstrated in Figures 2, 3, 4, 15 and 16. In Figure 5, we compare PSLD and DPS in random inpainting task for varying percentage of dropped pixels. Quantitatively, PSLD outperforms DPS in commonly used metrics: LPIPS, PSNR, and SSIM.
In our PSLD algorithm, we use Stable Diffusion v1.5 model and (zero-shot) test it on inverse problems. Table 6 compares the quantitative results of PSLD with related works on random inpainting, box inpainting, super-resolution, and Gaussian deblur tasks. PSLD significantly outperforms previous approaches on the relatively easier random inpainting task, and it is better or comparable on harder tasks. Table 4 draws a comparison between PSLD and the strongest baseline (among the compared methods) on out-of-distribution images. Table 1 shows the super-resolution results using nearest-neighbor kernels from [32] on FFHQ 256 validation dataset. Observe that PSLD outperforms state-of-the-art methods across diverse tasks and standard evaluation metrics.
In Table 3, we compare PSLD (using LDM-VQ-4) and DPS on random and box inpainting tasks with the same operating resolution  $(256\times 256)$  and training distributions (FFHQ 256). Although the LDM model exceeds DPS performance in box inpainting, it is comparable in random inpainting. As expected, using a more powerful pre-trained model such as Stable Diffusion is beneficial in
4https://github.com/mseitzer/pytorch-fid
(b) Groundtruth
(c) Comm. Serv. 1
(d) Comm. Serv. 2
Figure 2: Inpainting results in general domain images from the web (see Appendix C for image sources). Our model compared to state-of-art commercial inpainting services that leverage the same foundation model (Stable Diffusion v-1.5).
(e) PSLD (Ours)
Table 3: Quantitative inpainting results on FFHQ 256 validation set [25, 11]. We use the latent diffusion (LDM-VQ-4) trained on FFHQ 256. Note that in this experiment PSLD and DPS use diffusion models trained on the same dataset. As shown, PSLD with LDM-VQ-4 as diffusion model outperforms DPS in box inpainting and has comparable performance in random inpainting.
Method	Inpaint (random)	Inpaint (box)
PSNR (↑)	SSIM (↑)	LPIPS (↓)	PSNR (↑)	SSIM (↑)	LPIPS (↓)
PSLD (Ours)	30.31	0.851	0.221	24.22	0.819	0.158
DPS [11]	29.49	0.844	0.212	23.39	0.798	0.214
Table 4: Quantitative results of random inpainting and denoising on FFHQ 256 [25, 11] using Stable Diffusion v-1.5. Note that DPS is trained on FFHQ 256. The results show that our method PSLD generalizes well to out-of-distribution samples even without finetuning.
Method	Random inpaint + denoise σ = 0.00	Random inpaint + denoise σ = 0.05
PSNR (↑)	SSIM (↑)	LPIPS (↓)	PSNR (↑)	SSIM (↑)	LPIPS (↓)
PSLD (Ours)	34.02	0.951	0.083	33.71	0.943	0.096
DPS [11]	31.41	0.884	0.171	29.49	0.844	0.212
reconstruction—see Table 6. This highlights the significance of our PSLD algorithm that has the provision to incorporate a powerful foundation model with no extra training costs for solving inverse problems. Importantly, PSLD uses latent-based diffusion, and thus it avoids the curse of ambient dimension (Theorem 3.6), while still achieving comparable results to the state-of-the-art method DPS [11] that has been trained on the same dataset. Additional experimental evaluation is provided in Appendix C.
In this paper, we leverage latent diffusion models to solve general linear inverse problems. While previously proposed approaches only apply to pixel-space diffusion models, our algorithm allows us to use the image prior learned by latent-based foundation generative models. We provide a principled analysis of our algorithm in a linear two-step diffusion setting, and use insights from this analysis to design a modified objective (goodness and gluing). This leads to our algorithm - Posterior
(c) DPS [9]
(d) PSLD (Ours)
Figure 3: Left panel: Random Inpainting on images from FFHQ 256 [25] using PSLD with Stable Diffusion v-1.5. Notice the text in the top row and the facial expression in the bottom row. Right panel: Block  $(128\times 128)$  inpainting, using the LDM-VQ-4 model trained on FFHQ 256 [25]. Notice the glasses in the top row and eyes in the bottom row.
(d)PSLD (Ours)
(c) DPS [11]
(d) PSLD (Ours)
Figure 4: Inpainting (random and box) results on out-of-distribution samples,  $256 \times 256$  (see Appendix C for image sources). We use PSLD with Stable Diffusion v-1.5 as generative foundation model.
Figure 5: Comparing DPS and PSLD performance in random inpainting on FFHQ 256 [25, 11], as the percentage of masked pixels increases. PSLD with Stable Diffusion outperforms DPS.
Sampling with Latent Diffusion (PSLD) – that experimentally outperforms state-of-art baselines on a wide variety of tasks including random inpainting, block inpainting, denoising, destriping, and super-resolution.
Limitations. Our evaluation is based on Stable Diffusion which was trained on the LAION dataset. Biases in this dataset and foundation model will be implicitly affecting our algorithm. Our method can work with any LDM and we expect new foundation models trained on better datasets like [19] to mitigate these issues. Second, we have not explored how to use latent-based foundation models to solve non-linear inverse problems. Our method builds on the DPS approximation (which performs well on non-linear inverse problems), and hence we believe our method can also be similarly extended.
This research has been supported by NSF Grants 2019844, 2112471, AF 1901292, CNS 2148141, Tripods CCF 1934932, the Texas Advanced Computing Center (TACC) and research gifts by Western Digital, Wireless Networking and Communications Group (WNCG) Industrial Affiliates Program, UT Austin Machine Learning Lab (MLL), Cisco and the Stanly P. Finch Centennial Professorship in Engineering. Litu Rout has been supported by the Ju-Nam and Pearl Chew Endowed Presidential Fellowship in Engineering. Giannis Daras has been supported by the Onassis Fellowship (Scholarship ID: F ZS 012-1/2022-2023), the Bodossaki Fellowship and the Leventis Fellowship. We thank the HuggingFace team for providing us GPU support for the demo of our work.
[51] Yu Takagi and Shinji Nishimoto. "High-resolution image reconstruction with latent diffusion models from human brain activity". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 14453-14463 (page 3).
[52] Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. "Plug-and-play priors for model based reconstruction". In: 2013 IEEE Global Conference on Signal and Information Processing. IEEE. 2013, pp. 945-948 (page 1).
[53] Pascal Vincent. “A connection between score matching and denoising autoencoders”. In: Neural computation 23.7 (2011), pp. 1661–1674 (page 1).
[54] Su Wang, Chitwan Sahara, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. "Imagen editor and editbench: Advancing and evaluating text-guided image inpainting". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 18359-18369 (pages 1, 21).
[55] Yinhuai Wang, Jiwen Yu, and Jian Zhang. “Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model”. In: The Eleventh International Conference on Learning Representations. 2023. URL: https://openreview.net/forum?id=mRieQgMtNTQ (page 31).
[56] Jiahui Yu, Zhe Lin, Jamei Yang, Xiaohui Shen, Xin Lu, and Thomas Huang. "Free-Form Image Inpainting With Gated Convolution". In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (Oct. 2019). DOI: 10.1109/iccv.2019.00457. URL: http://dx.doi.org/10.1109/iccv.2019.00457 (page 1).
A Additional Theoretical Results
Notation and Measurement Matrix. We elaborate on the structure of the measurement matrix  $\mathcal{A} \in \mathbb{R}^{l \times d}$ . In our setting, we are considering linear inverse problems. Thus, this matrix is a pixel selector and consists of a subset of the rows from the  $d \times d$  identity matrix (the rows that are present correspond to the indices of the selected pixels from the image  $\vec{\pmb{x}_0} \in \mathbb{R}^d$ ). Given this structure, it immediately follows that  $\mathcal{A}^T \mathcal{A}$  is a  $d \times d$  matrix that has the interpretation of a pixel selection mask. Specifically,  $\mathcal{A}^T \mathcal{A}$  is a  $d \times d$  diagonal matrix  $D(m)$ , where the elements of  $m$  are set to 1 where data (pixel) is observed and 0 where data (pixel) is masked. Without the loss of generality, we suppose that the first  $k$  coordinates are known.
A.1 Posterior Sampling using Pixel-space Diffusion Model
We first consider the reverse process, starting with  $\overleftarrow{\pmb{x}_1} \sim \mathcal{N}(\pmb{0},\pmb{I}_d)$ , and borrow a result from [42] to show that the sample  $\overleftarrow{\pmb{x}_0}$  generated by the reverse process is a valid image from  $p(\overrightarrow{\pmb{x}_0})$ .
Theorem A.1 (Generative Modeling using Diffusion in Pixel Space, [42]). Suppose Assumption 3.1 holds. Let
$$
\boldsymbol {\theta} ^ {*} = \arg \min  _ {\boldsymbol {\theta}} \mathbb {E} _ {\overrightarrow {\boldsymbol {x} _ {0}}, \overrightarrow {\boldsymbol {\epsilon}}} \left[ \left\| \tilde {\mu} _ {1} \left(\overrightarrow {\boldsymbol {x} _ {1}} (\overrightarrow {\boldsymbol {x} _ {0}}, \overrightarrow {\boldsymbol {\epsilon}}), \overrightarrow {\boldsymbol {x} _ {0}}\right) - \mu_ {\boldsymbol {\theta}} \left(\overrightarrow {\boldsymbol {x} _ {1}} \left(\overrightarrow {\boldsymbol {x} _ {0}}, \overrightarrow {\boldsymbol {\epsilon}}\right)\right) \right\| ^ {2} \right].
$$
For a fixed variance  $\beta > 0$ , if  $\mu_{\theta}\left(\overrightarrow{\boldsymbol{x}_1} \left(\overrightarrow{\boldsymbol{x}_0}, \overrightarrow{\boldsymbol{\epsilon}}\right)\right) := \theta \overrightarrow{\boldsymbol{x}_1} \left(\overrightarrow{\boldsymbol{x}_0}, \overrightarrow{\boldsymbol{\epsilon}}\right)$ , then the closed-form solution  $\theta^*$  is  $\sqrt{1 - \beta} \boldsymbol{S} \boldsymbol{S}^T$ , which after normalization by  $1 / \sqrt{1 - \beta}$  recovers the true subspace of  $p\left(\overrightarrow{\boldsymbol{x}_0}\right)$ .
Though this establishes that  $\overleftarrow{\pmb{x}_0}$  generated by the reverse process is a valid image from  $p(\overrightarrow{\pmb{x}_0})$ , it is not necessarily a sample from the posterior  $p(\overrightarrow{\pmb{x}_0}|\pmb{y})$  that satisfies the measurements. To accomplish this we perform one additional step of gradient descent for every step of the reverse process. This gives us Algorithm 1, the DPS algorithm. The next theorem shows that the reverse SDE guided by these measurements (3) recovers the true underlying sample<sup>5</sup>.
Theorem A.2 (Posterior Sampling using Diffusion in Pixel Space). Suppose Assumption 3.1 and Assumption 3.2 hold. Let us denote by  $\sigma_{j},\forall j = 1,\dots ,r$  , the singular values of  $(\mathcal{A}\mathcal{S})^T (\mathcal{A}\mathcal{S})$  and