SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Notice that the minimizer in Eq 4.1 is not entirely of the form we originally desired, which was  $\pmb{h}(A, \pmb{y}_t, t) = \mathbb{E}[\pmb{x}_0 \mid A\pmb{x}_t, A]$ . In place of  $A$ , we now have  $\tilde{A}$ , which is a further degraded matrix.
$^{1}\mathrm{Ber}(q)$  indicates a Bernoulli random variable with a probability of  $q$  to equal 1 and  $1 - q$  for 0.
Indeed, one trivial way to satisfy the condition in Theorem 4.1 is by forming  $\tilde{A}$  completely independently of  $A$ , e.g. by always setting  $\tilde{A} = 0$ . However, in this case, the function we learn is not very useful. For this reason, we would like to add as little further noise as possible and ensure that  $\tilde{A}$  is close to  $A$ . In natural noise models such as the inpainting noise model, by letting the additional corruption probability  $\delta$  approach 0, we can indeed ensure that  $\tilde{A}$  follows a distribution very close to that of  $A$ .
5.1 Training from scratch on corrupted data
Our first experiment is to train diffusion models from scratch using corrupted training data at different levels of corruption. The corruption model we use for these experiments is random inpainting: we form our dataset by deleting each pixel with probability  $p$ . To create the matrix  $\tilde{A}$ , we further delete each row of  $A$  with probability  $\delta$  – this removes an additional  $\delta$ -fraction of the surviving pixels. Unless mentioned otherwise, we use  $\delta = 0.1$ . We train models on CIFAR-10, AFHQ, and CelebAHQ. All our models are trained with corruption level  $p \in \{0.0, 0.2, 0.4, 0.6, 0.8, 0.9\}$ . We use the EDM [30] codebase to train our models. We replace convolutions with Gated Convolutions [64] which are known to perform better for inpainting-type problems. To use the mask  $\tilde{A}$  as an additional input to the model, we simply concatenate it with the image  $x$ . The full training details can be found in the Appendix, Section C.
We first evaluate the restoration performance of our model for the task it was trained on (random inpainting and noise). We compare with state-of-the-art diffusion models that were trained on clean data. Specifically, for AFHQ we compare with the state-of-the-art EDM model [30] and for CelebA we compare with DDIM [56]. These models were not trained to denoise, but we can use the prior learned in the denoiser as in [60, 29] to solve any inverse problem. We experiment with the state-of-the-art reconstruction algorithms: DDRM [34] and DPS [11].
We summarize the results in Table 1. Our model performs similarly to other diffusion models, even though it has never been trained on clean data. Further, it does so by requiring only one step, while all the baseline diffusion models require hundreds of steps to solve the same task with inferior or comparable performance. The performance of DDRM improves with more function evaluations at the cost of more computation. For DPS, we did not observe significant improvement by increasing the number of steps to more than 100. We include results with noisy inpainted measurements and comparisons with a supervised method in the Appendix, Section E, Tables 3, 5. We want to emphasize that all the baselines we compare against have an advantage: they are trained on uncorrupted data. Instead, our models were only trained on corrupted data. This experiment indicates that: i) our training algorithm for learning the conditional expectation worked and ii) that the choice of corruption that diffusion models are trained to reverse matters for solving inverse problems.
Next, we evaluate the performance of our diffusion models as generative models. To the best of our knowledge, the only generative baseline with quantitative results for training on corrupted data is AmbientGAN [7] which is trained on CIFAR-10. We further compare with a diffusion model trained without our further corruption algorithm. We plot the results in Figure 3. The diffusion model trained without our further corruption algorithm performs well for low corruption levels but collapses entirely for high corruption. Instead, our model trained with further corruption maintains reasonable corruption scores even for high corruption levels, outperforming the previous state-of-the-art AmbientGAN for all ranges of corruption levels.
For CelebA-HQ and AFHQ we could not find any generative baselines trained on corrupted data to compare against. Nevertheless, we report FID and Inception Scores and summarize our results in Table 4 to encourage further research in this area. As shown in the Table, for CelebA-HQ and AFHQ, we manage to maintain a decent FID score even with  $90\%$  of the pixels deleted. For CIFAR-10, the performance degrades faster, potentially because of the lower resolution of the training images.
5.2 Finetuning foundation models on corrupted data
We can apply our technique to finetune a foundational diffusion model. For all our experiments, we use Deepfloyd's IF model [2], which is one of the most powerful open-source diffusion generative
Dataset	Corruption Probability	Method	LPIPS	PSNR	NFE
CelebA-HQ	0.6	Ours	0.037	31.51	1
DPS	0.053	28.21	100
DDRM	0.139	25.76	35
0.088	27.38	99
0.069	28.16	199
0.8	0.8	Ours	0.084	26.80	1
DPS	0.107	24.16	100
DDRM	0.316	20.37	35
0.188	22.96	99
0.153	23.82	199
0.9	0.9	Ours	0.152	23.34	1
DPS	0.168	20.89	100
DDRM	0.461	15.87	35
0.332	18.74	99
0.242	20.14	199
AFHQ	0.4	Ours	0.030	33.27	1
DPS	0.020	34.06	100
DDRM	0.122	25.18	35
0.091	26.42	99
0.088	26.52	199
0.6	0.6	Ours	0.062	29.46	1
DPS	0.051	30.03	100
DDRM	0.246	20.76	35
0.166	22.79	99
0.160	22.93	199
0.8	0.8	Ours	0.124	25.37	1
DPS	0.107	25.30	100
DDRM	0.525	14.56	35
0.295	18.08	99
0.258	18.86	199
Table 1: Comparison of our model (trained on corrupted data) with state-of-the-art diffusion models on CelebA (DDIM [56] model) and AFHQ (EDM [30] model) for solving the random inpainting inverse problem. Our model performs on par with state-of-the-art diffusion inverse problem solvers, even though it has never seen uncorrupted training data. Further, this is achieved with a single score function evaluation. To solve this problem with a standard pre-trained diffusion model we need to use a reconstruction algorithm (such as DPS [11] or DDRM [34]) that typically requires hundreds of steps.
models available. We choose this model over Stable Diffusion [51] because it works in the pixel space (and hence our algorithm directly applies).
**Memorization.** We show that we can finetune a foundational model on a limited dataset without memorizing the training examples. This experiment is motivated by the recent works of Carlini et al. [8], Somepalli et al. [55], and Jagielski et al. [27] that show that diffusion generative models memorize training samples and they do it significantly more than previous generative models, such as GANs, especially when the training dataset is small. Specifically, Somepalli et al. [55] train diffusion models on subsets of size  $\{300, 3000, 30000\}$  of CelebA and they show that models trained on 300 or 3000 memorize and blatantly copy images from their training set.
We replicate this training experiment by finetuning the IF model on a subset of CelebA with 3000 training examples. Results are shown in Figure 1. Standard finetuning of Deepfloyd's IF on 3000 images memorizes samples and produces almost exact copies of the training set. Instead, if we corrupt the images by deleting  $80\%$  of the pixels prior to training and finetune, the memorization decreases sharply and there are distinct differences between the generated images and their nearest neighbors from the dataset. This is in spite of finetuning until convergence.
Figure 3: Performance on CIFAR-10 as a function of the corruption level. We compare our method with a diffusion model trained without our further corruption trick and AmbientGAN [7]. Ambient Diffusion outperforms both baselines for all ranges of corruption levels.
Figure 4: Inception/FID results on random inpainting for models trained with our algorithm on CelebA-HQ, AFHQ and CIFAR-10.
Dataset	Corruption Probability	FID	Inception Score
CelebA-HQ	0.0	3.26	N/A
0.2	4.18
0.6	6.08
0.8	11.19
0.9	25.53
AFHQ	0.0	2.41	N/A
0.2	4.47
0.4	6.96
0.6	10.11
0.8	16.78
0.9	41.00
CIFAR-10	0.0	1.85	9.94
0.2	11.70	7.97
0.4	18.85	7.45
0.6	28.88	6.88
0.8	46.27	6.14
Figure 5: Left panel: We finetune Deepfloyd's IF diffusion model to make it a generative model for MRI images of brains with tumors. We use a small dataset [25] of only 155 images that was corrupted by removing large blocks as shown. Right panel: Generated samples from our finetuned model. As shown, the model learns the statistics of full brain tumor MRI images. The training set was resized to  $64 \times 64$  but the generated images are at  $256 \times 256$ . The higher resolution is obtained by simply leveraging the power of the cascaded IF model.
To quantify the memorization, we follow the methodology of Somepalli et al. [55]. Specifically, we generate 10000 images from each model and we use DINO [9]-v2 [46] to compute top-1 similarity to the training images. Results are shown in Figure 6. Similarity values above 0.95 roughly correspond to the same person while similarities below 0.75 typically correspond to random faces. The standard finetuning (Red) often generates images that are near-identical with the training set. Instead, finetuning with corrupted samples (blue) shows a clear shift to the left. Visually we never observed a near-copy generated from our process - see also Figure 1.
We repeat this experiment for models trained on the full CelebA dataset and at different levels of corruption. We include the results in Figure 8 of the Appendix. As shown, the more we increase the corruption level the more the distribution of similarities shifts to the left, indicating less memorization. However, this comes at the cost of decreased performance, as reported in Table 4.
New domains and different corruption. We show that we can also finetune a pre-trained foundation model on a new domain given a limited-sized dataset in a few hours in a single GPU. Figure 5 shows generated samples from a finetuned model on a dataset containing 155 examples of brain tumor MRI images [25]. As shown, the model learns the statistics of full brain tumor MRI images while only trained on brain-tumor images that have a random box obfuscating  $25\%$  of the image. The training set was resized to  $64 \times 64$  but the generated images are at  $256 \times 256$  by simply leveraging the power of the cascaded Deepfloyd IF.
Figure 6: Distribution of similarity values to the nearest neighbor in the dataset for a finetuned IF model on a 3000 samples CelebA subset. Please note that similarity values above 0.95 roughly correspond to the same person and similarities below 0.75 typically correspond to random faces. Therefore the baseline finetuning process (red) often generates images that are near copies of the training set. On the contrary, our fine-tuning with corrupted samples (blue) shows a clear shift to the left. Visually we never observed a near-identical image generated from our process, see also Figure 1 for qualitative results.
Limitations. Our work has several limitations. First, there is a tradeoff between generator quality and corruption levels. For higher corruption, it is less likely that our generator memorizes parts of training examples, but at a cost of degrading quality. Precisely characterizing this trade-off is an open research problem. Further, in this work, we only experimented with very simple approximation algorithms to estimate  $\mathbb{E}[\pmb{x}_0|\pmb{x}_t]$  using our trained models. Additionally, we cannot make any strict privacy claim about the protection of any training sample without making assumptions about the data distribution. We show in the Appendix that it is possible to recover  $\mathbb{E}[\pmb{x}_0|\pmb{x}_t]$  exactly using our restoration oracle, but we do not have an algorithm to do so. Finally, our method cannot handle measurements that also have noise. Future work could potentially address this limitation by exploiting SURE regularization as in [1].
The authors would like to thank Tom Goldstein for insightful discussions that benefited this work. This research has been supported by NSF Grants CCF 1763702, AF 1901292, CNS 2148141, Tripods CCF 1934932, NSF AI Institute for Foundations of Machine Learning (IFML) 2019844, the Texas Advanced Computing Center (TACC) and research gifts by Western Digital, WNCG IAP, UT Austin Machine Learning Lab (MLL), Cisco and the Archie Straiton Endowed Faculty Fellowship. Giannis Daras has been supported by the Onassis Fellowship (Scholarship ID: F ZS 012-1/2022-2023), the Bodossaki Fellowship and the Leventis Fellowship. Aravind Gollakota was at UT Austin while this work was done.
[1] Asad Aali, Marius Arvinte, Sidharth Kumar, and Jonathan I Tamir. "Solving Inverse Problems with Score-Based Generative Priors learned from Noisy Data". In: arXiv preprint arXiv:2305.01166 (2023) (page 10).
[2] Stability AI. Deepfloyd IF. https://github.com/deep-floyd/IF. 2013 (pages 7, 18).
[3] Brian D.O. Anderson. "Reverse-time diffusion equation models". In: Stochastic Processes and their Applications 12.3 (1982), pp. 313-326 (page 3).
[58] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. "Score-based generative modeling through stochastic differential equations". In: arXiv preprint arXiv:2011.13456 (2020) (pages 1, 3, 18).
[59] Julián Tachella, Dongdong Chen, and Mike Davies. "Unsupervised Learning From Incomplete Measurements for Inverse Problems". In: arXiv preprint arXiv:2201.12151 (2022) (page 2).
[60] Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. "Plug-and-play priors for model based reconstruction". In: 2013 IEEE Global Conference on Signal and Information Processing. IEEE. 2013, pp. 945-948 (page 7).
[61] Pascal Vincent. “A connection between score matching and denoising autoencoders”. In: Neural computation 23.7 (2011), pp. 1661–1674 (page 4).
[62] Ge Wang, Jong Chul Ye, and Bruno De Man. "Deep learning for tomographic image reconstruction". In: Nature Machine Intelligence 2.12 (2020), pp. 737-748 (page 2).
[63] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas Huang. “Free-Form Image Inpainting With Gated Convolution”. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (Oct. 2019). DOI: 10.1109/iccv.2019.00457. URL: http://dx.doi.org/10.1109/iccv.2019.00457 (page 2).
[64] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. “Free-form image inpainting with gated convolution”. In: Proceedings of the IEEE/CVF international conference on computer vision. 2019, pp. 4471–4480 (pages 7, 17).
[65] Yide Zhang, Yinhao Zhu, Evan Nichols, Qingfei Wang, Siyuan Zhang, Cody Smith, and Scott Howard. "Aoisson-gaussian denoising dataset with real fluorescence microscopy images". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, pp. 11710-11718 (page 2).
Proof of Theorem 4.1. Let  $h_{\theta^*}$  be a minimizer of equation 3.2, and for brevity let
$$
\boldsymbol {f} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) = \boldsymbol {h} _ {\theta *} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) - \mathbb {E} [ \boldsymbol {x} _ {0} \mid \tilde {A} \boldsymbol {x} _ {t}, \tilde {A} ]
$$
be the difference between  $h_{\theta^*}$  and the claimed optimal solution. We will now argue that  $f$  must be identically zero.
First, by adding and subtracting  $A\mathbb{E}[x_0 \mid \tilde{A} x_t, \tilde{A}]$ , we can expand the objective value achieved at  $\theta = \theta^*$  in equation 3.2 as follows:
$$
\begin{array}{l} J ^ {\mathrm {c o r r}} \left(\theta^ {*}\right) = \mathbb {E} _ {\boldsymbol {x} _ {0}, \boldsymbol {x} _ {t}, A, \tilde {A}} \left[ \right.\left\| \right.\left(A \boldsymbol {x} _ {0} - A \mathbb {E} [ \boldsymbol {x} _ {0} \mid \tilde {A} \boldsymbol {x} _ {t}, \tilde {A} ]\right) - A \boldsymbol {f} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A})\left. \right)\left. \right\| ^ {2} \left. \right] \\ = \mathbb {E} _ {\boldsymbol {x} _ {0}, \boldsymbol {x} _ {t}, A, \tilde {A}} \left[ \left\| A \boldsymbol {x} _ {0} - A \mathbb {E} [ \boldsymbol {x} _ {0} \mid \tilde {A} \boldsymbol {x} _ {t}, \tilde {A} ] \right\| ^ {2} \right] + \mathbb {E} _ {\boldsymbol {x} _ {0}, \boldsymbol {x} _ {t}, A, \tilde {A}} \left[ \left\| A \boldsymbol {f} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) \right\| ^ {2} \right] \\ - 2 \mathbb {E} _ {\boldsymbol {x} _ {0}, \boldsymbol {x} _ {t}, A, \tilde {A}} \left[ \left(A \boldsymbol {x} _ {0} - A \mathbb {E} [ \boldsymbol {x} _ {0} \mid \tilde {A} \boldsymbol {x} _ {t}, \tilde {A} ]\right) ^ {T} A \boldsymbol {f} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) \right]. \\ \end{array}
$$
Here the first term is the irreducible error, while the third term vanishes by the tower law of expectations:
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x} _ {0}, \boldsymbol {x} _ {t}, A, \tilde {A}} \left[ (A \boldsymbol {x} _ {0} - A \mathbb {E} [ \boldsymbol {x} _ {0} | \tilde {A} \boldsymbol {x} _ {t}, \tilde {A} ]) ^ {T} A \boldsymbol {f} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) \right] \\ = \mathbb {E} _ {\boldsymbol {x} _ {0}, A, \tilde {A}, \tilde {A} \boldsymbol {x} _ {t}} \left[ \left(\boldsymbol {x} _ {0} - \mathbb {E} [ \boldsymbol {x} _ {0} \mid \tilde {A} \boldsymbol {x} _ {t}, \tilde {A} ]\right) ^ {T} A ^ {T} A \boldsymbol {f} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) \right] \\ = \mathbb {E} _ {A, \tilde {A}, \tilde {A} \boldsymbol {x} _ {t}} \left[ \mathbb {E} _ {\boldsymbol {x} _ {0} | A, \tilde {A}, \tilde {A} \boldsymbol {x} _ {t}} \left[ \boldsymbol {x} _ {0} - \mathbb {E} [ \boldsymbol {x} _ {0} \mid \tilde {A} \boldsymbol {x} _ {t}, \tilde {A} ] \right] ^ {T} A ^ {T} A \boldsymbol {f} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) \right] \\ = \mathbb {E} _ {A, \tilde {A}, \tilde {A} \boldsymbol {x} _ {t}} \left[ \left(\mathbb {E} [ \boldsymbol {x} _ {0} \mid \tilde {A} \boldsymbol {x} _ {t}, \tilde {A} ] - \mathbb {E} [ \boldsymbol {x} _ {0} \mid \tilde {A} \boldsymbol {x} _ {t}, \tilde {A} ]\right) ^ {T} A ^ {T} A \boldsymbol {f} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) \right] \\ = 0. \\ \end{array}
$$
Thus the only part of  $J^{\mathrm{corr}}(\theta^{*})$  that actually depends on the parameter value  $\theta^{*}$  is the second term. We now show that the second term can be made to vanish, and that this occurs precisely when  $\pmb{f}$  is identically 0:
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x} _ {0}, \boldsymbol {x} _ {t}, A, \tilde {A}} \left[ \left\| A \boldsymbol {f} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) \right\| ^ {2} \right] \\ = \mathbb {E} _ {\boldsymbol {x} _ {0}, \boldsymbol {x} _ {t}, A, \tilde {A}} \left[ \boldsymbol {f} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) ^ {T} A ^ {T} A \boldsymbol {f} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) \right] \\ = \mathbb {E} _ {\boldsymbol {x} _ {0}, \boldsymbol {x} _ {t}, \tilde {A}} \left[ \boldsymbol {f} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) ^ {T} \mathbb {E} _ {A | \boldsymbol {x} _ {0}, \boldsymbol {x} _ {t}, \tilde {A}} \left[ A ^ {T} A \right] f (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) \right] \\ = \mathbb {E} _ {\boldsymbol {x} _ {0}, \boldsymbol {x} _ {t}, \tilde {A}} \left[ f (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) ^ {T} \mathbb {E} _ {A | \tilde {A}} \left[ A ^ {T} A \right] \boldsymbol {f} (\tilde {A} \boldsymbol {x} _ {t}, \tilde {A}) \right]. \\ \end{array}
$$
For every  $\tilde{A}$  and  $\boldsymbol{x}_0, \boldsymbol{x}_t$ , by assumption we have that  $\mathbb{E}_{A|\tilde{A}}[A^T A]$  is full-rank, and so the inner quadratic form is minimized when  $\boldsymbol{f}(\tilde{A}\boldsymbol{x}_t, \tilde{A}) = 0$ . Further, the term as a whole vanishes exactly when this holds for every  $\tilde{A}$  and  $\boldsymbol{x}_0, \boldsymbol{x}_t$  in the support, which means  $\boldsymbol{f}$  must be identically zero.
Corollary A.1 (Inpainting noise model). Consider the following inpainting noise model:  $A \in \mathbb{R}^{n \times n}$  is a diagonal matrix where each entry  $A_{ii} \sim \mathrm{Ber}(1 - p)$  for some  $p > 0$  (independently for each  $i$ ), and the additional noise is generated by drawing  $\tilde{A}|A$  such that  $\tilde{A}_{ii} = A_{ii} \mathrm{Ber}(1 - \delta)$  for some small  $\delta > 0$  (again independently for each  $i$ ). Then the unique minimizer of the objective in equation 3.2 is
$$
\boldsymbol {h} _ {\theta^ {*}} (\tilde {\boldsymbol {A}} \boldsymbol {x} _ {t}, \tilde {\boldsymbol {A}}) = \mathbb {E} [ \boldsymbol {x} _ {0} \mid \tilde {\boldsymbol {A}} \boldsymbol {x} _ {t}, \tilde {\boldsymbol {A}} ].
$$
Proof. By Theorem 4.1, what we must show is that for any  $\tilde{A}$  in the support,  $E_{A|\tilde{A}}[A^T A]$  is full-rank. Fix any particular realization of  $\tilde{A}$ , which will be a diagonal matrix with only 0s and 1s. For indices  $i$  where  $\tilde{A}_{ii} = 1$ , we know that for any  $A$  drawn conditional on  $\tilde{A}$ ,  $A_{ii} = 1$  as well, i.e.
$$
\Pr (A _ {i i} = 1 \mid \tilde {A} _ {i i} = 1) = 1.
$$
And for indices  $i$  where  $\tilde{A}_{ii} = 0$ , by Bayes' rule we have
$$
\operatorname * {P r} (A _ {i i} = 1 \mid \tilde {A} _ {i i} = 0) = \frac {\operatorname * {P r} (A _ {i i} = 1 , \tilde {A} _ {i i} = 0)}{\operatorname * {P r} (A _ {i i} = 0 , \tilde {A} _ {i i} = 0) + \operatorname * {P r} (A _ {i i} = 1 , \tilde {A} _ {i i} = 0)} = \frac {(1 - p) \delta}{(1 - p) \delta + p} =: q.
$$
Thus we see that  $E_{A|\tilde{A}}[A^T A] = E_{A|\tilde{A}}[A]$  is a diagonal matrix whose entries are 1 wherever  $\tilde{A}_{ii} = 1$  and  $q$  wherever  $\tilde{A}_{ii} = 0$ . This is clearly of full rank since  $q > 0$ .
Corollary A.2 (Gaussian measurements noise model). Consider the following noise model where we only observe  $m$  independent Gaussian measurements of the ground truth, and then one of the measurements is further omitted:  $A \in \mathbb{R}^{m \times n}$  consists of  $m$  rows drawn independently from  $\mathcal{N}(0, I_n)$ , and  $\tilde{A} \in \mathbb{R}^{m \times n}$  is constructed conditional on  $A$  by zeroing out its last row. Then the unique minimizer of the objective in equation 3.2 is
Proof. Again by Theorem 4.1, we must show that for any  $\tilde{A}$  in the support,  $E_{A|\tilde{A}}[A^T A]$  is full-rank. Fix any realization of  $\tilde{A}$ , which will have the following form:
$$
\tilde {A} = \left[ \begin{array}{c c c} \_ & \boldsymbol {a} _ {1} ^ {T} & \_ \\ & \vdots & \\ \_ & \boldsymbol {a} _ {m - 1} ^ {T} & \_ \\ \_ & 0 ^ {T} & \_ \end{array} \right].
$$
Then it is clear that conditional on  $\tilde{A}$ ,  $A$  has the following distribution:
$$
A \mid \tilde {A} = \left[ \begin{array}{c c c} - & \boldsymbol {a} _ {1} ^ {T} & - \\ & \vdots & \\ - & \boldsymbol {a} _ {m - 1} ^ {T} & - \\ - & \boldsymbol {b} _ {m} ^ {T} & - \end{array} \right] \quad \text {w h e r e} \quad \boldsymbol {b} _ {m} \sim \mathcal {N} (0, I _ {n}).
$$
Here  $\pmb{b}_{m}$  is drawn entirely independently from  $\mathcal{N}(0,I_n)$ . Elementary manipulations now reveal that  $\mathbb{E}_{A|\tilde{A}}[A^T A] = \mathbb{E}_{\pmb{b}_m\sim \mathcal{N}(0,I_n)}[\tilde{A}^T\tilde{A} +\pmb {b}_m\pmb {b}_m^T ] = \tilde{A}^T\tilde{A} +I_n$ , which is clearly full rank (indeed, it is PSD with strictly positive eigenvalues).
A.1 Reduction
In this section we argue that if there is an algorithm that recovers the target distribution  $p_0(\pmb{x}_0)$  from i.i.d. samples  $(A\pmb{x}_0, A)$  where  $A \sim p(A)$  and  $\pmb{x}_0 \sim p_0(\pmb{x}_0)$ , then, there is an algorithm that recovers  $p_0(\pmb{x}_0)$  without sample access, but instead, using access to an oracle that given  $t, \pmb{x}$  and  $A$  in the support of  $p(A)$ , returns  $\mathbb{E}[\pmb{x}_0 \mid A\pmb{x}_t]$ .