Ambient Diffusion: Learning Clean Distributions from Corrupted Data
Authors: Giannis Daras
Kulin Shah
Yuval Dagan
Aravind Gollakota
Alex Dimakis
Adam Klivans
Venue: NeurIPS 2023 poster (2023)
Keywords: corrupted data
generative models
ambient gan
inverse problems
learning from measurements
TL;DR: We present the first diffusion-based framework to learn an unknown distribution when the training set only contains highly-corrupted examples drawn from it.

Abstract: We present the first diffusion-based framework that can learn an unknown distribution using only highly-corrupted samples. This problem arises in scientific applications where access to uncorrupted samples is impossible or expensive to acquire. Another benefit of our approach is the ability to train generative models that are less likely to memorize any individual training sample, since they never observe clean training data. 

Our main idea is to introduce additional measurement distortion during the diffusion process and require the model to predict the original corrupted image from the further corrupted image.  We prove that our method leads to models that learn the conditional expectation of the full uncorrupted image given this additional measurement corruption.  This holds for any corruption process that satisfies some technical conditions (and in particular includes inpainting and compressed sensing).  We train models on standard benchmarks (CelebA, CIFAR-10 and AFHQ) and show that we can learn the distribution even when all the training samples have 90\% of their pixels missing. We also show that we can finetune foundation models on small corrupted datasets (e.g. MRI scans with block corruptions) and learn the clean distribution without memorizing the training set.

Ambient Diffusion: Learning Clean Distributions from Corrupted Data
Giannis Daras UT Austin gi ammisdaras@utexascco edu
Kulin Shah UT Austin
Yuval Dagan UC Berkeley yuvald@berkeley..o edu
kul mulinshah@tteas.. edu
Aravind Gollakota
Alexandros G. Dimakis
Adam Klivans UT Austin
UT Austin aravindgocs utexas. edu dimakis@austin, utexas. edu
Apple
kli klivans@utexas.c edu
Abstract
We present the first diffusion-basse framework that can learn an unknown dis- tribution using only applications where access to uncorrupted samples impossible or expensive acquire. Another benefit of our approach is the ability to train generative models that are less likely to memorize any individual training sample, since they never observe clean training data. Our main idea is to introduce additional measurement distortion during the diffusion process and require the model to predict the original corrupted image from the further corrupted image. We prove that method Ieads to models that learn the conditional expectation of the full uncorrupted image given this additional measurement corruption. This holds for any corruption pro- cess that satisfies some technical conditions (and in particular includes inpainting and compressed sensing). We train models standard benchmarks (CelebA, CIFAR- 10 and AFHQ) and show that we can learn the distribution even when all the training samples have 90% of their pixels missing. We also show that finetune foundation models on small corrupted datasets (e.g. MRI scans with block corruptions) and learn the clean distribution without memorizing the training
samples. This problem arises in scientific
1 Introduction
Diffusion generative models [54, 23, 57] emerging versatile and powerful frameworks for learning high-dimensional distributions and solving inverse problems [34, 11, 36, 28). Numerous recent developments [58, 30] have led text conditional foundation models like Dalle-2 [45], Latent Diffusion [50] and Imagen [52] with an incredible performance in general image domains. Training these models requires access high-quality datasets which may be expensive impossible obtain. For example, direct images of black holes cannot be observed [12, 19] and high quality MRI images require long scanning times, causing patient discomfort and motion artifacts [28].
Recently, Carlini et al. [8], Somepalli et al. [55]. and Jagielski et al. [27] showed that diffusion models memorize examples from their training set. Further, adversary can extract dataset samples given only query access to the model, leading to privacy, security and copyright concerns. For many applications, may want to learn the distribution but not individual training images e.g. might want to learn the distribution of X-ray scans but not memorize images of specific patient scans from the dataset. Hence, we may want introduce corruption as a design choice. We show that it is possible to train diffusions that learn a distribution of clean data by only observing highly corrupted samples.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Training Images
Corrupted Training Images
40
samples
Nearest neighbors
Generated samples
Nearest neighbors
Figure 1: Left panel: Baseline method of vanilla finetuning Deepfloyd IF using 3000 images from CelebA-HQ. We show generated sample images and nearest neighbors from the finetuning set. As shown, the generated samples are often near-identical copies from training data. This verifies related work Carlini et al. [8], Somepalli et al. [55], and Jagielski et al. [27] that pointed out that diffusions often generate training samples. Right panel: We finetune the same foundation model (Deepfloyd IF) using our method and 3000 highly corrupted training images. The corruption adds noise and removes 80 percent random pixels. We show generated samples and nearest neighbors from the training set. Our method still learns the clean distribution of faces (with some quality deterioration. as shown) but does not memorize training data. We emphasize that our training is performed without ever accessing clean training data.
Prior work in supervised learning from corrupted data. The traditional approach to solving such problems involves training a restoration model using supervised learning to predict the clean image based on the measurements [47, 48, 63, 41]. The seminal Noise2Noise [39] work introduced a practical algorithm for learning how to denoise in the absence any non-noisy images. This framework and its generalizations! [5, 38, 59] have found applications in electron microscopy [16], tomographic image reconstruction [62]. fluorescence image reconstruction [65], blind inverse prob- lems [20, 5], monocular depth estimation and proteomics [6]. Another related line of work uses Stein's Unbiased Risk Estimate (SURE) to optimize an unbiased estimator of the denoising objective without access non-noisy data [18]. We stress that the aforementioned research works study the problem of restoration, whereas are interested in the problem of sampling from the clean distribution. Restoration algorithms based on supervised learning are only effective when the corruption level is relatively low [15]. However, it might be either not possible or not desirable to reconstruct individual samples. Instead, the desired goal may to learn generate fresh and completely unseen samples from the distribution of the uncorrupted data but without reconstructing individual training samples.
Indeed, for certain corruption processes, it is theoretically possible perfectly learn distribution only from highly corrupted samples (such as just random one-dimensional projections), even though individual sample denoising usually impossible in such settings.
AmbientGAN [7] showed that general dimensional distributions can be learned from scalar micddmmenioonl random Gaussian vectors, in the
observations. by observing only projections infinite training data limit. The theory requires an infinitely powerful discriminator and hence does not apply to diffusion models. MCFlow [49] is framework, based on a variant of the EM algorithm, that can be used to train normalizing flow models from missing data. Finally, MIWAE [42] and Not-MIWAE [26] frameworks leam deep latent models (e.g. VAEs) from missing data when the corruption process is known or unknown respectively.
Our contributions. We present the first diffusion-based framework to learn an unknown distribution D when the training set only contains highly-corrupedd examples drawn from D. Specifically, we consider the problem of learning to sample from the target distribution po(xo) given corrupted

---

Random
in- Further corrup- tion
inpaint- Further corrup- ing
Block
painting
tion
Figure 2: Illustration of our method: Given training data with deleted pixels, corrupt further by erasing (illustrated with green color). We feed the learner the further corrupted images and evaluate it the originally observed pixels. We can do this during training since the green pixel values are known us. The score network learner has no way of knowing whether a pixel was missing from the beginning or whether was corrupted by us. Hence, the score network learns to predict the clean image everywhere. Our method is analogous to grading random subset of the questions in test, but the students not knowing which questions will be graded.
samples Azo where A ~ p(A) is a random corruption matrix (with known realizations and prior distribution) and 3o (oo). Our main idea is introduce additional measurement distortion during the diffusion process and require the model to predict the original corrupted image from the further corrupted image.
We provide algorithm that provably learns B[zo|A(<xo + o.m), AI, for all noise levels t and for À ~ p(A A) being further corrupted version of A. The result holds for general family of corruption processes A ~ p(A). For various corruption processes, we show that the further degradation introduced by A can be very small.
We use our algorithm to train diffusion models on standard benchmarks (CelebA, CIFAR-10 and AFHQ) with training data at different levels of corruption.
Given the learned conditional expectations we provide an approximate sampler for the target distribution Po (2o).
We show that for up to 90% missing pixels, we can learn reasonably well the distribution of uncorrupted images. We outperform the previous state-of-the-art AmbientGAN [7] and
natural baselines.
We show that our models perform on par even outperform state-of- aart diffusion models for solving certain inverse problems even without ever seeing clean image during training. Our models do with single prediction step while our basclines require hundreds diffusion steps.
We use our algorithm to finetune foundational pretrained diffusion models. Our finetuning can be done few hours on a single GPU and we can use it to learn distributions with few corrupted samples.
We show that models trained on sufficiently corrupted data do not memorize their training
set. We measure the tradeoff between the amount of corruption (that controls the degree of memorization), the amount of training data and the quality of the learned generator.
We open-source code and models:
2 Background
Training a diffusion model involves two steps. First, we design a corruption process that transforms the data distribution gradually into distribution that can sample from [58, 14). Typically, this corruption process is described by an Ito SDE of the form: dx = f(x, t) + (t)ddw, where un is reversible and the reverse process is also described by an Ito SDE [3]: dæ (f( t) g² Vt) p. dt + g(t) The designer of the diffusion model is usually free to choose the drift function f(...) and the diffusion function
the standard Wiener process. Such corruption processes
g(-). Typical choices are setting f(z,t) t) g(t / da? (Variance Exploding SDE) setting {(x -B(()ar, = /B(T) (Variance Preserving SDE). Both of these choices lead to Gaussian terminal distribution and are equivalent a linear transformation the input. The goal of diffusion model training is to leamn the function V, log Pe (x), which is known as the score function.
3

---

To simplify the presentation of the paper, we will focus on the Variance Exploding SDE that leads to conditional distributions z; = 2o + o.7)
Vincent [61] showed that we can learn the score function at level & by optimizing for the score- matching objective:
J(0) I (m).m.) [ ((),))))))) 1) zoll"
(2.1)
Specifically, the score function can be written in terms of the minimizer of this objective
Va, pe(c)) hg- (Ft, St.
(2.2)
This result reveals a fundamental connection between the score-function and the best restoration model of To given æt, known Tweedie's Formula [17]. Specifically, the optimal hg- (x t) is given by E [[ which means that
best restoration
E[zo| xz
V æ, log
(2.3)
de
Inspired by this restoration interpretation of diffusion models, the Soft/Cold Diffusion works [14, 4] generalized diffusion models look at non-Markovian corruption processes: 2; = C,ay + o, Specifically, Soft Diffusion proposes the Soft Score Matching objective:
Jooft (0)
2 E (wn. ||C, (he (3 () xo)ll²
(2.4)
and shows that it is sufficient to recover the score function via generalized Tweedie's Formula:
Va, pe C,Elaao [t xt
(2.5)
O2
For these generalized models, the matrix C. is a design choice (similar to how we could choose the functions f) Most importantly, for 0, the matrix C, becomes the identity matrix and the noise or becomes zero, i.e. we observe samples from the true distribution.
Method
As explained in the introduction, in many design (to avoid memorization and leaking of sensitive data) or because it is impossible to obtain clean data. Here we study the case where learner only has access linear measurements of the clean data, i.c. non-in non-invertible corruption matrices. We ask two questions:
do observe uncorrupted images æo, either by
Aro. and the corruption matrices A Rmxxn We note that we are interested in
1. Is it possible to learn E[ + a.1), A] for all noise levels I, given only access to corrupted samples (yo Axo, A)?
2. If is it possible for any noise level f, and thus sample from the true distribution through the score function given by Tweedie's formula (Eq. 2.3)?
this restoration model E [[zg]|A(a [A(z) a.n), A] recover E[xy x.)
We investigate these questions in the rest of the paper. For the first, the answer is affirmative but only after introducing additional corruptions, as explain below. For the second, at every time step t, we approximate E[z0 [x] directly using E[æo|Ax, A] (for a chosen A) and substitute into Eq. 2.3. Empirically, we observe that the resulting approximate sampler yields good results.
3.1 Training
For the sake of clarity, we first consider the case random inpainting. the image 2i is viewed as a vector, we can think of the matrix A as a diagonal matrix with ones in the entries that correspond to the preserved pixels and zeros in the erased pixels. We assume that p( () samples matrix where each entry in the diagonal is sampled i..dd with probability 1 - p to be and p to be zero.

---

We would like to train function hg which receives corruption matrix A and a noisy version of a corrupted image, yr A cD o. where v) NN I). and produces an estimate for the
conditional expectation. The simplest idea would be simply ignore the missing pixels and optimize for:
Joore (0) ( ||A (he Az,, xu)ll
(3.1)
Despite the similarities with Soft Score Matching (Eq 2.4), this objective will not learn the conditional expectation. The reason is that the learner is never penalized for performing arbitrarily poorly in the missing pixels. Formally, any function hg satisfying Ahg. (A. yr. t) A Ax, A] is a minimizer.
Instead, we propose to further corrupt the samples before feeding them the model, and ask the model to predict the original corrupted sample from the further corrupted image.
Concretely, we randomly corrupt A to obtain À = BA for some matrix that is selected randomly given A. In our example of missing pixels, Á obtained from by randomly erasing an additional fraction 8 of the pixels that survive after the corruption A. Here, B will be diagonal where each element is with probability 1 1-6 and 0 w.p. We will penalize the model on recovering all the pixels that are visible in the sample Axo: this includes both the pixels that survive in Axg and those that are erased by Å. The formal training objective given by minimizing the following loss:
Joom (0)
B n A.A) || A (no(A, Ax, 1) - mo)
(3.2)
The key idea behind our algorithm is as follows: the learner does not know if a missing pixel is missing because never had it (and hence do not know the ground truth) because it was deliberately erased as part of the further corruption (in which case we do know the ground truth). Thus, the best learner cannot be inaccurate in the unobserved pixels because with non-zero probability it might be evaluated on some of them. Notice that the trained model behaves as a denoiser in the observed pixels and as an inpainter in the missing pixels. We also want emphasize that the probability å of further corruption can be arbitrarily small as long as it stays positive.
The idea of further corruption can be generalized from the case of random inpainting to much broader family of corruption processes. For example, if A a random Gaussian matrix with m rows, we can form Á by deleting one row from at random. If A is a block inpainting matrix (i.e. random block of fixed size missing from all of the training images), we can create À by corrupting further with one more non-overlapping missing block. Examples of our further corruption are shown in Figure 2. In our Theory Section, we prove conditions under which it is possible to recover E[xo| Axx A] using our algorithm and samples (3/ = Axo. A). Our goal is to satisfy this condition while adding minimal further corruption, i.e. while keeping A close to A.
3.2 Sampling
To sample from Pa (xa) using the standard diffusion formulation, we need access to V me log Pa (xt). which is equivalent having access to E[ [x:] (see Eq. 2.3). Instead, our model is trained to predict E[Zg Aw, A] for all matrices A in the support of p(A).
We note that for random inpainting, the identity matrix is technically in the support of p(A). However, the corruption probability p least a constant, the probability of seeing the identity matrix is exponentially small in the dimension of æ;, Hence, we should not expect our model to give good estimates of Ax;, A] for corruption matrices A that belong to the tails of the distribution p(A).
The simplest idea is to sample a mask À ~ p(A) and approximate Elzo axr] with Elaco| Ax,, A).
Under this approximation, the discretized sampling rule becomes:
Tt At
V2
0,
o At
(ao Aace A)
(3.3)
01
Tt
7
I 1-76
5

---

This idea works surprisingly well. Unless mentioned otherwise, we use it for all the experiments in the main paper and we show that we can generate samples that are reasonably close the true distribution (as shown by metrics such as FIID and Inception) even with 90% of the pixels missing.
We include pseudocode for sampling in Algorithm 1. In the Appendix (Section E.4), ablate an alternative choice for sampling that can lead to slight improvements at the cost of increased function evaluations.
Algorithm Ambient Diffusion Fixed Mask Sampling (for VE SDE)
Input: noise schedule {orJō. model parameters 0, and step size Al for discretizing the Reverse ODE.
Sample FT o2)
Sample initial noise Sample a mask
Sample À p(A)
t 1 T
repeat
y: t Ax
> Create measurements D Use the network to estimate E(aro Aax, A] Run one step of the Reverse ODE approximating
Lg (y:: A,
3:-Ar +
xt +
al Lo
a:
B B| [r]] with to
t 3- t At
until S At
return Z0
We underline that the Fixed Mask Sampler does not come with theoretical guarantee of sampling from po (20). In the Appendix, corrupted samples, it is also possible to reconstruct it using access to B[zol AAx A] (Lemma A.3). However, as stated in the Limitations section, we were not able to find any practical algorithm to do so. Nevertheless, experimentally the Fixed Mask Sampler has strong performance and beats previously proposed baselines for different classes of generative models.
prove that whenever it is possible reconstruct po(ara) from
4 Theory
As elaborated in Section 3, one of our key goals is to learn the best restoration model for the measurements all noise levels, i.e., the function h (A, yt. E(wo yt, A). We now show that under a certain assumption on the distribution of and Á, the true population minimizer Eq. 3.2 is indeed essentially of the form above. This assumption formalizes the notion that even conditional on A, A has considerable variability, and the latter ensures that the best way to predict Axg as function of Ax, and Á is to optimally predict zo itself. All proofs are deferred to the Appendix.
Theorem 4.1. Assume a joint distribution corruption matrices A and further corruption Å. for all Á in the support holds that E AAA A] is full-rank, then the unique minimizer of the objective in equation 3.2 is given by
hgg (A, t), = E, (x0 Ax,, A]
(4.1)
Two simple examples that fit into this framework (see Corollaries A.I and A.2 in the Appendix) are:
Inpainting: A E pax= is diagonal matrix where each entry A. Ber(1 p) for some p 0 (independently for each i), and the additional noise is generated by drawing Á|A such that Ä A Ber(1 8) for some small 8 > 0 (again independently for each i).' E Rmxx consists of m rows drawn independently from (00,, I,), and À E mR constructed conditional on by zeroing out its last row.
Gaussian measurements:
Notice that the minimizer in Eq 4.1 is not entirely of the form we originally desired, which was h (A, ()
E[zo Ac,, A). In place of A, we now have Á, which is further degraded matrix.
BBr((g) indicates Bernoulli random variable with probability of g equal and for 0.

---

Indeed, one trivial way to satisfy the condition in Theorem 4.1 is by forming A completely indepen- dently of A. c.g. by always setting A = O. However, in this case, the function we learn is not very useful. For this reason, we would like to add as little further noise as possible and ensure that A close to A. In natural noise models such as the inpainting noise model, by letting the additional corruption probability ò approach 0, we can indeed ensure that À follows distribution very close to that of A.
5 Experimental Evaluation
5.1 Training from scratch on corrupted data
Our first experiment to train diffusion models from scratch using corrupted training data at different levels of corruption. The corruption model we use for these experiments is random inpainting: we form our dataset by deleting each pixel with probability p. To create the matrix A, we further delete each row of with probability ó this removes an additional 8-fraction of the surviving pixels. Unless mentioned otherwise, we use 5 = 0.1. We train models on CIFAR- 10, AFHQ, and CelebA- HQ. All our models are trained with corruption level p E {0.0, 0.2, 0.4, 0.6, 0.8, 0.9). We use the EDM [30] codebase to train our models. We replace convolutions with Gated Convolutions [64] which are known to perform better for inpainting-type problems. To use the mask Á as additional input to the model, we simply concatenate it with the image æ. The full training details can be found in the Appendix, Section C.
We first evaluate the restoration performance of our model for the task it was trained on (random inpainting and noise). We compare with saate-of-the-aee diffusion models that were trained on clean data. Specifically, for AFHQ we compare with the state-of-the-arr EDM model [30] and for CelebA we compare with DDIM [56]. These models were not trained to denoise, but we can use the prior learned in the denoiser as in [60, 29] to solve any inverse problem. We experiment with the state-of-the-aa reconstruction algorithms: DDRM [34] and DPS [11].
We summarize the results in Table 1. though it has never been trained on clean data. Further, it does so by requiring only one step, while all the baseline diffusion models require hundreds of steps solve the same task with inferior or comparable performance. The performance of DDRM improves with more function evaluations at the cost of more computation. For DPS, we did not observe significant improvement by increasing the number of steps to more than 100. We include results with noisy inpainted measurements and comparisons with a supervised method in the Appendix, Section E, Tables 3, 5. We want to emphasize that all the baselines we compare against have an advantage: they are trained on uncorrupted data. Instead, our models were only trained on corrupted data. This experiment indicates that: i) our training algorithm for learning the conditional expectation worked and ii) that the choice of corruption that diffusion models are trained to reverse matters for solving inverse problems.
Our model performs similarly other diffusion models, even
Next, we evaluate the performance of our diffusion models as generative models. To the best of our knowledge, the only generative baseline with quantitative results for training on corrupted data is AmbientGAN [7] which is trained on CIFAR-10. We further compare with a diffusion model trained without our further corruption algorithm. We plot the results in Figure The diffusion model trained without our further corruption algorithm performs well for low corruption levels but collapses entirely for high corruption. Instead, our model trained with further corruption maintains reasonable corruption scores even for high corruption levels, outperforming the previous state-of-the- art AmbientGAN for all ranges of corruption levels.
For CelebA-HQ and AFHQ we could not find any generative baselines trained on corrupted data to compare against. Nevertheless, report FID and Inception Scores and summarize our results in Table 4 to encourage further research in this area. As shown in the Table, for CelebA-HQ and AFHQ, we manage to maintain a decent FID score even with 90% of the pixels deleted. For CIFAR-10, the performance degrades faster, potentially because of the lower resolution of the training images.
5.2 Finetuning foundation models on corrupted data
We can apply our technique to finetune a foundational diffusion model. For all our experiments, we use Deepfloyd's s IF model [2], which is one of the most powerful open-source diffusion generative
7

---

Dataset
Corruption Probability
Method
LPIPS
PSNR
NFE
CclebA-HQ
Ours
0.037
31.51
1
DPS
0.053
28.21
100
0.6
0.139
25.76
35
DDRM
0.088
27.38
99
0.069
28.16
199
Ours
0.084
26.80
DPS
0.107
24.16
100
0.8
0.316
20.37
35
DDRM
0.188
22.96
99
0.153
23.82
199
Ours
0.152
23.34
DPS
0.168
20.89
100
0.9
0.461
15.87
35
DDRM
0.332
18.74
99
0.242
20.14
199
AFHQ
Ours
0.030
33.27
DPS
0.020
34.06
100
0.4
0.122
25.18
35
DDRM
0.091
26.42
99
0.088
26.52
199
Ours
0.062
29.46
DPS
0.051
30.03
100
0.6
0.246
20.76
35
DDRM
0.166
22.79
99
0.160
22.93
199
Ours
0.124
25.37
1
DPS
0.107
25.30
100
0.8
0.525
14.56
35
DDRM
0.295
18.08
99
0.258
18.86
199
Table 1: Comparison our model (trained on corrupted data) with state-of ofttee--tt-ertt diffusion models on CelebA (DDIM [56] model) and AFHQ (EDM [30] model) for solving the random inpainting inverse problem. Our model performs on par with statc-of-the-ar art diffusion inverse problem solvers, though it has never uncorrupted training data. Further, this achieved with single score function evaluation. To solve this problem with a standard pre-trained diffusion model we need to reconstruction algorithm (such as DPS [11] DDRM (34)) that typically requires hundreds of steps.
models available. We choose this model over Stable Diffusion [51] because it works in the pixel space (and hence our algorithm directly applies).
Memorization. We show that can finetune foundational model on a limited dataset without memorizing the training examples. This experiment is motivated by the recent works of Carlini et al. [8]. Somepalli et al. [55], and Jagielski et al. [27] that show that diffusion generative models memorize training samples and they do it significantly more than previous generative models, such as GANs, especially when the training dataset is small. Specifically, Somepalli al. [55] train diffusion models on subsets of size (300, 3000, 30000) of CelebA and they show that models trained on 300 or 3000 memorize and blatantly copy images from their training set.
We replicate this training experiment by finetuning the IF model on subset of CelebA with 3000 training examples. Results are shown in Figure 1. Standard finetuning of Deepfloyd's ' IF on 3000 images memorizes samples and produces almost exact copies of the training set. Instead, we corrupt the images by deleting 80% of the pixels prior to training and finetune, the memorization decreases sharply and there are distinct differences between the generated images and their nearest neighbors from the dataset. This is in spite of finetuning until convergence.

---

CIFAR-10 Inception Scores
10
Amb entGAN
Dutaset
Corruption Probability
FID
Inception Score
Diffusion No Further Corruption
Arbiintt Diffusion
0.2
4.18
Inception Score
Ce
0.6
6.08
N/A
0.8
11.19
0.9
25.53
7
0.0
2.41
0.2
6
0.4
AFHQ
0.6
10.11
N/A
5
0.9
41,00
4
0.2
11.70
7.97
0.2
CIFAR-10
0.4
18.85
7.45
Corruption Probability
0.6
28.88
6.88
0.8
46.27
6.14
Figure 3: Performance on CIFAR-10 10 as a function of the corruption level. We compare our method with dif- fusion model trained without our further coruuption trick AmbientGAN [7]. Ambient Diffusion outperforms both baselines for all ranges of corruption levels.
Figure 4: Inception/FID results on random in-
painting for models trained with algorithm on CelchA-HQ. AFHQ and CIFAR-10.
Figure 5: Left panel: We finetune Deepfloyd's IF diffusion model to make it generative model for MRI images of brains with tumors. We use small dataset [25] of only 135 images that was corrupted by removing large blocks as shown. Right panel: Generated samples from our finetuned model. the statistics full brain tumor MRI images. The training set was resized to 64 x 64 but the generated images are at 256 x 256. The higher resolution is obtained by simply leveraging the power of the cascaded IF model.
shown, the model learns
To quantify the memorization, we follow the methodology of Somepalli et al. [55]. Specifically, we generate 10000 images from each model and the training images. Results are shown in Figure 6. Similarity values above 0.95 roughly correspond the same person while similarities below 0.75 typically correspond to random faces. The standard finetuning (Red) often generates images that are near-ida caar-idetiiall with the training set. Instead, fine- tuning with corrupted samples (blue) shows clear shift to the left. Visually never observed near-copy generated from our process see also Figure 1.
DINO [9]-v2 [46] to compute top-1 similarity to
We repeat this experiment for models trained on the full CelebA dataset and at different levels of corruption. We include the results in Figure 8 of the Appendix. As shown, the more we increase the соrrptiion level the more the distribution of similarities shifts to the left, indicating less memorization. However, this comes at the cost of decreased performance, as reported in Table 4.
New domains and different corruption. We show that we can also finetune pre-trained foundation model on a new domain given a limited-sizze dataset in few hours single GPU. Figure 5 shows generated samples from finetuned model on dataset containing 155 examples of brain tumor MRI images [25]. As shown, the model learns the statistics of full brain tumor MRI images while only trained on brain-tumor images that have random box obfuscating 25% of the image. The training set was resized 64 64 the generated images are 256 x 256 by simply leveraging the power of the cascaded Deepfloyd IF.

---

Distribution of Similarity Values to the nearest training sample
680
Corruption probabiltty 0.8
Corruption probabilty
Number of generated samples
400
200
100
0.60
0.65
Similarity Value
Figure 6: Distribution of similarity values to the nearest neighbor in the dataset for a finetuned IF model on 3000 samples CelebA subset. Please note that similarity values above 0.95 roughly correspond to the same person and similarities below 0.75 typically correspond to random faces. Therefore the baseline finetuning process (red) often generates images that are near copies of the training set. On the contrary, our fine-tuning with corrupted samples (blue) shows a clear shift to the left. Visually we never observed a near-identical image generated from our process, see also Figure for qualitative results.
Limitations. Our work has several limitations. First, there is tradeoff between generator quality and comuptiin levels. For higher corruption, it is less likely that generator memorizes parts of training examples, but at a cost of degrading quality. Precisely characterizing this trade-off an open research problem. Further, in this work, only experimented with very simple approximation algorithms to estimate E[(oola,, using our trained models. Additionally, we cannot make any strict privacy claim about the protection of any training sample without making assumptions about the data distribution. We show in the Appendix that it is possible to recover EE| [[ exactly using our restoration oracle, but do not have an algorithm do so. Finally, our method cannot handle measurements that also have noise. Future work could potentially address this limitation by exploiting SURE regularization as in [1].
Acknowledgments and Disclosure of Funding
The authors would like to thank Tom Goldstein for insightful discussions that benefited this work.
This research has been supported by NSF Grants CCF 1763702, AF 1901292, CNS 2 2148141, Tripods CCF 1934932, NSF AI Institute for Foundations of Machine Learning (IFML) 201 9019844, the Texas Advanced Computing Center (TACC) and research gifts by Western Digital, WNCG IAP, UT Austin Machine Learning Lab (MLL), Cisco and the Archie Straiton Endowed Faculty Fellowship. Giannis Daras has been supported by the Onassis Fellowship (Scholarship ID: F ZS 112-1/2022-2025 the Bodossaki Fellowship and the Leventis Fellowship. Aravind Gollakota UT Austin while this work was done.
References
[1] Asad Aali, Marius Arvinte, Sidharth Kumar, and Jonathan Tamir. "Solving Inverse Prob- lems with Score-Based Generative Priors learned from Noisy Data". In: arXiv preprint arXiv:2305.011
(2023) (page 10).
ceitthuu.com/deepp.... cow/deep-floydeeeeeeeeo 2013 (pages 7, 18). [3] Brian D.O. Anderson. "Reverse-time diffusion equation models". In: Stochastic Processes and their Applications 12.3 (1982), pp. 313-326 (page 3).
[2] Stability AI. Deepfloyd IF.
10

---

[4] Arpit Bansal, Eitan Borgnia, Hong- Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. "Cold diffusion: Inverting arbitrary image transforms without noise". In: arXiv preprini arXiv:2208. 209229900 (2022) (page 4).
[5] Joshua Batson and Loic Royer. "Noise2self: Blind denoising by self- self-supervision". In: Interna- tional Conference on Machine Learning. PMLR. 2019, pp. 524-533 (page 2).
[6] Felix JB Bäuerlein and Wolfgang Baumeister. "Towards visual proteomics at high resolution".
In: Journal of Molecular Biology 433.20 33 (2021), 167187 (page 2).
[7] Ashish Bora, Eric Price, and Alexandros G Dimakis. "AmbientGAN: Generative models from lossy measurements". In: International conference on learning representations. 2018 (pages 2,
3, 9).
[8] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski. Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. "Extracting Training Data from Large Language Models". In: 30th USENIX Secu- rity Symposium (USENIX Security 21). 2021, pp. 2633-2650. ISBN: 978-1-939133-24-33 URL: https: / /www. (visited 11/10/2022) (pages 1, 8).
carlini- usenix. org /oooffrencce / wttp/useiisseeee pttprt/tyty//eetttttooon/o
[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. "Emerging properties in self-supervised vision transformers". In: Pro- ceedings of the IEEE/CVF international conference on computer vision. 2021, pp. 9650-9660 (page 9).
[10] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. "Sampling is easy as learning the score: theory for diffusion models with minimal data assumptions". In: arXiv preprint arXiv: 1215 (2022) (pages 16, 17).
[1]] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. "Diffusion Posterior Sampling for General Noisy Inverse Problems", In: The Eleventh International Conference on Learning Representations, 2023. URL: https:/ /penrevieee
net/forummin wttffformmiddddddd wtw.:m//iddd0000AAG000000 (pages 1, 7, 8, 22).
[12] The Event Horizon Telescope Collaboration et al. "First M87 Event Horizon Telescope Results. IV. Imaging the Central Supermassive Black Hole". In: The Astrophysical Journal Letters
875.1 (Apr. 2019), p. L4. DOE: 10. 3847 / 2041- 8213/ ab0e85. URL: https: / //dxx doi.
org/10. 10// 3847/2041 220/2224111122 (01--8221/abbbee
(page 1).
[13] Zhuo-Xu Cui, Chentao Cao. Shaonan Liu, Qingyong Zhu, Jing Cheng, Haifeng Wang, Yanjie Zhu, and Dong Liang. "Self-score: Self-supervised learning on score-based models for mri reconstruction". In: arXiv preprint arXivv2209.0085 (2022).
[14] Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alex Dimakis, and Peyman Milanfar. "Soft Diffusion: Score Matching with General Corruptions". In: Transactions on Machine Learning Research (2023). ISSN: 2835-8856. URL: https: http:/:///pp/eeeeeeeeee weteforrumiicommmmmmm
w98rebBx1Q (pages 3, 4).
[15] Mauricio Delbracio and Peyman Milanfar. "Inversion by direct iteration: An alternative denoising diffusion for image restoration". In: arXiv preprint arXiv:2303.11435 (2023)
(page 2).
[16] Jeffrey M Ede. "Deep learning in electron microscopy". In: Machine Learning: Science and Technology 2.1 (2021), p. 01 1004 (page 2).
[17] Bradley Efron. "Tweedie's formula and selection bias". In: Journal of the American Statistical
Association 106.496 1), pp. 1602-1614 (page 4).
[18] Yonina C. Eldar. "Generalized SURE for Exponential Families: Applications to Regulariza- tion". In: JEEE Transactions on Signal Processing 57.2 (2009), pp. 471-481. DOI: 10. 1109/ TSP 2008. 2008212 (page 2).
[19] Angela F Gao, Oscar Leong, He Sun, and Katherine L Bouman. "Image Reconstruction without Explicit Priors". In: ICASSP 202.3-003 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2023, pp. 1-5 (page 1).
[20] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. "Toward convolutional blind denoising of real photographs" In: Proceedings of the IEEF/CVF conference on computer vision and pattern recognition. 2019, pp. 1712-1722 (page 2).
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. "Masked autoencoders are scalable vision learners".. In: Proceedings of the IEFF/CV Conference on Computer Vision and Pattern Recognition. 2022, pp. 16000-16009 (pages 19, 20).
11

---

[22] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. "Gans trained by two time-scale update rule converge a local nash equilibrium". In: Advances in neural information processing systems 30 (2017) (page 18).
[23]
Jonathan Ho, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models". In: Advances in Neural Information Processing Systems 33 (2020), pp. 6840-685 (pages 1, 18).
[24] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. "Cascaded Diffusion Models for High Fidelity Image Generation." " In: Mach. Learn. Res. 23.47 (2022), pp. 1-33 (page 18).
[25] "Huggingface Brain Tumor MRI Dataset". In: (2020). URL: https://huggiiggaace. co/ datasets/ /mmmmmmlffffftt adf BBrain-annn- Marrin-MRI--A- Images-foor for--rair--------- Tumor- Diooorrittttionn/ (page 9). [26] Niels Bruun Ipsen, Plerre-Alexandre Mattei, and Jes Frellsen. "not-MIWAE: Deep generative modelling with missing not at random data". In: arXiv preprint arXiv:2006. 12871 (2020)
(page 2).
[27] Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Chiyuan Zhang. "Measuring Forgetting of Memorized Training Examples". In: wwxii:2007.0009/ (June 2022). DOI: 10. 48550/ arxiv.. 2207 00099. arXiv: 2207.00099 [cs]. URL: http: //arxiv. org/ aataaabs/22002 22077 .00099 (visited on 11/10/2022) (pages 2, 8).
[28] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. "Robust compressed sensing mri with deep generative priors". In: Advances in Neural Informa- tion Processing Systems 34 (2021), pp. 14938-14954 (page 1).
[29] Zahra Kadkhodaie and Eero P Simoncelli. "Solving linear inverse problems using the prior implicit in a denoiser". In: arXiv preprint arXiv:2007. (2020) (page 7).
[30] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. "Elucidating the design space of diffusion- based generative models". In: arXiv preprint arXiv:2206.0000 000600 (2022) (pages 1, 7, 8, 17-19).
[31] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. "Alias-free generative adversarial networks". In: Advances in Neural Information Processing Systems 34 (2021), pp. 852-863 (page 17).
[32] Tero Karras, Samuli Laine, and Timo Aila. "A style-based generator architecture for generative adversarial networks". In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition. 2019, pp. 4401-4410 (page 17).
[33] Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. "Analyzing and improving the image quality of stylegan" In: Proceedings of the IEEE/CVF conference on computer vision and paltern recognition. 2020, pp. 8110-8119 19 (page 17).
[34] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. "Denoising Diffusion Restora- tion Models". In: Advances in Neural Information Processing Systems (pages 1, 7, 8, 22).
[35] Bahjat Kawar, Noam Elata, Tomer Michaeli, and Michael Elad. AASURE-Based Diffusion Model Training with Corrupted Data". In: arXiv preprini arXiv:2305.13: 13/8 (2023).
[36] Bahjat Kawar, Gregory Vaksman, and Michael Elad. "SNIPS: Solving noisy inverse problems stochastically". In: Advances in Neural Information Processing Systems 34 (2021), pp. 21757-
21769 (page 1).
[37] Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and I1-Chul Moon. "Soft truncation: A universal training technique of score-based diffuusion model for high precision score estimation". In: International Conference on Machine Learning. PMLR. 2022, pp. 11201
1 11228 (page 17).
[38] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. "Noise2void-leamin denoising from single noisy images". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, pp. 2129-2137 (page 2).
[39] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala,
and Timo Aila. "Noise2Noise: Learning image restoration without clean data". In: arXiv preprint arXiv:
(2018) (page 2).
[40] Steven Cheng-Xian Li, Bo Jiang, and Benjamin Marlin. "Misgan: Learning from incom- (2019)
plete data with generative adversarial networks", In: arXiv preprint (page 19).
12

---

[41] Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. "Coherent Semantic Attention for Image Inpainting". In: 2019 /EEE/CVF International Conference on Computer Vision (ICCV) (Oct. URL: http: // dx doi org/ 10 1109 ICCV.
2019). DOT: 10.
2019. 00427.
2019.00427 (page 2).
[42] Pierre-Alexandre Mattei and Jes Frellsen. "MIWAE: Deep generative modelling and imputation of incomplete data sets". In: International conference on machine learning. PMLR. 2019,
pp. 4413-4423 (page 2).
[43] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. "Pulse: Self- supervised photo upsampling via latent space exploration of generative models". In: Proceed- ings of the ieee/cvf conference on computer vision and pattern recognition. 2020, pp. 2437- 2445 (page 17).
[44] Charles Millard and Mark Chiew. "A Theoretical Framework for Self- Supervised MR Image Reconstruction Using Sub-Samplln Satb-lngning via Variable Density Noisier2Noise", In: IEEE Transac- tions on Computational Imaging 9 (2023), pp. 707-720. DOI: 10. 1111/11//TII
[45] Alex Nichol, Aditya Ramesh, Pamela Mishkin, Prafulla Dariwal, Joanne Jang, and Mark Chen. DALL-E 2 Pre- Pre-Training Mitigations. June 2022. URL: https: /pw//ppenii..... cww wwom//llog blog////ooooooo dall- e- 2- er miatiiiinniiiigiiinnnniii (visited on 1 11/10/2022) (page 1).
[46] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. "DINOv2: Learning Robust Visual Features without Supervision". In: arXiv preprint ar arXiv:2304.00199 (2023) (page 9).
[47] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. "Con- encoders: Feature learning by inpainting". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 2536-2544 (page 2).
[48] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. "Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation". In: arXiv preprint arXiv:2008. 0095] (2020) (page 2).
[49] Trevor W Richardson, Wencheng Wu, Lei Lin, Beilei Xu, and Edgar Bernal. "Meflow: Monte carlo flow models for data imputation".. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2020, pp. 14205-14214 (page 2).
[50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. "High- -High-resolution image synthesis with latent diffusion models". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. 2022, pp. 10684-10695 (page 1).
[51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. "High-resolution image synthesis with latent diffusion models". In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2022, pp. 10684-10695 (page 8).
[52] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. "Photorealistic text-to-image diffusion models with deep language understanding", In: Ad- vances in Neural Information Processing Systems 35 (2022), pp. 36479-3000 36494 (page 1).
[53] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li. Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. "Photorealistic text-to-image diffusion models with deep language understanding", In: Ad- vances in Neural Information Processing Systems 35 (2022), pp. 36479-36499 (page 21).
[54] Jascha Sohl-Dickster Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. "Deep unsu- pervised learning using nonequilibrium thermodynamics" In: International Conference on Machine Learning. PMLR. 2015, pp. 2256-2266655 2265 (page 1).
[55] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. "Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models". In: arXiv preprint arXiv:2212 22..2.2...00 238600 (2022) (pages 1, 2, 8, 9).
[56] Jiaming Song, Chenlin Meng, and Stefano Ermon. "Denoising diffusion implicit models". In: arXiv preprint arXiv:2010.0255 (2020) (pages 7, 19).
[57] Yang Song and Stefano Ermon. "Generative modeling by estimating gradients of the data
distribution". In: Advances in Newral Information Processiny Systems 32 (2019) (page 1).
13

---

[58] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma. Abhishek Kumar, Stefano Ermon, and Ben Poole. "Score-based generative modeling through stochastic differential equations". In: arXiv preprint arXiv:201 J. 13456 (2020) (pages 3, 18).
[59] Julián Tachella, Dongdong Chen, and Mike Davies. "Unsupervised Learning From Incomplete Measurements for Inverse Problems". In: arXiv preprint arXiv:2201.1215/ (2022) (page 2).
[60] Singanallur V Venkatakrishnan, Charles A Bouman. and Brendt Wohlberg. "Plug-and-play priors for model based reconstruction". In: 2013 IEEE Global Conference on Signal and Information Processing. IEEE. 2013, pp. 945-948 (page 7).
[61] Pascal Vincent. "A connection between score matching and denoising autoencoders". In: Neural computation 23.7 (2011), pp. 1661-1674 (page 4).
[62] Ge Wang. Jong Chul Ye, and Bruno De Man. "Deep learning for tomographic image recon- struction". In: Nature Machine Intelligence 2.12 (2020), pp. 737-748 (page 2).
[63] Jiahui Yu, Zhe Lin, Jimei Yang. Xiaohui Shen, Xin Lu, and Thomas Huang. "Free-Form Image Inpainting With Gated Convolution". In: 20/9 JEEE/CVE International Conference on Computer Vision (ICCV) (Oct. 2019). DOT: 10. 10.1109//11 /icev.iee.... 2019. 00457. URL: http: //dx. doi. org/oo//// 10. 1119//190OO0 2019. 00457 (page 2).
[64] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. "Free-form image inpainting with gated convolution".. In: Proceedings of the IEEF/CVF international conference computer vision. 2019, pp. 4471-4480 (pages 17).
[65] Yide Zhang, Yinhao Zhu, Evan Nichols, Qingfei Wang, Siyuan Zhang, Cody Smith, and Scott Howard. "A poisson-gausseeeeeeo In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, pp. 11710-117 11718 (page 2).
denoising dataset with real fluorescence microscopy images".
14

---

A Proofs
Proof of Theorem 4.1. Let hg- be minimizer of equation 3.2, and for brevity let
f f'A,, A) hgg (Aæ,, A) E[(z Axx, A]
be the difference between hg- and the claimed optimal solution. We will now argue that f must be identically zero.
First, by adding and subtracting AE [Ax,, A}, we can expand the objective value achieved at i = H" in equation 3.2 follows:
Jooer (f*) = 20 A,,AA
(Axro - AE( Aæx, A)) - Af Ani|i
Axx, A] |1/ + +E Bo A,
Af (Axs,, 2)1²
Ax00 AIE
20 m, A,A
2 no A.A
(Aaoo - ATE Am,, Af (f A)
Here the first term is the irreducible error, while the third term vanishes by the tower law of expecta- tions:
E , (ADo - ABEwo Ax,, A)) Af A))
E A.Aa,,
(Io - E|xo
Åxz A))² A Af (Ax,, A)]
20 - E(zo Axx, All T
E A,
E
A Af Af (Azoe, A)
= E A,
(E(zg)
A
A]
E|zo
Âx,, A\)² AT Af (Axi, A)
O.
Thus the only part of (8") that actually depends on the parameter value H* the second term. We now show that the second term be made to vanish, and that this occurs precisely when f is identically 0:
EAAAA ||| | A)
E -ooo (S(Ao A) A Aʃ Af(Ar, Ai]
E -x
((AR1,)) A) æ a..A [AT A] ff fAxc, A)}
E =Ego. [[(Aa, A AIA [AT A) of fAr,, A))
For every À and Fo. Tt, by assumption we have that IE AJā [A is full-rank, and so the inner quadratic form is minimized when fAr, Á) Further, the term as whole vanishes exactly when this holds for every À and 2o, 3t the support, which means f must identically zero. D
Corollary A.1 (Inpainting noise model). Consider the following inpainting noise model: A Raxx is a diagonal matrix where each entry Aii ~ Ber(( p) for some p > 0 (independently for each i), and the additional noise is generated by drawing A such that Ag = Avs Ber(1 8) for some small 8 > 0 (again independently for each i). Then the unique minimizer of the objective equation 3.2 is
ho- (Ax,, A) 16
Ax,, A).
Proof. By Theorem 4.1, what we must show is that for any À in the support, E EAA [AT A] is full-rank. Fix any particular realization of Å, which will be diagonal matrix with only Os and 1s. For indices where Ass 1, we know that for any A drawn conditional on Á. Au well, i.e.
Pr(Aa
Au =
15

---

And for indices i where A. 0, by Bayes' rule we have
Pr(Ag L, Au 0)
(1 p)d
Pr(An; = 1 Au = 0) Pr(A,; 0, A, 0) Pr(A; A, 0) (1 p)6 +p
= q.
Thus we see that [ = E [ is a diagonal matrix whose entries are wherever A. and wherever Au = 0. This clearly of full rank since > O.
Corollary A.2 (Gaussian measurements noise model). Consider the following noise model where we only observe m independent Gaussian measurements of the ground truth, and then one of the measurements is further omitted: A m consists of m rows drawn independently from (O, In). and Å mhxx constructed condtional on A by zeroing out its last row. Then the unique minimizer of the objective in equation 3.2 is
hg- (Axi, A) E xc
AZ A).
Proof. Again by Theorem 4.1, we must show that for any À the support, AAji(ATA) is full-rank.
Fix any realization of Å, which will have the following form:
0
Then it is clear that conditional on A, A has the following distribution:
af
A A
where bn (O
b
Here bm drawn entirely independently from (0 In). Elementary manipulations now reveal that E EAlaAA" A] = Eb..,~N(o.. ".")
[,A"A' Á + b,, A" A I,, which is clearly full rank (indeed, is
PSD with strictly positive eigenvalues).
A.1 Reduction
In this section we argue that if there is an algorithm that recovers the target distribution (a from i.i.d. samples (Aæo, A) where A p( and an po(())) then, there is algorithm that recovers Po(2o) without sample access, but instead, using access to an oracle that given f, æ and A in the support of p(A), returns E[xx, Axx)]-
Indeed, for any A, Chen et al. [10] show that is possible to recover the distribution Awo given access to A[ Aw;] for any L and æ under some minimal assumptions the data distribution Po(zo). see [10, Assumptions 1-3]. Using our oracle and using this theorem, we can recover the well obtain samples Awo for ~ p(A) and wo ~ Po (2o). Hence, if these samples are sufficient to recover Po (x)), then, having oracle to these conditional expectations is sufficient well.
distribution of Az for all A in the support. By sampling from these distributions,
This intuition can be formalized as follows. Fix distribution (A) over corruption matrices. For distribution Pa (wa), denote by corrupt(pA, Po) the distribution over pairs (A, Aza) where ~ (A) and Vo ~ po(as) (x)). We say that is possible to reconstruct po(#o) from random corruptions A PA (A) if the following holds: for any two distributions, po (xx) and p6 (x(,) that satisfy Assumptions Similarly, we say that it is possible to reconstruct Pu (u( from conditional expectations given A ~ p(A) the following holds: for any distribution Po( and pb(z/) that satisfy Assumptions 1-3 of Chen al. [10], if for all x, and A in the support of PA
of Chen et al. [10], corrupt(pA, = corrupt(par
then po = po-
((pp..e))==p..t00...)) Art 27] Er oi) (0%, -
Any = x))
(A.1)
16

---

then Po = Pb- Here, Po, xx) is obtained by sampling to Po and = 2o + dz where n -N(0,)) I). Similarly, po. (2/). 2/) is obtained by the same process where z/ instead sampled
from ph- We state the following lemma:
Lemma A.3. Fix distribution PA (A). if possible to reconstruct po(aro) from random corruptions Yo Ar A ~ (A), then it is possible to reconstruct po (To) given access to an oracle that computes the conditional expectations B| Axt, for A ~ PA and x, 2o + o.
Proof. Assume that it is possible to reconstruct po(2r) from random corruptions ~ PA PA(A) and we will prove that it is possible to reconstruct Po(xo) from conditional expectations given A ~ P.A (A). To do so, let po (x)) and (2(2)) be two distributions that satisfy Assumptions 1-3 of Chen et al. [10]. Assume that equation A. holds. We will prove that po pg. Fix some A in the support of PA- By Chen et al. [10], there is an algorithm that samples from the distribution of Axg, Tn ~ po that only has access B[Aar A A] and similarly, there is an algorithm that samples from the distribution of Axd that only has access to E[Aao Aa,,, expectations are assumed to be the same, then the distribution of Azo equals the distribution of Axty Consequently, (p reconstruct (2oo) from random corruptions A Pa(A), this implies that po = p6- This completes the proof.
Since these two conditional
corrupt (PA, A,))) ) By the assumption that it is possible to
B Broader Impact and Risks
Generative models in general hold the potential to have far-reaching impacts on society in a variety of forms, coupled with several associated risks [43, 32, 33, 31J. Among other potential applications, they be utilized create deceptive images and perpetuate societal biases. To the best of our knowledge, our paper does not amplify any of these existing risks, Regarding the included MRI results, we want to clarify that we make no claim that such results are diagnostically useful. This experiment serves only as toy demonstration that it can be potentially feasible to learn the distribution of MRI scans with corrupted samples. Significant further research must be done in collaboration with radiologists before our algorithm gets tested in clinical trials. Finally, we want to underline again that even though our approach seems to mitigate the memorization issue in generative models, we cannot guarantee the privacy of any training sample unless we make assumptions about the data distribution. Hence, we strongly discourage using this algorithm in applications where privacy is important before this research topic is investigated further.
C Training Details
We open-source our code and models
facilitate further
research in this area:
For all the experiments in this paper, concatenated the image with the mask along the channel dimension to condition the network on the measurement operator, i.e. the inpainting mask is an additional input channel.
Models trained from scratch. We trained models from seratch at different corruption levels on CelebA-HQ, AFHQ and CIFAR-10. The resolution of the first two datasets set 64 x 64 and for CIFAR-I0 we trained on 32 x 32.
We started from EDM's [30] official implementation and made some necessary changes. Architec- turally, the only change we made was to replace the convolutional layers with Gated Convolutions [64] that are known perform well for inpainting problems. We observed that this change stabilized the training significantly, especially in the high-corruptions regime. As in EDM, we use the architecture from the DDPM++ [37] paper.
To avoid additional design complexity, we tried to keep our hyperparameters as close as possible to the EDM paper. We observed that for high corruption levels, it was useful to add gradient clipping, otherwise, the training would often diverge. For all our experiments, we use gradient clipping with max-norm set to 1.0. We underline that unfortunately, even with gradient clipping, the training at high corruption levels (p 2 0.8), still diverges sometimes. Whenever this happened, we restarted the training from an earlier checkpoint. We list the rest of the hyperparameters used in Table 2.
17

---

Table 2: Training Hyperparameters
Dataset
Iters
Batch
I.R
SDE
Aug. Proh.
Ch. Multipliers
Dropout
CIFAR-10
512
le-3
(0.2.0.4..6.00.))
0.12
1,
0.13
AFHQ 200000
00.200.4.0.6.000
0.93
0.1
0.25
VP
256
2e-4
0..2.0.6,00 0.87
0.15
(1,
CIQ
0.9
0.3
Training diffusion models from scratch is quite computationally intensive. We trained all our models for 200000 iterations. Our CIFAR-I0 models required x 2 days of training each on A100 GPUs. Our AFHQ and CelebA-HQ models required = 6 days of training each on 6 A100 GPUs. These numbers roughly match the performance reported in the EDM paper, indicating that the extra corruption we need to do on matrix A does not increase training time.
Due to the increased computational complexity training these models, we could not extensively optimize the hyperparameters e.g. the ó probability in our extra corruption. For higher corruption, e.g. for p 0.9, we noticed that we had increase ö order for the model learn to perform well on the unobserved pixels. For small-scale ablation study on CIFAR-10, see Table 7.
Finetuning Deepfloyd's IF. We access Deepfloyd's 's IF [2] model through the diffusers library. The model is Cascaded Diffusion Model [24]. The first part of the pipeline is a text-conditionll diffusion model that outputs images at resolution 64 x 64. Next in the pipeline, there are two diffusion models that are conditioned both in the input text and the low-resolution output of the previous stage. The first upscaling module increases the resolution from 64 x 64 to 256 x 256 and the final one from 256 x 256 to 1024 1024.
To reduce the computational requirements the finetuning, we only finetune the first text-conditional diffusion model that works with 64 64 resolution images. Once the finetuning is completed, we use again the whole model to generate high-resolution images.
For the finetuning, we set the training batch size to 32 and the learning rate to 3e 6. We train for maximum of 15000 steps and we keep the checkpoint that gives the lowest error on the pixels that we further corrupted. To further reduce the computational requirements, we use an 8-bit Adam Optimizer and we train with half- -alf-precision.
For our CelebA finetuning experiments, we set 8 = 0.1 and 0.8. We experiment with the full training set, subset of size 3000 (see Figure 1) and subset of 300. For the model trained with only 300 heavily corrupted samples, we did not observe memorization but the samples were of very low quality. Intuitively, our algorithm provides way control the trade-off between memorization and fidelity. Fully exploring this trade-off is a very promising direction for future work. For our MRI experiments, we use two con-overlapping blocks that each obfuscates 25% of the image and we evaluate the model in one of them.
All of our fine-tuning experiments be completed in few hours. Training for 15000 iterations takes R 10 hours on an A100 GPU, but we usually get the best checkpoints earlier the training.
D Evaluation Details
FID evaluation. Our FID [22] score is computed with respect to the training set, as is standard practice, e.g. see [23, 30, 58]. For each of our models trained from scratch, generate 50000 images using the seeds 0 - 49999. Once we generate our images, we use the code provided in the official implementation of the EDM [30] paper for the FID computation.
E
Additional Experiments
E.1
Restoration performance with noisy measurements
In Table 1 of the main paper, we compare the restoration performance of our models and vanilla diffusion models (trained with uncorrupted images). We compare the restoration performance the task random inpainting because it is straightforward to models solve this inverse problem.
18

---

It is potentially feasible to use our trained generative models to solve any (linear or non-linear) inverse problem but we leave this direction for future work.
In this section, we further compare our models (trained with randomly inpainted images) in the task of inpainting with noisy measurements. Concretely,
want predict æo. given measurements y/: = A (z + y 77, 7 10, As in the rest of the paper, we assume that the mask matrix is known. We can solve this problem with one model prediction using our trained models since yr.
according to Eq. 4.1, we are learning: ha (Yr, A, t)
We the EDM [30] state-of-the- model trained AFHQ paper). To use this pre-trained model to solve the noisy random inpainting inverse problem, we need a reconstruction algorithm. We experiment again with DPS and DDRM which can both handle inverse problems with noise in the measurements. We present our results in Table As shown, our models significantly outperform the EDM models that the DDRM reconstruction algorithm and perform on-par with the EDM models that use the DPS reconstruction algorithm.
baseline (as did in the main
Dataset
Corruption Probability
Measurement Noise yo
Method
LPIPS
PSNR
NFE
AFHQ
Ours DPS
0.0861
29.46
0.0846
29.83
100
0.4
0.05
0,2061
24.47
DDRM
0.1739
25.38
99
0.1677
25.58
199
Ours
0.1031
27.40
DPS
0.0949
27.92
100
0.6
0.05
0.4066
18.73
35
DDRM
0.3626
19.49
99
0.3506
19.70
199
Ours
0.1792
23.21
DPS
0.1778
23.01
100
0.8
0.05
0,5879
13.65
35
DDRM
0,5802
13.99
99
0.5753
14.09
199
Table 3: Comparison of our model (trained on corrupted data) with state-of-the-art diffusion models on CelehA (DDIM [56] model) and AFHQ (EDM [30] model) for solving the random inpainting inverse problem with noise
E.2 Additional Comparisons
We incude additional comparisons CIFAR-10 and CelebA with MisGAN [40]. The MisGAN work generalizes AmbientGAN in the case where measurement operator is unknown. Specifically, the authors propose an additional generator that learns to model the corruption mechanism with adversarial training. The results are shown in Table
E.3
Comparison with Supervised Methods
For completeness, we include a comparison with Masked AutoEncoders [21], a rate-of-the-ar supervised method for solving the random inpainting problem. The official repository of this paper does not include models trained on AFHQ. We compare with the available models that are trained on the iNaturalist dataset which is the most semantically close dataset could find. We emphasize that this model was trained with access to uncorrupted images. Results are shown in 5. As shown, our method and DPS outperform this supervised baseline. We underline that this experiment is included for completeness and does not exclude the possibility that there are more performant supervised alternatives for random inpainting.
E.4 Sampler A Ablation
By Lemma A.3, if it is possible to learn po (aro) using corrupted samples then it is also possible to use our learned model to sample from Po(mo). Even though such algorithm exists, we do not know which one it is.
19

---

CelebA
Corruption Probability
Method
FID
0.6
MisGAN
37.42
0.6
Ambient Diffusion
6.08
0.8
MisGAN
100.0
0.8
Ambient Diffusion
11.19
0.9
MisGAN
141.11
0.9
Ambient Diffusion
25.53
CIFAR-10
Corruption Probability
Method
FID
0.4
MisGAN
18.95
0.4
Ambient Diffusion
18.85
0.6
MisGAN
49.30
0.6
Ambient Diffusion
28.88
0.8
MisGAN
111.50
0.8
Ambient Diffusion
46.27
Table 4: Comparison with MisGAN.
Corruption Probability
Method
LPIPS
PSNR
NFE
Ours
0.0304
33.27
0.4
DPS
0.0203
34.06
100
MAE
0.0752
28.88
Ours
0.0628
29.46
0.6
DPS
0.0518
30.03
100
MAE
0.0995
25.89
Ours
0.1245
25.37
0.8
DPS
0.1078
25.30
100
MAE
0.1794
22.01
Table 5: Comparison with MAE [21], state-of-the-art supervised method for solving restoration task of random inpainting.
In the paper, we proposed simple idea for sampling, the Fixed Mask Sampler. The Fixed Mask Sampler fixes mask throughout the sampling process. Hence, sampling with this algorithm is equivalent to first sampling some pixels from the marginals (o( and then completing the rest of the pixels with the best reconstruction (the conditional expectation) in the last step. This simple sampler performs remarkably well and we use throughout the main paper.
Sampling with Reconstruction Guidance. In the Fixed Mask Sampler, at any point the prediction convex combination of the current value and the predicted denoised image. As f - 0, % - Hence, for the masked pixels, the fixed mask sampler outputs the conditional expectation of their value given the observed pixels. This leads averaging effects the corruption gets higher. To correct for this problem, we add one more term in the update: the Reconstruction Guidance term. The issue with the previous sampler is that the model never sees certain pixels. We would like to evaluate the model using different masks. However, the model outputs for the denoised image might be very different when evaluated with different masks. To account for this problem, we add an additional term that enforces updates that lead to consistency on the reconstructed image. The update of the sampler with Reconstruction Guidance becomes:
Ct-At
Ft +
Øt Oe At
Eizo Aw,, A)
wi V.. BA
[|alla| Aw,, A] E[zoo A' 11]
T
(E.1)
20

---

This sampler is inspired by the Reconstruction Guidance term used in Imagen [53]. In Table 6 we ablate the performance of this alternative sampler. For the reconstruction guidance sampler, we select each time four masks at random and we add an extra update term to the Fixed Mask Sampler that ensures that the predictions of the Fixed Mask Sampler are not very different compared the predictions with the other four masks (that have different context regarding the current iterate zz2). We set the guidance parameter Wy to the value 5e As shown, this sampler improves the performance, especially for the low corruption probabilities where the extra masks give significant information about the current state to the predictions given only one fixed mask. However, the benefits this sampler are vanishing for higher corruption. Given that the two samples perform par and that the Reconstruction Guidance Sampler is much more computationally intensive (we need one extra prediction per step for each extra mask), choose to the Fixed Mask Sampler for all the experiments in the paper.
Sampler Type
Corruption Probability
FID
Inception Score
0.2
11.70
7.97
0.4
18.85
7.45
Fixed Mask
0.6
28.88
6.88
0.8
46.27
6.14
0.2
11.59
8.01
18.52
7.51
Reconstruction Guidance
0.6
28.90
6.91
0.8
46.31
6.13
Table 6: Comparison between the Fixed Mask sampler and the Reconstruction Guidance Sampler.
E.5 8 Ablation
hyperparameter of method is 6, which controls the strength of the additional corruption. For the random inpainting case, needs to be positive in order to learn to predict the clean image for the missing pixels. On the other hand, (which finally controls generation performance). In Table we show how performance changes on CIFAR-10 for different values of The recommended setting is 0.1 but hyperparameter search for the value of 8. might lead to performance improvements.
increases, so does the difficulty of the prediction task
Inception Score
0.4
0.0
6.70 (from Figure 5)
0.4
0.1
7.45 (from Figure 6)
0.4
0.4
6.95 (experiment added for the rebuttal)
Table 7: Ablation Study for the value of
E.6 Additional Figures
Figure shows reconstructions of AFHQ corrupted images with the EDM AFHQ model trained on clean data (columns 3, 4) and our model trained on corrupted data (column 5). The restoration task random inpainting at probability p = 0.8. The last two rows also have measurement noise with yoo 0.05.
In Figure 8, we repeat the experiment of Figure 6 of the main paper but for models trained on the full CelebA dataset and at different levels of corruption. As shown, increasing the corruption level leads clear shift of the distribution the left, indicating less memorization. This comes at the cost of decreased performance, as reported in Table 4.
21

---

Input
Ground Truth
DDRM [34]
DPS [11]
Ours
Figure 7: Reconstructions of AFHQ corrupted images with the EDM AFHQ model trained on clean data (columns 3, 4) and our model trained on corrupted data (column 5). The restoration task is random inpainting at probability = 0.6. The last two rows also have measurement noise with yy 0.05.
Distribution of Similarity Values to the nearest training sample
Corruption probability 0.0
Corruption probability
Corruption probability 0.4
Number of generated samples
400
Corruption probability 0.6
Corruption probability
300
200
100
0.60
0.65
0.70
0.75
0.85
0.90
0.95
1.00
Similarity Value
Figure 8: Distribution of similarity values the nearest neighbor in the dataset for finetuned IF models on the full CelebA dataset at different levels of corruption. Please note that similarity values above 0.95 roughly correspond the same person, while similarities below 0.75 correspond to almost random faces. As shown, increasing the corruption level leads to clear shift of the distribution to the left, indicating less memorization.
22

---

In the remaining pages, we include uncurated unconditional generations of our models trained at different corruption levels p. Results are shown Figures 9,10,11.
CelebA dataset, p = 0.6, ò 0.1
Uncurated samples from our model
=
CelebA dataset, p 0.8, 0.1
Uncurated samples from model
CelebA dataset, p = 0.9, 6 0.3
Uncurated samples from our model
Figure 9: Left column: CelebA-HQ training dataset with random inpainting at different levels of corruption p (the survival probability is - p) (1 - -)). Right column: Unconditional generations from our models trained with the corresponding parameters. As shown, the generations become slightly worse as we increase the level of corruption, but we can reasonably well learn the distribution even with 93% pixels missing (on average) from each training image.
23

---

AFHQ dataset, p 0.4, = 0.1
Uncurated samples from model
AFHQ dataset, = 0.6, 5 = 0.1
Uncurated samples from our model
AFHQ dataset, 0.8, = 0.1
Uncurated samples from model
24

---

AFHQ dataset, p 0.9, 8 0.1
Uncurated samples from our model
Figure 10: Left column: AFHQ training dataset with random inpainting at different levels of corruption p (the survival probability is (1-p) from our models trained with the corresponding parameters. As shown, the generations become slightly worse even with 91 % pixels missing (on average) from each training image.
p) (1 - 5)). Right column: Unconditional generations
increase the level of corruption, but
reasonably well learn the distribution
25

---

CIFAR-10 training dataset, Uncurated samples from p = 0.2,6 0.1
model
CIFAR-10 training dataset, Uncurated samples from our p 0.4, 6 0.1
model
CIFAR-10 training dataset, Uncurated samples from our
p 0.6, 8 0.1
model
CIFAR-11 10 training dataset, Uncurated samples from
p = 0.8, = 0.1 model
Figure 11: Left column: CIFAR-10 training dataset with random inpainting at different levels of corrptiion p (the survival probability is (1-p) (1 (1-8)), 8)). Right column: Unconditional generations from our models trained with the corresponding parameters. As shown, the generations become slightly worse as we increase the level of corruption, but we can reasonably well leam the distribution even with high corruption.
26


Reviews:
[{"review_id": "MSRtNN40oq", "reviewer": "Reviewer_GQa2", "summary": "The paper focuses on learning clean distributions from corrupted data. In the training diffusion model, the training dataset contains only highly-corrupted examples. They propose a training algorithm of restoration model by introducing additional measurement distortion.They also provide sampling methods and theoretical analysis. Experimental results show that their superior performance.", "questions": "Please see Weaknesses part.", "limitations": "They provided in the last paragraph in the main paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The problem of handling corrupted datasets is significant even in generative model learning. This paper could be seen as one that explores this directions.\n* The paper is well-structured and easy to follow.\n* The fact that a model which is trained by corrupted data does not memorize the training dataset, as supported by Figure 1 and Figure 4, is very important. This might be gives significant implications for various applications.", "weaknesses": "* Sampling Process\n  * It would be helpful to provide an algorithm or a detailed explanation of the sampling process from scratch.\n  * Equations 3.3 and 3.4 describe how the sample at time $t$ is generated from sample at $t-\\Delta t$. It is necessary to explain how the evaluation of single networks allows the generation of images in the experiments.\n\n* Related work\n  * It would be beneficial to include a discussion of previous research on dealing with incomplete datasets in generative models, such as [1, 2, 3, 4].\n\n[1] Li, S. C. X., Jiang, B., & Marlin, B. (2018, September). MisGAN: Learning from Incomplete Data with Generative Adversarial Networks. In International Conference on Learning Representations.\n\n[2] Mattei, P. A., & Frellsen, J. (2019, May). MIWAE: Deep generative modelling and imputation of incomplete data sets. In International Conference on Machine Learning (pp. 4413-4423). PMLR.\n\n[3] Ipsen, N. B., Mattei, P. A., & Frellsen, J. (2020, September). not-MIWAE: Deep Generative Modelling with Missing not at Random Data. In International Conference on Learning Representations.\n\n[4] Richardson, T. W., Wu, W., Lin, L., Xu, B., & Bernal, E. A. (2020). Mcflow: Monte carlo flow models for data imputation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14205-14214).\n\n* Experiments\n  * It would be beneficial to provide the FID results for Figure 5 to demonstrate the differences.\n  * In addition to Figure 5, it would be valuable to compare the proposed model with existing generative models such as AmbientGAN in various experiments.\n\n* Presentation\n  * It would be helpful to have better spacing between subfigures in Figure 2 to improve the clarity of the captions.\n  * Proper citations are needed in the background section.\n  * Figure 3 would be better placed within a paragraph rather than between paragraphs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the constructive feedback! We are glad that the Reviewer appreciated the novelty, the presentation and the implications our work could have in various applications related to memorization.\n\n> It would be helpful to provide an algorithm or a detailed explanation of the sampling process from scratch.\n\nWe agree with the Reviewer. Please see the detailed sampling algorithm in the one page PDF accompanying this rebuttal. We will include this algorithm in the next revision of our work.\n\n> it is necessary to explain how the evaluation of single networks allows the generation of images in the experiments.\n\nThis becomes clear with the sampling algorithm we include in our rebuttal. This works by a process of iterative restoration-degradation, similar to how all diffusion models operate. \n\n> It would be beneficial to include a discussion of previous research on dealing with incomplete datasets in generative models, such as [1, 2, 3, 4].\n\nWe thank the Reviewer for bringing this relevant work to our attention! We will definitely add these references in the camera-ready version of our work. All these works propose ways to learn other classes of generative models (such as GANs, Normalizing Flows and VAEs) from missing data. The MisGAN work generalizes AmbientGAN in the case where the measurement operator is unknown. Specifically, the authors propose an additional generator that learns to model the corruption mechanism with adversarial training. MCFlow is a framework, based on a variant of the EM algorithm, that can be used to train normalizing flow models from missing data. Finally, MIWAE and Not-MIWAE are frameworks to learn deep latent models (e.g. VAEs) from missing data when the corruption process is known or unknown respectively. Our work provides a diffusion-based framework for the missing data problem and thus expands this interesting prior work.\n\n\n> It would be beneficial to provide the FID results for Figure 5 to demonstrate the differences.\n\nFigure 5 shows the Inception results for CIFAR-10. The FID results for the same dataset are shown in Figure 6. Unfortunately, the authors of AmbientGAN do not report FID scores.\n\n> In addition to Figure 5, it would be valuable to compare the proposed model with existing generative models such as AmbientGAN in various experiments.\n\n\nWe thank the Reviewer for the suggestion. AmbientGAN only provides quantitative results in MNIST and CIFAR-10 (Figures 7 and 8 in the AmbientGAN paper). We have the comparison with CIFAR-10 (Figure 5) in the paper. We did not experiment on the MNIST dataset since it is a rather toy problem for image generation.\nThe Reviewer brought to our attention the follow-up work to the AmbientGAN paper, MisGAN. The authors of MisGAN report FID scores for different erasure probabilities for CIFAR-10 and CelebA. We will include these comparisons in the next version of our paper. A short comparison is provided below:\n\n\n**CelebA**:\nCorruption Probability | Method | FID |\n--------------------------| --------- | ---- |\n0.6                               |    MisGAN      |  37.42 \n0.6                                    | Ambient Diffusion | **6.08**\n0.8                              |    MisGAN      |  100.0 \n0.8                                    | Ambient Diffusion | **11.19**\n0.9                               |    MisGAN      |  141.11 \n0.9                                   | Ambient Diffusion | **25.53**\n\n\n**CIFAR-10**:\nCorruption Probability | Method | FID |\n--------------------------| --------- | ---- |\n0.4                               |    MisGAN      |  18.95 \n0.4                                    | Ambient Diffusion | **18.85**\n0.6                              |    MisGAN      | 49.30 \n0.6                                    | Ambient Diffusion | **28.88**\n0.8                               |    MisGAN      |  111.50 \n0.8                                   | Ambient Diffusion | **46.27**\n\nWe will include this comparison in the camera-ready version of our work. It can also be found in the one page PDF accompanying this rebuttal. We emphasize that MIsGAN is solving a harder problem than we are, since for MisGAN the corruption operator is not known and needs to be inferred-- we will explain this in the paper. \n\nWe finally want to thank the reviewer for the comments in the presentation of our work. We will make sure to add the additional citations, fix the spacing and improve the placement of Figure 3, as suggested.\n"}, {"review_id": "NRczaPU0UB", "reviewer": "Reviewer_qLWX", "summary": "This paper describes a method to learn a denoising diffusion model only with corrupted data. This is an important problem in many areas of  applied science where there is no access to ground truth. Another important potential benefit of this method is to overcome memorization of the training images. The main idea in this method is described as introducing additional corruption and training on this doubly corrupt data. The authors describe a derivation to estimate conditional expectation of the uncorrupted image only using the corrupted data. They show (Table 1) that their method outperforms other methods in solving random inpainting problem. Also, the method is used to fine tune Deepfloyd IF on smaller samples. This result is used to show that the method overcomes memorization.", "questions": "In addition to theoretical questions and comments I made under weaknesses: \n\nWhy do you think you achieve optimal sampling performance in one iteration? \nHow does the training time compares to uncorrupted models? Do you need to train on more samples? \n________________________________________________________\nNote: I am willing to raise the score if the questions asked under Weaknesses and here are addressed.", "limitations": "The main limitation of the work (aside from some jumps  and ambiguity in the theoretical results) is the type of corruption this method can work with. The corruption must be linear and in addition $E_{A|\\tilde{A}}[A^TA]$ must be full rank. This rules out many real world applications.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This work is motivated in two ways: learning the score of distribution of clean data with access to corrupted data, and avoiding memorization. These are both very important practical and theoretical topics which this work tries to address. The trick which is used to train the models (adding more corruption) is clever and the network achieves to learn to inpaint and denoise images (figure 7). As mentioned under weaknesses, I think the first goal is not fully achieved and the theoretical results seem incorrect and practical results seem to be constrained to specific types of corruption. However, the memorization results seem very impressive, although the memorization analysis is carries only for the fine tuned model.", "weaknesses": "There are a number of fundamental technical and conceptual flaws which need to be considered and fixed: \n\n1) The training setup was not clear in the text, but this is what I gathered: the target image during training is a corrupted image $Ax_0$, and the input is a noisy and more corrupted image $BAx_0 + \\sigma \\eta $. All throughout the corruptions are assumed to be random or block missing pixels. The network learns to remove the noise and inpaint pixels that are removed by B. At the test time, the network removes noise and also inpaint *all* pixels since it doesn't know which pixels are dropped due to A and which pixels are dropped due to B.\nIf this is the training setup, please clarify in the text. If not, please describe what was the training setup. \n\n2) Equation (3), the objective after additional corruption, is the distance between the doubly corrupted image and the $\\textbf{clean}, x_0,$ image. The entire premise of the work is that clean image is not available so why is it assumed to be available during training? The objective should be the distance from $Ax_0$ instead. It needs to be clarified whether this is a typo or the authors actually used clean data, $x_0$, during training. \n\n2) If that is a typo and the target image during training is $Ax_0$ then after training is completed the network works as a denoiser plus inpainter. As a result, the Tweedie equation is not a good description of the output of this network anymore. That is, the output is not $E[x_0|x_t]$, where $x_t = x_0 + \\sigma_t \\eta$. So, this output cannot be used directly as an estimate of score in the diffusion model .\n\n3) To remedy the above mentioned problem, the authors propose eq 3.3 in which they claim to approximate the score, $E[x_0|x_t],$ with $E[x_0|\\tilde{A}x_t, \\tilde{A}]$. It is not at all clear what is the justification behind this approximation. Again, this expectation is a very different entity from the actual score, and it is not clear why direct use of it makes sense to estimate the score. This expectation is the solution for inverse problem given the forward measurement $A$, as apposed to the solution for mere denoising problem (i.e. the score). \n\n4) Additionally, two more terms are added to the update line in eq 3.4. The justification for this is described from line 166 to 175. It starts with $\\gamma_t$ going to zero when $t$ approaches zero. It is not clear as to why $\\gamma_t$ goes to zero when $t$ goes to zero. Please clarify this assumption. The reasoning that follows this assumption is also not clear and sounds ad hoc. Did you add these terms because the update line of eq 3.3 did not work in practice? What is the intuition or theory behind this choice? Please clarify both eq 3.3. (why did you estimate one expectation with another?) and eq 3.4 (why did you add two terms and why do they make sense?). \n\n5) The theory section needs at least a re-write because it is not clear what is the goal of this section. The section starts with the goal of proving that the optimal estimate of clean image given the corrupted image is equal to $E[x_0| Ax_t = y, A]$. It is not clear why the authors need to prove this, since this is a basic fact from Bayesian machine learning: the optimal estimation of corrupted data is the conditional mean of the predictive distribution (refer to textbooks like Bishop). The problem is that this expectation is not equal to the score, so theoretically it can't be used to estimate the score. Of course what is learned by this network is approximately the optimal reconstruction of $A(x+\\sigma \\eta)$, but as long as it is not the score, it is not theoretically valid to use it iteratively in a diffusion model. In a nutshell, the learned network is a denoiser/inpainter not a score estimator (i.e. pure denoiser). \n\n6) On top of the above mentioned issues, this network learns to reconstruct + denoise images only if $E_{A|\\tilde{A}}[A^TA]$ is full rank. This strong assumption requires a high level of randomness in the corruption which is not very common in many real-world applications. For example, if the ground truth images miss some information systematically (let's say they are all blurred) the network will not be able to reconstruct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their feedback!\n\n> Equation (3), [...] is the distance between the doubly corrupted image and the clean image. The objective should be the distance from $Ax_0$.  It needs to be clarified whether this is a typo or the authors actually used clean data.\n\nThe Reviewer has misread Equation 3. *There is a parenthesis outside*, A multiplies both the output of the network and the clean image. Hence, the only thing needed to train is the corrupted image Ax and not the clean image itself. **We do not use clean data during training.**\n\n> The theory section needs at least a re-write because it is not clear what is the goal of this section. [...] It is not clear why the authors need to prove this, since this is a basic fact from Bayesian machine learning [...].\n\nThe Reviewer may have a misunderstanding regarding our theoretical results, potentially because of misreading Equation 3. \nProving that the minimizer of  $\\mathbb E_{x_0,Ax_t}[ || f(Ax_t) - x_0||^2]$  is the conditional expectation is a textbook argument. However, this is *not* what we prove. We prove something much stronger which is that there is an objective function (Eq. 3.2) that doesn’t need access to the clean images $x_0$ and still has as a minimizer the conditional expectation (Theorem 4.1). Our proof does build on the standard techniques but it has the key benefit of making clear what condition we need on the distribution of $A, \\tilde{A}$ for the theorem to hold. Thus we argue that our result is novel, and we think that the Reviewer may not have appreciated it fully. We kindly ask the Reviewer to reconsider their evaluation, keeping in mind that our loss does not require clean images and that the proof technique has some differences compared to the standard setting. We would be happy to take further feedback if there are additional concerns.\n\n\n> If this is the training setup, please clarify in the text. If not, please describe what was the training setup.\n\nThe Reviewer’s understanding of the training setup is correct. We will clarify this further in the main text.\n\n> [...] This expectation is a very different entity from the actual score, and it is not clear why direct use of it makes sense to estimate the score.\n\nIt is true that we are making an approximation there, as we acknowledge in the paper. We approximate $\\mathbb E[x_0 | x_t]$ with $\\mathbb E[x0 | Axt, A]$. Intuitively, if A is not dropping a lot of information, these two conditional expectations are close. This approximation becomes worse for high corruption. This is a limitation of this work. In Lemma A.3. we prove that access to $\\mathbb E[x_0 | Ax_t, A]$ is enough to reconstruct the true distribution whenever it is possible to reconstruct it from measurements. However, for the time being, we do not have an efficient algorithm to do so and thus we resort to this approximation.\n\n> two more terms are added to the update line in eq 3.4. [...] It is not clear as to why $\\gamma_t$ goes to zero when t goes to zero. [...] Did you add these terms because the update line of eq 3.3 did not work in practice? Please clarify both eq 3.3. and eq 3.4.\n\nWe think that the Reviewer has a misconception here. The $\\gamma_t$ appears already from Equation 3.3 and it is not something that we added, it is the term that appears in all diffusion models when you discretize the reverse SDE/ODE. It always goes to zero and it stems from the fact that the magnitude of the noise is an increasing function of time and at the limit $t\\ to 0$ we do not have any noise. The only thing difference to the classic ODE discretization is that we have $\\mathbb E[x_0 | Ax_t, A]$ instead of $\\mathbb E[x_0 | x_t]$.\n\nThe Reconstruction Guidance sampler adds an additional gradient update that refines all pixels throughout the denoising process. The Fixed Mask sampler uses a fixed mask $A$ and thus the masked pixels are not getting refined at every iteration. Hence, for high corruption, these pixels get more “averaged” values. The reconstruction guidance sampler mitigates this. We ablate this Sampler in Table 5 in the Appendix and we see modest improvements that were not worth the extra computational cost hence we did not use this sampler in any experiment in the main paper. We hope this clarifies things.\n\n> This network learns to reconstruct + denoise images only if $E_{A|\\tilde A}[A^TA]$ is full-rank. This strong assumption requires a high level of randomness in the corruption which is not very common in many real-world applications.\n\nWithout this assumption, it is *impossible* to reconstruct the distribution from corrupted samples without making assumptions on the distribution to be reconstructed. This is not a limitation of our method; any method that tries to recover the distribution from corrupted samples would fail. The reason is that it is impossible to distinguish between two clean distributions that become identical after the pushfoward function. E.g., in the blurring example, depending on what is the blurring kernel, there could be many distributions that lead to the same blurred measurements.\n\n> Why do you think you achieve optimal sampling performance in one iteration?\n\nWe achieve optimal reconstruction in one step because we learn the conditional expectation which is the best reconstruction under L2 loss.\n\n> How does the training time compares to uncorrupted models? Do you need to train on more samples?\n\nWe thank the Reviewer for this great question! The training time and the dataset size are the same as for the training of the uncorrupted models. All models are trained for 200K steps (following the EDM paper) and we use the full dataset. For the finetuning experiments, we show that we can even finetune with 300 images or less. We refer to the Appendix, Section C, for the full training details.\n\nWe hope that the Reviewer's comments are addressed and the Reviewer will consider increasing their score as they noted in their review.\n"}, {"review_id": "TV4BzgXiFp", "reviewer": "Reviewer_Tq9h", "summary": "In summary, the authors propose a diffusion-based framework that can learn unknown distributions from highly-corrupted samples, allowing the training of generative models without relying on clean training data. Their approach introduces additional measurement distortion and successfully predicts original corrupted images. The method is applicable to various corruption processes and achieves promising results on benchmark datasets.", "questions": "- Can you provide the performance of the proposed method in Table 1 with more timesteps of sampling? Does the performance improve? If no, then can you please explain why?\n", "limitations": "The author lists the limitation of the proposed method in the paper which is reasonable. I appreciate the authors' adequate limitation summary and it sheds light on future improvements.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem formulation by itself is interesting, and the proposed method is novel.\n- The paper proposes a new and interesting domain of learning the image data distribution w/o access to the ground truth data samples.\n- The paper is well written and easy to follow.\n- The proposed training and sampling procedure is scalable and easy to incorporate into the current diffusion model framework.\n- The authors attach the theory for the effectiveness of the proposed method.\n- The proposed method only needs 1 NFE to produce comparable results.\n- The method paves a way to alleviate the memorize issue of diffusion model.", "weaknesses": "- From my perspective, I do not see significant weaknesses in this paper. \n- One potential issue is the significant drop of FID when the model is trained with images corrupted with a large ratio of pixels. However, I think this should not be criticized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very glad that the Reviewer appreciated the importance of the problem, the novelty, the presentation, and the theoretical and practical implications of our work!\n\n> Can you provide the performance of the proposed method in Table 1 with more timesteps of sampling? Does the performance improve? If no, then can you please explain why?\n\nWe thank the Reviewer for the great question. We achieve optimal reconstruction in one step because we learn the conditional expectation (as predicted by our theory) which is the best reconstruction under the $l_2$  loss.\n\nWe thank again the Reviewer for their positive feedback and we would be happy to take further questions, if any."}, {"review_id": "Ut0ZXSMUaB", "reviewer": "Reviewer_q1F8", "summary": "This paper proposes to train diffusion models that can recover corrupted data without training on clean data. The key idea is, given a corruption matrix $A$ one can further sample a corruption matrix $\\tilde{A}$ given $A$ and the model learns to predict all the existing pixels. It is empirically shown that this trick ensures robustness against higher corruption levels and can restore data with better performance than those that's trained on clean data.", "questions": "My questions and concerns are listed in the section above.", "limitations": "- The authors have adequately discussed limitations of the model. Some further discussion on societal impact is encouraged as it is closely related to privacy issues of training data.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors propose the first diffusion-based method that can restore data from corruption without training from clean data. \n\n- The method can reuse regular diffusion sampler without much modification. \n\n- It is guaranteed theoretically for the model to recover clean data with some rank assumptions.\n\n- The method alleviates the memorization issues of diffusion models by considering corruption schemes.", "weaknesses": "- More exposition is needed on some details. What kind of $\\tilde{A}$ is used during inference? Is it necessary to sample using the same $\\delta$ during training? Do we fix a $\\tilde{A}$ for all sampling steps or resample each step? Does taking the expectation over multiple samples of such $\\tilde{A}$ work better?\n\n- The paper lacks implementation details. E.g. $h_\\theta(\\tilde{A}, \\tilde{A}x_t, t)$ explicitly depends on $\\tilde{A}$. How is this dependency implemented in practice? \n\n- Line 248 the authors forgot line break.\n\n- The authors only considered random masking corruption. However, there are many other types of linear corruption schemes. Does the method stay robust against other corruption such as Gaussian blur, etc.?\n\n- It would be great to analyze the effect of different sampling scheme for $\\tilde{A}$. How does FID change w.r.t $\\tilde{A}$ with increasing levels of further corruption? Is there a sweet spot for further corruption?\n\n- Why is there no comparison with AmbientGAN for Table 1, as it is an important baseline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their positive and constructive feedback. We are glad that the Reviewer appreciated many aspects of our work, including the novelty of the method, the theoretical analysis, and the implications of reducing memorization. \n\n> More exposition is needed on some details. What kind of $\\tilde{A}$ is used during inference? Is it necessary to sample using the same $\\delta$ during training? Do we fix a $\\tilde{A}$ for all sampling steps or resample each step? Does taking the expectation over multiple samples of such $\\tilde{A}$ work better?\n\nWe use exactly the same process for generating $\\tilde{A}$ during inference. We first sample $A$ from the corruption distribution and $B$ with the same $\\delta$ and then set $\\tilde{A} = BA$. We cannot use a much different value of $\\delta$ during the inference because during the training, the network is trained to take input as the images with missing entries corresponding to $\\tilde{A}$.\n\nRegarding fixing or resampling the matrix A, in all of our experiments, we use a fixed $\\tilde{A}$ for all sampling steps. \nWe tried both approaches and found that the first approach works better. We also tried fixing multiple $\\tilde{A}$ for all sampling steps and using an average of $\\mathbb E[x_0 | \\tilde{A} x_t]$ as an approximation of $\\mathbb E[x_0 | x_t]$ and found that this sampler generates blurry images. We will clarify these points in the paper.\n\n> The paper lacks implementation details. E.g.  explicitly depends on $h_{\\theta}( \\tilde{A}x_t, \\tilde{A}, t )$. How is this dependency implemented in practice?\n\nWe thank the Reviewer for raising this important clarification question: We concatenate the image with the mask along the channel dimension (e.g. the mask is an extra channel). We will make this clear in the paper. \n\n> The authors only considered random masking corruption. However, there are many other types of linear corruption schemes. Does the method stay robust against other corruption such as Gaussian blur, etc.?\n\nWe have only experimented with masking (pixels and blocks), but we do not see any reason why this method wouldn't work for other corruptions, as long as the assumptions needed for our theory are satisfied. We are currently in the process of setting up some non-masking experiments and we will try to include them in the Appendix for the camera-ready version.\n\n> It would be great to analyze the effect of different sampling scheme for $\\tilde A$. How does FID change w.r.t with increasing levels of further corruption? Is there a sweet spot?\n\nWe thank the Reviewer a lot for raising this question. For a fixed corruption $p$, as we increase $\\delta$, we increase the number of missing pixels in the inputs of the network during training. Intuitively, we want to keep $\\delta$ as low as possible since we do not want to add additional corruption that is not present in our data. Since the network is trained to predict $\\mathbb E[x_0 | \\tilde A x_t, \\tilde A]$, as $\\delta$ goes to $1$, $\\tilde A$ becomes the zero matrix and the network only learns the mean of the data distribution. On the other hand, as $\\delta \\to 0$, the network is not penalized for mistakes in the pixels that are not observed and hence can make arbitrarily wrong predictions in the missing pixels. Empirically, we want to set $\\delta$ to the smallest possible value for which the network learns to predict correctly all the missing pixels.\n\nAs the Reviewer suggested, there is a sweet spot! To illustrate this, we do a small ablation for the value of $\\delta$ for random inpainting in CIFAR-10:\n\n$p$ | $\\delta$ | Inception Score\n------------------------- | ---------  | ------------------\n0.4 | 0.0 | 6.70 (from Figure 5)\n0.4 | 0.1 | 7.45 (from Figure 6)\n0.4 | 0.4 | 6.95 (experiment added for the rebuttal)\n\nAs seen, setting $\\delta$ to $0$ leads to poor performance: the network does not learn to predict the missing values. Setting a value of $\\delta$ to $0.4$ makes sure that the network learns to make accurate predictions everywhere, but now the inputs are more corrupted (only 36% of the pixels survive on average), and hence the predictions are more coarse. Remarkably, the performance of the model with hyperparameters $p=0.4, \\delta=0.4$ is only marginally better than the performance for $p=0.6, \\delta=0.4$ which has an Inception Score of $6.88$. This is because even though in the former case we have less corruption in our dataset, the effective corruption in the inputs of both models is the same, i.e. only $36%$ of the pixels survive. The optimal value of $\\delta$ depends on the dataset and more importantly the resolution – in higher resolution datasets there is more redundancy and we can afford to be more aggressive in the extra corruption. One could try to find the exact value that maximizes performance for a given $p$ (binary search), but due to the high-computational cost of the training, we choose to not ablate this further. \n\n> Why is there no comparison with AmbientGAN for Table 1, as it is an important baseline?\n\nIt would be hard, since we would need to train AmbientGANs for these datasets, plus it would be much worse than the baselines we compare against. \nTo explain why: AmbientGAN is a framework to train GANs with missing data. Table 1 compares inpainting, i.e. how well we restore a given image with missing pixels, using a pre-trained generator. GANs can be used as priors for image inpainting, but no pre-trained AmbientGANs are available for these large datasets.  Also, diffusions perform better compared to GANs for inpainting and other inverse problems. Therefore, beating state-of-the-art pre-trained diffusion models (trained on clean data!) used for inpainting, is a harder task than beating AmbientGAN-guided inpaining. Table 1 shows that we can match the performance or even outperform these stronger baselines even without having access to clean data. We hope this clarifies things and we plan to include this discussion to make it more clear in the paper.\n"}]

Author rebuttal:
We thank the Reviewers for their constructive feedback! We are very glad that our work was well-received and that the novelty, the experimental and the theoretical contributions were generally appreciated by the Reviewers.

We include separate replies to each one of the Reviewers. 

We also attach a one-page PDF that contains additional experiments and a formal statement of our sampling algorithm, as requested by some of the Reviewers. 

We remain available to answer additional questions if any!



Meta-review: The authors propose a diffusion-based framework for learning unknown distributions from heavily corrupted samples. This addresses scenarios in scientific applications where access to clean samples is infeasible or costly. A key idea involves introducing "additional measurement distortion" during diffusion and training the model to predict the original corrupted image from the further corrupted version. The authors also validate their approach on standard benchmarks (CelebA, CIFAR-10, AFHQ), showcasing the ability to learn distributions even when training samples have significant pixel gaps. 

All reviewers thought this is a nicely written paper and interesting despite some limitations. I concur and recommend acceptance.
Decision: Accept (poster)