An Expectation-Maximization Algorithm for Training Clean Diffusion Models from Corrupted Observations
Authors: WeiminBai
Yifei Wang
Wenzheng Chen
He Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion model
Inverse Problem
Computational Imaging
Primary area: diffusion_based_models

Abstract: Diffusion models excel in solving imaging inverse problems due to their ability to model complex image priors. However, their reliance on large, clean datasets for training limits their practical use where clean data is scarce. In this paper, we propose EMDiffusion, an expectation-maximization (EM) approach to train diffusion models from corrupted observations. Our method alternates between reconstructing clean images from corrupted data using a known diffusion model (E-step) and refining diffusion model weights based on these reconstructions (M-step). This iterative process leads the learned diffusion model to gradually converge to a local optimum, that is, to approximate the true clean data distribution. We validate our method through extensive experiments on diverse computational imaging tasks, including random inpainting, denoising, and deblurring, achieving new state-of-the-art performance.

An Expectation-Maxiniaation Algorithm for Training Clean Diffusion Models from Corrupted Observations
Weimin B,i
Yifei Wang
Wenzheng Chen 5.6
He Sun 1,2,3
Academy for Advanced Interdisciplinary Studies, Peking University
College of Future Technology, Peking University
National Biomedical Imaging Center, Peking University Yuanpei College, Peking University Wangxuan Institue of Computer Technology. Peking University
State Key Laboratory of Multimedia Information Processing, Peking University, Beijing, China {weiminbai, wyf181030) pku. edu. cn, (wenzhengchen, hesun)@pku. edu. cn
Abstract
Diffusion models excel in solving imaging inverse problems due their ability to model complex image priors. However, their reliance on large, clean datasets for training limits their practical use where clean data scarce. In this paper, we propose EMDiffusion, an expectation-maxiiizzton maximization (EM) approach to train diffusion models from corrupted observations. Our method alternates between reconstructing clean images from corrupted data using a known diffusion model (E- step) and refining diffusion model weights based on these reconstructions (M-step). This iterative process leads the learned diffusion model to gradually converge to local optimum, that is, approximate the clean data distribution. We validate our method through extensive experiments on diverse computational imaging tasks, including random inpainting, denoising, and deblurring, achieving new state-of-the art performance. The code is available at /tttt::///giii //gttubb......... com/ a ai4imagiiing EMDiffusoin
Introduction
Diffusion models (DMs) (I 2 have demonstrated remarkable versatility in capturing complex real-world data distributions, excelling in diverse applications like image generation 4
audio synthesis ②,, and molecular design (10). DMs approximate distributions by learning their score functions-the the gradient of the log-likelihood of the data distribution Vx Pdtta (x). This enables high-quality sample generation by simulating reverse-time stochastic differential equations (SDEs) (2) during inference.
Recently, there has been growing interest in leveraging DMs as priors for computational imaging inverse problems IT 12
16), which aim recover underlying images from corrupted observations The Bayesian framework for computational imaging defines the posterior distribution of images x given observations y:
(1) where p(y x) defines the forward model of observations and p(x) defines an image prior. DMs offer efficient, data-driven priors that outperform traditional handcrafted priors prone oversimplification and human biases, such as sparsity (17 total variation (TV) (1)..
( y) α p(y xx)p(x),
However, major limitation of DM-based solvers is their reliance on substantial volumes of high- quality, clean signals for pre-training requirement often infeasible in real-world settings, especially
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Observations
Noise
Posterior Samples
Diffusion from masked data
Diffusion from blurry data
E-steps
Diffusion
1 M-step
Diffusion from noisy data
2nd M-step
Final M-step
Figure Overview EMDiffusion. The paper proposes an expectation-maxxiiiaall (EM) approach to jointly solve imaging inverse problems and train diffusion model from corrupted observations. Left: In each E-step, we assume known diffusion model and perform posterior sampling to reconstruct images from corrupted observations. In the M-step, update the weights of the diffusion model based on these posterior samples. By iteratively alternating between these two steps, the diffusion model gradually learns the clean image distribution and generates high-quality posterior samples. Right: Raw observations and reconstructed clean images based on the diffusion model learned from corrupted data.
for scientific and biomedical imaging. In contrast, corrupted noisy observations with differentiable forward models are casier to acquire, such as blurred images from mobile photography or 2D projections of 3D structures in X-ray computed tomography (CT) (20 2D and cryogenic electron microscopy (cryo-EM) 22 23). Our paper seeks to answer a pivotal question: Can a DM be effectively trained solve inverse problems primarily using large-scale corrupted observations? This presents chicken-egg dilemma: training an accurate DM requires clean images, but reconstructing clean images from corrupted observations requires a good DM.
Utilizing the Expectation-Maxxx Maximizatiinn (EM) framework, we introduce a novel approach called EMDiffusion. This approach initializes with diffusion prior trained minimal set of clean images, then alternates between two steps across multiple iterations: reconstructing clean images from corrupted observations using the current diffusion prior (E-step), and refining the DM parameters based on these reconstructions (M-step). The sparse clean data provides a good initialization of the DM's manifold, preventing collapse into a distorted or biased distribution characterized solely by corrupted inputs. Each E-M iteration leverages the current diffusion prior to generate cleaner reconstructions from the corrupted data, and these enhanced reconstructions then update the DM, providing an improved prior for the next iteration. This cycle continues, with the generated samples and DM progressively converging toward local optima, which equals to approximate the true clean data distribution. The forward operator and noise process do not affect this type of convergence but only influence the convergence speed by determining the amount of information in the corrupted observations.
We validate the generalizability and effectiveness EMDiffusion through extensive experiments, applying it to diverse imaging inverse problems across various datasets, including random inpainting, denoising, and deblurring, and achieving compelling results.
2 Related Works
Inverse problems in computational imaging. lying signals x E IRd from corrupted observations E Rmm where the image formation process is probabilistically modeled as:
Computational imaging aims to reconstruct under-
y y~ p>y()x))
Since m I d and observation noise is inevitable, inverse problems in computational imaging are iol-posed, with the inverse mapping y - x being one-to-many. To address this complexity, Bayesian inference introduces prior distribution of underlying images, p(x), to constrain the solution space for the image posterior, p(x p(x as illustrated by Eq. [ Employing Maximum a Posteriori (MAP) estimation, one can derive point estimate of the underlying image by maximizing ((xy() Alternatively, posterior image samples of reconstructed images can be obtained through methods like Markov Chain Monte Carlo (MCMC) (2) or Variational Inference (VI) (25) 26 (2). However, the

---

performance of many computational imaging solvers is limited by their reliance on oversimplified, handcrafted priors such as sparsity and total variation (TV). These priors fail capture the true complexity of natural image distributions, hindering the solvers' ability to achieve high-quality reconstructions.
Diffusion models for inverse problems. Diffusion models (DMs) 123 have recently emerged as powerful data-driven priors for solving imaging inverse problems. By mastering the intricate distribution of images through training extensive image data, DMs facilitate both point estimates
via Plug-and-Pal (PnP) optimization (2 29) and posterior sampling through generative PnP
(GPnP) (30), PnP Monte Carlo (PMC) 31). or Diffusion Posterior Sampling (DPS) (13 [4| 32).
These approaches have demonstrated remarkable efficacy in addressing broad spectrum of noisy inverse problems, with applications spanning diverse fields, including astronomy
and
biomedical imaging 15 34).
Learn diffusion models from corrupted data. In many real-world scenarios, acquiring large-scale clean data is costly or infeasible, motivating efforts to learn DMs directly from corrupted data. Data corruptions stem from under-determined forward models (e.g., 2D projections, inpainting, compressed sensing) and measurement noise. Recent studies have explored various strategies to address these challenges. For instance, in inverse graphics, researchers integrate the forward model into the diffusion process and introduce view-consistency loss over multiple noiseless projections of the same object to learn a 3D DM from 2D images (35 36). In image inpainting, AmbientDiffusin (37) randomly masks additional pixels and forces the DM to restore these deliberate corruptions. Since the model cannot distinguish between original and further corruptions, it effectively learns the uncorrupted image distribution. However, the AmbientDiffusion limited by the additional masking technique and fails to achieve good performance with noisy observations. (38) cleverly finetunes Stable Diffusion (SD) leverage the pre-trained knowledge denoising tasks, but does not support training a DM from scratch. Meanwhile, SURE-Score (39) proposes to jointly learn an image denoiser and score score-based DM using Stein's 's unbiased risk estimate (SURE) loss, where the SURE loss acts as an implicit regularizer on the model weights. Despite its innovative approach, SURE-Score often struggles with significant data corruption, such as inpainting tasks with a large fraction of missing pixels, and tends to produce overly smooth results. A general approach for learning DMs from arbitrarily corrupted data remains an open challenge.
3 Preliminary
3.1 Score-based Diffusion Models
A diffusion model captures the data distribution by learning score function, i.c. the gradient of the logarithm of the likelihood of data distribution Vxx log Pdata Consequently, a diffusion model generates samples by gradually removing noise from random input, which is equivalent to a reverse-time stochastic differential equation (SDE) the solution to a forward-time SDE that gradually injects noise,
forward-time SDE:
dx; (x+,t) g(()dw, reverse-time SDE: dx; [f (xx,,) gtt)) logpt (x:) dt + g(t)dw,
(3)
where / E [0, f (x+,t) IRd R.. is the drift function, g(t) controls the rate of the Brownian motion w E _dd and w denotes the Brownian motion running back. A tractable isotropic Gaussian distribution is achieved when
T, i.e. XT No(,, I), and the data distribution is achieved when / - i.e. Xe Pdata- X, E Rd denotes the image Xa diffused time Vx, log.t (x;) is a time-dependent score function, which is usually approximated by a deep neural network, so parameterized by 0. The generated data distribution from the reverse-time SDE depends only this time-dependent score function.
3.2 Diffusion Posterior Sampling
Many images are consistent with a single observation due to the ill-posed nature of the image formation model. By combining the forward model with the diffusion prior using Bayes' rule,
3

---

define a conditional diffusion process that samples the posterior distribution
dx; = [f (xx: 1) Vx, log Pr (x; y)] di + s())dw
(4)
where the conditional score function can be further decomposed as:
Vx, log (xt y) Xx, log (xx) + Vx, logpt (y xx)
(5)
Sg- (x,, 1) + xx log
ply xx)p(xx xx)dxo,
X0
Since the likelihood function is only defined for t O, the dependence between y and is implicit, making Vx, log Pt (y x²) an intractable integral at each diffusion step. Various techniques have been proposed to address this intractable likelihood function, including exactly computing the probability using an ODE flow (II, bounding the probability through an evidence lower bound (ELBO) and approximating the probability using Tweedic's formula 40 efficiency, we adopt the approximation proposed in (13)
41 42).
To ensure computational
ps (y x+) p (y {xx())),, where (x(x)) E Xx xx]
for diffusion posterior sampling in all the following sections.
3.3 Expectation Maximum Algorithm
The inpectation-Mxxiiizatinn (EM) algorithm 443 44 is iterative technique for estimating parameters in statistical models involving latent variables. When the true values of the latent variables unknown, maximum likelihood estimation (MLE) cannot be directly applied to identify the model parameters. Instead, the EM algorithm maximizes a lower bound of the log-likelihood function, derived using Jensen's inequality:
(og)o = log /
/ po y) log (y x) po(x y) dx
2
/ po y) log x x)dx / po(x po y)dx
Ex- plo x) + log (x()) log po (x Iy)l ((0),
(7)
where y, and & denote the latent variables, observations, and model parameters, respectively. The algorithm alternates between two steps:
Expectation step (E-step): Sample latent variables from the current estimate of the conditional
distribution, po (x y), and compute the expected likelihodood lower bound L(0).
Maximization step (M-step): Maximize L(0) Ex~po(xly) [log (x)) update parameters 6.
This iterative procedure allows the EM algorithm to converge local maximum of the observed data log-likelihood making it powerful technique for estimation problems involving latent variables, such as Gaussian mixture clustering (45), and dynamical system identification(46)
Proposed Method
Given corrupted observations and a known forward model p(y x). learning DMs from corrupted data is a parameter estimation problem involving latent variables. The latent variables are the underlying clean images and the goal is to estimate the DM parameters that govern the image prior pe (x). Consequently, we can leverage an iterative EM approach to reconstruct clean images and train the DM using corrupted data jointly, described in Fig. 1 and Algorithm T
4.1 Initialization: Training a Vague Diffusion Model using I Limited Clean Images
The Expectation-Maxinn Maximizatiinn (EM) algorithm needs a good initialization to begin its iterative process, as an improper initialization can result in convergence at an incorrect local minimum. While clean data is often available. This
obtaining a large dataset of clean images is difficult, small limited clean data can be used to train initial DM to start the EM iterations. For example, in all

---

(a) Observations
bb)
(c) A = 10
(d) 20
(e) 50 clean images for training the initial diffusion model
Figure 2: ddaptive diffusion posterior sampling on CIFAR- R-10 inpainting, (a) Corrupted obser- vations from the test set, with 60% of the pixels masked in each image. (b), (c), and (d) Diffusion posterior samples with the diffusion prior weighted by different scaling factors: À = 1, 10, 20. The small, there is
diffusion prior is pre-trained using the 50 clean images shown in (c). When obvious mode collapse, and all posterior samples come from the training set of 50 clean images, unrelated to the observations. As À increases, the data likelihood gains more significance, resulting in reconstructed images that are more consistent with the inpainting observations.
the following experiments, 50 randomly selected clean images used to train the initial DM, serving as the starting point for the EM algorithm. As demonstrated in Sec. [5.4] clean images do not need to be from the same dataset; those from unt-of-diitrrbution datasets also serve as reasonable initializations.
4.2 E-step: Adaptive Diffusion Posterior Sampling
In the E-step, we assume a known diffusion prior and reconstruct the underlying clean images through diffusion posterior sampling. We adopt the standard variance- variance-preserving preserving form of the stochastic differential equation (VP-SDE) (2, which equivalent the Denoising Diffusion Probabilistic Models (DDPM) (I. The drift function f(x,, t) takes the form ((()x,/ and the diffusion rate g()) is 3()). Therefore, the reverse diffusion sampler in Eq. can be represented as:
B() 2
dx;
- 3(t) log (x, y) df + BB(t)dw
(8)
Considering known imaging forward model, AA, and additive Gaussian noise, ply x) ~ Ny A(x), IT),, the conditional score function can be represented as:
Vx, log (xe Iy) = Vx, log (x+) - Vx, log (y xx) 1
(9)
(x(e t)
Vx, Ily A (xt (x:))l12
20
where
xx(xx)=
[x + (1 - -((t) (x),,
a(t) II (1 -x)))
(10)
v a(t))
However, a naive diffusion posterior sampling approach using Eqs. and often fails to produce high-quality reconstructions. This is because the learned DM inaccurate during the early EM iterations. We demonstrate this issue with a toy experiment. We performed diffusion posterior sampling (DPS) on randomly masked observations, as shown in Fig. 2a), a). using an initial DM trained on only 50 clean images. The resulting posterior samples, depicted in Fig. [(b). show mode collapse due to the severely limited prior. All recovered samples come from the training set of 50 clean images and are unrelated to the observations. Similarly, if the DM is trained on blurry, noisy images with artifacts, naive DPS also performs poorly in image reconstruction.
5

---

Algorithm EMDiffusion: Learning Score-based Priors for Inverse Imaging
Require: DM Se, Observations A), few clean data x, Cycles N, Timesteps T, Epoches M, Measurement noise (0, I²I), Diffusion rate {
1: Initialize So x through denoising score matching (47
2: for = to N do
3: f) ~ Dataset( AA
4: Xy ~ I),
5: for 7 to do
6:
d₂ - II.-I1 B.)
(8)
+ (1 az) so (x ,1)
7:
&o +
8:
z ~ (0,10,
9:
Take reverse-time SDE step (Sampling in Sec.
4.2
1-1 - (0) + B(L)
2 (x
lly I (800111)
+ /3(1zz
20
end for
11:
for m = 0 to M do
xdata Shufflo(X (6)
po (x(01) |y(1)
13:
1 ~~ Uniform((11 (I....,, TT)
14:
ā -I-
15:
E c~N(0,1)
16:
Take gradient descent step (Optimization in Sec. 4.3
V |e - co (Vāxxdata + VI - &,e,t)
17:
end for
18:
end for
does not mean that these low-quality DMs cannot provide any prior information. Although the prior is poor in the early training stages, it has learned common features and structures shared among natural images, such as the continuity and smoothness of natural images and profiles of specific object types. By introducing collapse, we find that the low-quality DM can also act weak prior for posterior sampling, where the reverse-time SDE can be written as:
À to rescale the likelihood term and avoid mode
dx B(1)
2 (Vx, log (xx) XVx, logp, (y x:)) di /B(l)dw
(1I)
(x) A Vx.
B(t)
- ( (xx)) dt V B(t)dw,
20
The hyper-parameter À efficiently balances the diffusion prior and the data likelihood, resulting in reliable reconstructed images even when the prior is poor. As demonstrated in Fig. (b). (c), and (d),
as À increases from 1 to 20, the data likelihood term gains more emphasis, making the reconstructed images more consistent with the inpainting observations. The choice of the hyper-parametee is automated in each E-step by finding the value that minimizes the data loss,
|ly - A(Xo.)))1
X = arg mink Ko.a
(12)
where Yg,A represents the diffusion posterior samples of reconstructed images with À scaling.
4.3 M-step: Optimizing Score-Based Priors
During the M-step, update the weights of the score-based models using the posterior samples obtained in the E-step. This resembles training standard clean DM, Sg. to approximate the time- dependent score function, Xx, log p(x, 2o), through denoising score matching (47):
0 arg MiinE,,... 8o
[|so(xe,)] t) Vx, p(x xt)]1
(13)
so
where L Uniform(() ((), TJ), 2o = X0.x- represents the posterior samples from the previous E-step, and x; p(x; Xo) are generated by the forward-time SDE in Eq. 3

---

Misked Olvervation
Ambient Diflision th
Oue Deration
Our Final
Ground Trun
Score (w
Ntentoon
Clean Price
Figure 3: Results on CIFAR-10 inpainting. In each image, 60% of the pixels are masked. As the EM iterations progress, the diffusion model learns cleaner prior distributions, improving the quality of posterior samples. Our method significantly outperforms the baselines, SURE-Score and AmbieniDiffussin, achieving reconstruction quality comparable DPS with clean prior.
To accelerate training, especially during the early stages when the posterior samples are noisy, the M-step does not always train the score function s-(-) from scratch. In the imitial M-steps, we inherit the DM weights from the previous iteration and fine-tune them only using posterior samples from subset of observations (e.g., randomly select 10% of total observations). However, once the quality of reconstructed images improves sufficiently, we reinitialize the DM weights and retrain the model with 100% data for few more iterations. The training strategy transitions when the optimal balancing parameter, X", falls below I, or fails to decrease for more than three consecutive iterations.
5 Experiments
In this section, we demonstrate the performance of our method learning DMs from corrupted data and solving inverse problems using these models. We validate the method three imaging tasks: random inpainting, denoising, and deblurring. Our main results are presented in Fig Fig 4 and Table with additional ablation studies in Fig. Further details on neural network architectures, training settings, and additional reconstruction and generation samples are provided in the appendix.
5.1 Datasets and Evaluation Metrics
The experiments are conducted on the CIFAR-10 (88) and CelebA 44) datasets at resolutions of 32 32 and 64 x 64, respectively. CIFAR-10 consists of 50,000 images across 10 classes for training, while CelebA contains 30,000 images of human faces. At each iteration, 5,000 corrupted images are randomly chosen for posterior sampling and training, and 250 corrupted images from the test set chosen for evaluation.
We evaluate the performance of method using two groups of metrics. First, compute the peak signal-to-non to--tooooo--rttii (PSNR) and learned Perceptual Image Patch Similarity (LPIPS) scores between the reconstructed and ground-truth images, quantifying the accuracy of inverse imaging using learned DMs. Additionally, we compute the Fréchet Inception Distance (FID) between the learned DMs and reserved test data to assess their image generation quality.
5.2 Baseline and Training Settings
We compare method with three related baselines: AmbieniDiffusion (37). SURE-Score (39), and DPS with clean prior (13), AmbicntDiffusion and SURE-Score have similar settings to our method, which do not require DMs pre-trained on large-scale clean signals. Considering AmbientDiffusion is well-designed for masked observations, we only use it as the baseline of the image inpainting task. On the other hand, DPS leverages a pre-trained clean diffusion prior for posterior sampling, it defines the performance upper bound for our method.
In our experiments, we randomly select 50 clean images from each dataset to train the initial DMs for the EM iterations. AmbientDiffusion is trained with the standard setting in (37). The key hyper- parameter of SURE-Score dw, is set to the observation noise's 's standard deviation (0.2 for denoising,

---

Table I: Numerical Results of inverse imaging and learned priors. The average values of PSNR/LPIPS are from 250 samples randomly selected from the test set. FID is used evaluate the quality of learned priors by comparing 50,000 generated samples to the train set. Optimal results are highlighted in bold and suboptimal results in underline. Note that we take DPS w/ clean prior as the upper bound.
CIFARIO-Inpaintin
Dennsii-Deniisigg
CelebA-Deblurring
Method
PSNRT
LPIPS|
FID
PSNRT
LPIPS
FID4
PSNRT
LPIPS|
FID
Observations
13.49
0.295
234.47
18.05
0.047
132.59
22,47
0.365
72.83
DPS w/ clean prior
25.44
0.008
7.08
25.91
0.010
7.08
29.05
0.013
10.24
Noise2Self (50)
21.32
0.227
92.06
SURE-Score (39)
15.75
0.182
220.01
22.42
0.138
132.61
22.07
0.383
191.96
AmbientDiffusion
(37
20.57
0.027
28.88
Ours
24.70
0.009
21.08
23.16
0.022
86.47
23.74
0.103
91.89
Noisy Observation
Score
SL Ku
Ours
DPS LB w/
Clean
Ground Touth
Observation
Score 20
SURE
Ours
DPS (B) w/
Clean
Ground Truth
(a) CIFARIO, Denoising
(b) CelebA, Deblurring
Figure 4: Results on (a) CIFAR-10 denoising and (b) CelebA deblurring. Our method significantly outperforms the baseline, SURE-Score, and approximates DPS with clean prior.
and 0.01 for inpainting and deblurring). To ensure a fair comparison, we also provide the same 50 clean images for training AmbientDiffusion and SURE-Score. Details are in Appendix A
5.3 Results
Image inpainting, We conduct random inpainting (with mask probability 0.6) on CIFAR-10 10. As shown in Fig. 3 and Table []] our method significantly outperforms AmbicntDiffusion and SURE- Score, achieving reconstruction quality similar to DPS with prior trained on the clean CIFAR-10 dataset. The iterative training process also illustrated in Fig. 3 Initially, our method performs poorly with the DM trained on only 50 randomly selected clean samples. However, as the E-step and M-step alternate iteratively, the quality of posterior sampling improves. Large-scale posterior samples enrich the priors, leading to enhanced performance at each stage.
Image denoising. We perform image denoising on CIFAR-I0 with Gaussian noise n ~ (N0 and 0.2. The results are shown in Fig. 4 a) and Table I Our method outperforms SURE-Score, and the self-supervised denoising benchmark, Noise2Self (50), though it slightly lags behind DPS with clean priors. However, while method's reconstructions may appear noisier than DPS results, they sometimes reproduce more details, such as the car wheels in the second row and the cat face in the third row of Fig. 4a), showcasing the better diversity of our learned DMs.
Image deblurring. We validate image deblurring on CelebA using a Gaussian blur kernel with a size 9 and standard deviation of a pixels. The results are shown in Fig. [bb) and Table [ As with the other tasks, our method significantly outperforms SURE-Score in solving imaging inverse problems, recovering fine details of human faces. However, the FID score of our learned diffusion models lags behind the original blurred observations. This is primarily because the FID score measures image similarity mainly through smooth features, making it a less effective metric for deblurring tasks.
8

---

100
180
Inseiming
50k comppped dota
Iteration heration Iteration
Denotsing
50k cise dete
28
Deblurring
540
20K clean data
40K clear data
120
15k clivet dade
Iteratoon
10k da13
Optimal
.10
40
500.0
100.0
50:00
2010
50,000
0.2.
Data
EM Iteration
Scaling factor
(a) Initialization DMs
(b) DMs after each EM iteration
(c) Trend of optimal
Figure 5: Ablation studies. (a) PSNR of diffusion posterior samples generated by the initial diffusion
models trained on different amounts (10, 50, 100, 500) or types (in-distribution or out-of -f--s--ibiittioon) of clean data. (b) FID scores of learned diffusion models after each EM iteration. The diffusion model trained on 50,000 corrupted images achieves a similar performance to those trained on 15,000-20,000 clean images. (c) PSNR of diffusion posterior samples weighted by different scaling factors À at each stage. The optimal À for posterior sampling decreases as the EM iterations progress.
Comparing the results of all three tasks, we find that AmbientDiffusiin only works well for inpainting because its additional masking technique is specifically designed for that purpose. SURE-Score consistently produces over-smoothed results because the SURE loss regularizes the gradient of generated images. As a comparison, our method does not make any special assumptions and provides general framework applicable to all three tasks. The generation results are in Appendix D
5.4 More Analysis and Ablation Studies
Number of clean images for training initial DMs. Our EM approach starts with DMs trained on a small set of clean images. Fig. 5)a) shows the PSNR of posterior samples generated by these models in the first E-step, allowing us to evaluate the impact of the number of clean training images on the performance of the initial DMs. Remarkably, DMs trained on as few as 10 clean images (0.02% of the corrupted images) can still act as reasonable priors. For inpainting and denoising tasks, DMs trained on 10 clean images provide nearly the same reconstruction quality those trained on 500 clean images, as these tasks primarily require priors for fwwefreuunncy features, and 10 images suffice for initial guess. However, the deblurring task benefits from DMs trained on more clean data since deblurring aims to recover gigh-frequency details where more data helps.
Surprisingly, we find that DMs trained on clean images from CelebA (downsampled to 32x32) can also be used to initialize tasks on CIFAR- 10. For inpainting and denoising tasks, DMs initialized with 50 clean images from an out-of-distriiution (OOD) dataset (CelebA) achieve similar performance to those initialized with in-distribution (ID) data (CIFAR-10), For the deblurring task, DMs trained on OOD data perform better than those trained on similar amount of ID data, suggesting that OOD data can sometimes serve as stronger priors for guessing high-frequency information.
Learned priors through iterative training. Fig.5(b) 5(b) shows the FID scores of the learned DMs in the inpainting task with 50,000 corrupted CIFAR-TO images after each EM stage. The generation ability of the DMs gradually improves the EM iterations progress. As explained in Sec. 4.3 initially the DM inherits weights from previous steps for fast training and converges at the sixth iteration. After resetting the DM, the training resumes for three more rounds and finally converges to a FID score of approximately 21.08, significantly better than AmbientDiffusion'ss 's 28.88, setting new state-of-the-art. Notably, our method achieves this performance using a DM architecture with far fewer parameters: our method employs a vanilla DDPM with 35.7 million parameters, while AmbientDiffusion uses improved DDPM++ architecture with over 100 million parameters. Additionally, we compared our method with DMs trained on different amounts of clean data. Our model, learned from 50,000 corrupted images (60% pixels masked) using EM, performs better than DM trained on around 15,000 clean images.
Scaling factor in adaptive diffusion posterior sampling. Fig. 5(c) shows the PSNR of recon- structed images at each EM stage with different scaling factors, using the inpainting task on CIFAR-10 10 as example. We observe that the quality of posterior sampling initially improves and then de-

---

teriorates as the scaling factor increases, confirming the existence of an optimal scaling factor as suggested in Eq. As the EM stages progress, the optimal scaling factor decreases, indicating that the learned priors progressively improve through the EM iterations. This observation justifies the need for adaptive scaling factors in our method.
Conclusion
In this paper, we proposed EMDiffusion, novel training diffusion models primarily from corrupted observations. The key assumption is that it is commaaaionnthhorticalll possible to learn the underlying distribution from measurements. Our method demonstrated state-of-the-art performance in image inpainting, denoising, and deblurring across various datasets. Additionally, an important finding is that a small amount of clean, in- distribution data act as an implicit regularizer, aiding the training of diffusion models from corrupted observations. Future work will aim to 1) extend initialization approaches, potentially by incorporating foundation models traditional machine learning techniques, such using prepro- cessed images from unsupervised inpainting, deblurring, or denoising for initialization, and 2) extend to various imaging inverse problems and learning unknown forward models or noise statistics.
(EM) framework for
Acknowledgement
This work was supported by the National Natural Science Foundation of China(62371007) and the National Key Research and Development Program of China (2022YFC3401100)
References
[1] J. Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models," Advances in neural information processing systems, vol. 33, pp. 6840-6851 6851, 2020.
[2] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-based generative modeling through stochastic differential equations," arXiv preprint arXiv:001..13456 2020.
[3] Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, "Deep unsupervised learning using nonequilibrium thermodynamics," in International conference on machine learning. PMLR, 2015, pp. 2256-2265.
[4] A. Q. Nichol and P. Dhariwal, "Improved denoising diffusion probabilistic models," in Interna- tional Conference on Machine Learning. PMLR, 2021, pp. 8162-8171.
[5] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, "Zero- shot text-to-image generation," in International Conference on Machine Learning. PMLR, 2021, pp. 8821-8831.
[6] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, "Hierarchical text-conditional image generation with clip latents," arXiv preprint 2204:2204.0 2022.
[7] R. Rombach, A. Blattmann, D. Lorenz, Esser, and B. Ommer, "High-resolution image synthesis with latent diffusion models," in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022, pp. 695-
[8] C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi, "Palette: Image-to-image diffusion models," in ACM SIGGRAPH 2022 Conference Proceedings, 2022, pp. 1-10.
[9] Kong, W. Ping, J. Huang, K. Zhao, and Catanzaro, "Diffwave: A versatile diffusion model for audio synthesis," arXiv preprint 20X0:20009.07611, 2020.
[10] M. Xu, I.. Yu, Song, Shi, S. Ermon, and J. Tang, "Geodiff: geometric diffusion model for molecular conformation generation," arXiv preprint arXiv:2203.0291 2022.
10

---

[11] B. T. Feng. J. Smith, M. Rubinstein, H. Chang, K. L. Bouman, and W. T. Freeman, "Score-based diffusion models as principled priors for inverse imaging," arXiv preprint arXiv:2304.11751 2023.
[12] G. Z.Zhang, Ji, Y. Zhang, M. Yu, T. Jaakkola, and S. Chang, "Towards coherent image inpainting using denoising diffusion implicit models," in International Conference on Machine Learning. PMLR, 2023, pp. 41 (64-11 193.
[13] H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye, "Diffusion posterior sampling for general noisy inverse problems," arXiv preprint arXiv:2209.1400 14687, 2022.
[14] H. Chung, B. Sim, D. Ryu, and J. C. Ye, "Improving diffusion models for inverse problems using manifold constraints," Advances in Neural Information Processing Systems, vol. 35, pp. 25 683-25 696, 2022.
[15] Y. Song, L. Shen, L. Xing, and S. Ermon, "Solving inverse problems in medical imaging with score-based generative models," arXiv preprint arXiv:21 2021.
[16] B. Kawar, N. Elata, T. Michaeli, and M. Elad, "Gsure-based difflusion model training with corrupted data," arXiv preprint arXiv: 23ti:22005.1122 13128, 2023.
[17] W. Zhao, S. Zhao, L. Li, X. Huang. S. Xing. Y. Zhang, G. Qiu, Z. Han, Y. Shang, D.-c. Sun et al., "Sparse deconvolution improves the resolution live-cell super-resolution fluorescence microscopy." Nature biotechnology, vol. 40, no. 4, pp. 606-617, 2022.
[18] C. Bouman and K. Sauer, "A generalized gaussian image model for edge-preserving map estimation," " IEEE Transactions on image processing, vol. 2, no. 3, pp. 296-310, 1993.
[19] K. Kuramochi, K. Akiyama, S. Ikeda, F. Tazaki, V. L. Fish, H.-Y. Pu, K. Asada, and M. Honma, "Superresolution interferometric imaging with sparse modeling using total squared variation: application to imaging the black hole shadow," The Astrophysical Journal, vol. 858, no. 1, p. 56, 2018.
[20] C. Wang, K. Shang, H. Zhang, Q. Li, Y. Hui, and S. K. Zhou, "Dudotrans: dual-domain transformer provides more attention for sinogram restoration sparse-view et reconstruction," arXiv preprint arXiv:2111.1077
2021.
[21] A. W. Reed, H. Kim, R. Anirudh, K. A. Mohan, K. Champley, J. Kang, and S. Jayasuriya, "Dy- namic et reconstruction from limited views with implicit neural representations and parametric motion fields," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2258-225666
[22] E. Zhong, T. Bepler, B. Berger, and J. H. Davis, "Cryodrgn: reconstruction of heterogeneous cryo-em structures using neural networks," Nature methods, vol. 18, no. 2, pp. 176-185, 2021.
[23] E. Ye, Y. Wang. H. Zhang, Y. Gao, H. Wang, and H. Sun, "Recovering a molecule's ' 3d dynamics from liquid-phase electron microscopy movies," " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 10767-10777.
[24] S. Brooks,, A. Gelman, G. Jones, and X.-L. Meng, Handbook of markov chain monte carlo. CRC press, 2011.
[25] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, "Variational inference: A review for statisti- cians," Journal of the American statistical Association, vol. 112, no. 518, pp. 859-877, 2017.
[26] H. Sun and K. L. Bouman, "Deep probabilistic imaging: Uncertainty quantification and multi- modal solution characterization for computational imaging," in Proceedings of the AAAI Con- ference on Artificial Intelligence, vol. 35, no. 3, 2021, pp. 2628-2637.
[27] H. Sun, K. L. Bouman, P. Tiede, J. Wang. S. Blunt, and D. Mawet, "a-deep probabilistic inference (a-dpi): efficient uncertainty quantification from exoplanet astrometry to black hole feature extraction." " The Astrophysical Journal, vol. 932, no. 2, p. 99, 2022.
[28] A. Graikos, N. Malkin, N. Jojic, and D. Samaras, "Diffusion models as alug-and-play priors," Advances in Neural Information Processing Systems, vol. 35, pp. 14 7128, 2022.
11

---

[29] Y. Zhu, K. Zhang, J. Liang, J. Cao, B. Wen, R. Timofte, and L. Van Gool, "Denoising diffusion models for plug-ant and-play image restoration," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1219-1229.
[30] C. A. Bouman and G. T. Buzzard, "Generative plug and play: Posterior sampling for inverse problems," arXiv preprint arXiv:2.306.00 07233,, 2023.
[31] Y. Sun, Z. Wu, Chen, B. T. Feng, and K. L.. Bouman, "Provable probabilistic imaging using score-based generative priors." arXiv preprint arXiv:2310. 10835, 2023.
[32] H. Chung, B. Sim, and J. C. Ye, "Come-close CCome-closer-oeee whaeeccooooooossee------------ee diffuse-faster. Accelerating conditional diffusion models for inverse problems through stochastic contraction," " in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp- 12413-12422.
[33] B. T. Feng and K. L. Bouman, "Efficient bayesian computational imaging with surrogate score-based prior," arXiv preprini arXiv:2309.01990 2023.
[34] H. Chung and J. C. Ye, based diffusion models for accelerated mri," Medical image analysis, vol. 80, p. 102479, 2022.
[35] A. Tewari, T. Yin, G. Cazenavette, S. Rezchikov, J. B. Tenenbaum, F. Durand, W. T. Freeman, and V. Sitzmann, "Diffusion with forward models: Solving stochastic inverse problems without direct supervision," arXiv preprint arXiv:2306. /1719, 2023.
[36] T. Anciukevičius. Z. Xu, M. Fisher, P. Henderson, H. Bilen, N. J. Mitra, and P. Guerrero, "Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation," " in Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, 2023, pp. 12 128-12611 12608-12618 618.
[37] G. Daras, K. Shah, Y. Dagan, A. Gollakota, A. G. Dimakis, and A. Klivans, "Ambient diffusion: Learning clean distributions from corrupted data," 2023.
[38] G. Daras, A. G. Dimakis, and C. Daskalakis, "Consistent diffusion meets tweedie: Training exact ambient diffusion models with noisy data," arXiv preprint arXiv:2404.10177 2024.
[39] A. Aali, M. Arvinte, Kumar, and J. I. Tamir, "Solving inverse problems with score-based generative priors learned from noisy data," arXiv preprint arXiv:2.35.00 /66, 2023.
[40] B. Kawar, M. Elad, S. Ermon, and J. Song, "Denoising diffusion restoration models," Advances Neural Information Processing Systems, vol. 35, pp. 23 23.593-23 606, 2022.
[41] A. Jalal, M. Arvinte, G. Daras, E. Price, A. Dimakis, and Tamir, "Robust compressed sensing mri with deep generative priors," Advances in Neural Information Processing Systems, vol. 34, pp. 14 948--444444 2021.
[42] Song, A. Vahdat, M. Mardani, and Kautz, "Pseudoinverse guiddd diffusion models for inverse problems," in International Conference on Learning Representations, 2022.
[43] A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the em algorithm," " Journal of the royal statistical society: series B (methodological), vol. 39, no. 1, pp. 1-22, 1977.
[44] A. Gao, J. Castellanos, Y. Yue, Z. Ross, and K. Bouman, "Deepgem: Generalized expectation- maximization for blind inversion." Advances in Neural Information Processing Systems, vol. 34, pp. 11 592-11 603, 2021.
[45] D. A. Reynolds et al., "Gaussian mixture models." Encyclopedia of biometrics, vol. 741, no. 659-663, 2009.
[46] H. Sun, J. Kasdin, and Vanderbei, "Identification and adaptive control of a high-contrast local plane wavefront correction system." Journal of Astronomical Telescopes, Instruments, and Systems, vol. 4, no. 4, pp. 049 049 006, 2018
[47] Vincent, "A connection between score matching and denoising autoencoders," " Neural compu- tation, vol. 23, no. 7. pp. 1661-1674, 2011.
12

---

[48] A. Krizhevsky, G. Hinton et al., "Learning multiple layers of features from tiny images," 2009.
[49] Z. Liu, P. Luo, X. Wang, and X. Tang, "Deep learning face attributes in the wild," in Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[50] Batson and L. Royer, "Nois22sllf: Blind denoising by self-supervision." in International Conference Machine Learning. PMLR, 2019, pp- 524-533.
[51] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation." " in Medical image computing and computer-assisted intervention-MICAAl 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer, 2015, pp. 234-241.
[52] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp- 770-778.
[53] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization." arXiv preprint arXiv17711.05001.0011 2017.
[54] D. Kim, S. Shin, K. Song, W. Kang, and L.-C. Moon, "Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation," arXiv preprint 1XXi:2206.05527, 2021.
[55] T. Karras, M. Aittala, T. Aila, and S. Laine, "Elucidating the design space of diffusion- based generative models," Advances in Neural Information Processing Systems, vol. 35, pp. 26 565- 26 577, 2022.
[56] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, "Free-form image inpainting with gated convolution," " in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 4471-4480.
[57] Y. Song and S. Ermon, "Improved techniques for training score-based generative models," Advances in neural information processing systems, vol. 33, pp. 12438-12 448, 2020.
[58] A. Danielyan, V. Katkovnik, and K. Egiazarian, "Bm3d frames and variational image deblurring," IEFE Transactions on image processing, vol. 21, no. 4, pp. 1715-1728, 2011.
[59] A. Beck and M. Teboulle, "Fast gradient- algorithms for constrained total variation image denoising and deblurring problems," IEEE transactions on image processing, vol. 18, no. 11, pp. 2419-2434, 2009.
13

---

We provide the implementation details and more results in the appendix. We first describe our network architecture and training settings Sec. A]
then show initialization details in Sec. More results are
also provided in Sec. C Cnnd Sec. D
A
Implementation Details.
Our neural network architecture follows the vanilla denoising diffusion probabilistic model (DDPM) ( For quick implementation, see htttps:/huugginffee co/ /oogoogle//
adpm-cifarll 32 and httts://huggingfaeee co/ wttpg/oo////oooool///oeeeeoeeeee
256
Model architecture. Our architecture is exactly aligned with DDPM (I, which is U-Net (51) based on a Wide ResNet (52). Diffusion time is implemented by adding the Transformer sinusoidal position embedding into each residual block. For CIFAR-I0, 10, our 32 32 models use four feature map resolutions (32 x 32 to 4 x 4) and convolutional residual blocks per resolution level. For CelebA we increase the feature map number for our 64 64 to six. We enable the dropout regularization to reduce overfitting, Our CIFAR-10 10 model has 35.7 million parameters and our CelebA model has 114 million parameters.
Noise schedule. We leverage the default settings on VP-SDE 2,, which uses linear schedule with timesteps T 1000, $1
Le 4, and BT 0.02.
Exponential moving average. To stabilize the training process and reduce the color shift of samples generated by trained DMs, we adopt an exponential moving average (EMA) technique with decay factor of 0.999 for all experiments.
Optimizer. We apply AdamW (53) and set the learning rate to 2e 4 for CIFAR-10 and 2e 5 for CelebA.
Hyperparameters for the training process. We set the batch size to 512 for CIFAR-10 and 64 for CelebA. We set the dropout rate to 0.1 for CIFAR-I0 and 0 for CelebA. As for the learning rate, we adapt le 4 for CIFAR-10 and 2e 5 for CelebA, for larger learning rate will result in unstable training.
Dataset argumentation. We only use random horizontal flips to CIFAR-10 during training to achieve better performance. We did not flip CelebA, as the distribution human faces is quite simple. We provide table of training hyperparameters in Table
Table 2: Training hyperparameters.
Dataset
Batch Size Epoches
LR
Dropout Optimizer
Data Augmentation
CIFAR-10
512
1000
2c-4
0.1
AdamW Random horizontal flips
CelebA
128
1500
2e-5
0
AdamW
Baselines. As for Ambient Diffusion, We use the official checkpoint, which adopts the improved DDPM++ architecture (54) and the EDM scheduler design (55). It also modifies the model ar- chitecture's 's convolutions to Gated Convolutions (56). as they are known to perform better for inpainting-type problems. Their 32 x 32 model has 113 million parameters, which is larger than ours. SureScore, on the other hand, uses the deepest NCSNv2 (57) model architecture, which has 95 million parameters, As for DPS with clean priors, adopt the pre-trained DDPMs provided by htttps:/hhuginnffaat co/ google The details of the architectures of all methods are shown in 3
Training schedule. At each EM iteration, we randomly choose 5,000 corrupted observations for diffusion posterior sampling and then train DMs. We further divide the iterations into two phases:
Phase - Resume training DMs: at the early EM iterations, we inherit weights of DMs from the last iteration for quick convergence. For CIFAR-I0, 10, this phase lasts for about 6-8 EM iterations, while for CelebA deblurring, this phase increases to 10 EM iterations.
14

---

Table 3: Method architecture comparison.
CIFAR-10
CelebA
Model
Param
Arch
Memory Schedule Param
Arch
Memory Schudule
Ours
35.7M DDPM 140MB
VP
114M DDPM
454MB
VP
Ambient Diffusion
113M
DDPM++ 408MB
EDM
445MB
DDPM++ 451MB
EDM
SureScore
90M NCSNv2 1.5GB
VE
90M NCSNv2
1.5GB
VE
Phase 2 Reset training DMs: at the later EM iterations, we reset the weights of DMs at each M-step, that is, training DMs from scratch. The key insight is that DMs from Phase always have a memory of bad posterior samples, which has negative effect on the learned distribution. For CIFAR-10, 10, this phase lasts for EM iterations, we found it significantly improves the FID score of DMs. While for CelebA deblurring, this phase lasts for 2 EM iterations until we find the improvement is not obvious.
B Additional Strategies for Training Initial DMs
To verify the sensitivity of EMDiffusion's initial DM training data, we provide more quantitative and qualitative results in this section, as shown Table 4 We draw the conclusion that EMDiffusion not sensitive to the initial data. Apart from evaluating different numbers of in-distribution (ID) images and out-of-distribution (OOD) images for training the initial DMs, we also test the initialization on preprocessed observations, and find all of them converge similarly.
Table Numerical results of different data for training initial DMs. We show the PSNR values of posterior sampling with diffusion initialized with different data. The results show that EMDiffusion is insensitive to initializations.
Initialization
Observations 500 ID 100 ID 50 ID 10 ID 50 OOD 50 preprocessed
CIFAR I0-nnniin
13.49
20.58
20.95
20.93
20.21
20.10
16.16
CIFAR 10-Denossin 18.05
19.89
20.48
20.47
19.88
19.57
19.96
CelebA-Deblurring 22.47
25.56
24.99
22.80
21.19
25.50
22.00
Specifically, we preprocess the noisy observations with BM3D (8), the blurry observations with Fast TV Constraint (59), and leave the masked observations unchanged. DMs initialized on these preprocess samples also perform well on the following E-step.
C Additional Results on Random Inpainting
Table 5: Comparison of FID scores between the EM approach and Ambient Diffusion across different
corruption levels (masking ratio p 0.4, 0.6, 0.8, 0.9) CIFAR-10 and CelebA.
CIFAR-10
CelebA
Model
0.4
0.6
0.8
0.6
0.8
0.9
Ambient Diffusion
18.85
28.88
46.27
6.08
11.19
25.53
Ours
13.75
21.08
45.24
5.98
13.26
29.09
As shown in Table we include additional comparisons to Ambient Diffusion 337 across various corruption conditions and datasets. The performance gap between the proposed method and Ambient Diffusion narrows under higher levels corruption, likely due to the simpler DDPM architecture we employ. In contrast, Ambient Diffusion utilizes the improved DDPM++ architecture (54), which is specifically modified to perform better for high-corrutiion inpainting-type problems. Nonetheless, the overall performance demonstrates the effectiveness of the proposed method, as we do not focus on empirically optimized architecture details, instead show the applicability of the new EM idea for training DMs from corrupted observations.
15

---

D
Generative Samples
EMDiffusion is proposed learn clean distributions from corrupted observations. In Sec.
present detailed posterior sampling results and FID scores of learned DMs.
Our model outperforms baselines by significant margin in three inverse imaging tasks two datasets. Though the FID score of our model trained on blurry CelebA is slightly higher than Ambient Diffusion, we argue that FID scores are easily influenced by sharp artifacts introduced by DPS (3), which is adopted in our E-steps. However, the distributions EMDiffusion learned from various types of corrupted observations are obviously better than baselines, shown in Fig. 768
Future work. To achieve a better posterior distribution through the proposed EM framework, an accurate and efficient E-step plays key role. We adopt the Diffusion Posterior Sampling (13) that could potentially introduce artifacts due to its approximation of the underlying data likelihood term. Therefore, a better FID score could be achieved by designing principled posterior sampling method.
(a) SURE-Score, FID=191.96
(c) Ours. FID=91.89
Figure 6: Uncurated Samples generated from models trained on blurry CelebA.
16

---

(a) SURE-Score, FID=220.01
(b) Ambient Diffusion, FID=28.88
(c) Ours, FID=21.08
Figure 7: Uncurated Samples generated from models trained on random masked CIFAR-10.
17

---

(a) SURE-Score, FID=133.61
(c) Ours, FID=86.47
Figure 8: Uncurated Samples generated from models trained noisy CIFAR-10 -00.
18

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We make clear claims that well reflect the paper's contributions and scope
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitation in the last.
19

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't ' acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We show ablation study for the proposed techniques.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide code in supp.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guideeee http://oeeubiiisssoooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
21

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all the experiment details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our experiments contain large amount of examples that don't need the error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details of the computation resources.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
22

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: attps://neurrps:: httpp//c..........t.tt..oooo....
Answer: [Yes]
Justification: Our research explores computational imaging algorithms in which we didn't see any ethics issues.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our research explores computational imaging algorithms in which we didn't "ttn't see any negative social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We study computational imaging algorithms and didn't 't see such risks.
23

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We use public datasets that are licensed for research purposes.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We did;t use new assets
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?

---

Answer:
[NA]
Justification:
We didn't do crowd sourcing with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We didn't do experiments related to human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "Ig8h2oSi5M", "reviewer": "Reviewer_qpvz", "summary": "The authors introduce a new framework for training diffusion models from corrupted data. Prior work on this research topic is based on the Ambient Diffusion framework or Stein's Unbiased Risk Estimate (SURE) idea. The authors propose an alternative methodology based on the Expectation-Maximization algorithm. The algorithm is tested on standard datasets and leads to improved performance over the considered baselines.", "questions": "See the Weaknesses Section above. I am willing to further increase my score if my concerns are properly addressed.", "limitations": "The authors have adequately addressed the limitations of their method.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. The research topic of training diffusion from corrupted data is relevant and timely.\n2. The authors propose a fresh idea for this interesting research problem. The proposed idea has several elegant properties: I) it is simple, ii) relatively easy to implement, and iii) the idea is oblivious to the type of corruption.\n3. The fact that the proposed algorithm is agnostic to the type of corruption is important: as the authors correctly mention, prior work, such as Ambient Diffusion, is designed (or at least tested) only for inpainting. \n4. The authors manage to outperform the prior state-of-the-art, Ambient Diffusion in the same setting (CIFAR-10 with 60% corruption). \n5. The authors have a nice way of leveraging clean samples. Prior work assumes that we don't have any access to clean samples. This paper naturally integrates the available clean data by using them to initialize the EM algorithm.\n\nOverall, I believe this submission offers a promising alternative to Ambient Diffusion for learning diffusion models with corrupted data.", "weaknesses": "1. The comparison to the baselines is somewhat limited/unfair. As the authors mention, Ambient Diffusion has been designed for inpainting. The authors use Ambient Diffusion for denoising and Deblurring. It is not clear how they do this since the original algorithm cannot be trivially extended to these settings. Even for the case of inpainting, the authors could report results for other levels of corruption as well (apart from $p=0.6$) and for other datasets apart from CIFAR-10. Regarding the comparisons with Ambient Diffusion, the only valid data point right now is for $p=0.6$. Providing a more comprehensive analysis would help the readers understand in what regime the proposed method works the best.\n2. Specifically for the case of denoising, the authors might want to provide comparisons with the more recent work Consistent Diffusion Meets Tweedie which extends the Ambient Diffusion idea to the setting of noisy samples. \n3. For some scientific applications, there is no available clean data at all. The proposed algorithm needs some clean data to initialize the EM and this could be limiting for such applications.\n4. One issue with the proposed method is that it relies on solving inverse problems with diffusion models. This problem is computationally intractable and for that reason, there has been a large body of research papers proposing approximations to the conditional score. The authors rely on such an approximation (DPS) which is known that doesn't truly offer samples from the posterior unless the data distribution is very simple (e.g. Gaussian).\n5. One issue with this method seems to be the training speed. Even for converge to a local optimum, EM typically requires many steps. Each step here is a new training of a diffusion model. Prior works on learning diffusion models from corrupted data only required a single training.\n6. For some corruption types, it is impossible to ever reconstruct the true distribution from noisy observations. The authors do not discuss how the proposed algorithm would work in such cases.\n7. Minor: There is a block that needs to be deleted from the NeurIPS checklist section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp; \n\nWe thank the reviewer for the invaluable feedback. As noted by the reviewer, our idea is \"fresh\" and \"elegant,\" our algorithm is \"agnostic to the type of corruption,\" and our results \"outperform the prior state-of-the-art.\"  Below we provide more explanations and additional results to address each concern from the reviewer.\n\n&nbsp; \n\n**1. Comparison to baselines**\n\n&emsp;  Ambient Diffusion pioneered in learning clean diffusion models from masked images, achieving current STOA performance. While this insightful idea was originally designed for inpainting task, we feel it is more comprehensive to test it on denoising and deblurring tasks, using its original implementation with a naïve further corruption technique (randomly masking pixels for all tasks). We acknowledge that the original algorithm cannot be trivially extended to different settings. To avoid misinterpretation, we are open to labeling Ambient Diffusion's denoising and deblurring results as \"not applicable,\" as these tasks fall outside its original design scope. In addition, we also provided SURE-Score in our paper as a versatile baseline capable of handling various corruptions.\n\n&emsp; We agree that validation on different levels of corruption and other datasets would strengthen the understanding of our method. As suggested by the reviewer, we conducted additional experiments comparing the EM approach with Ambient Diffusion across different corruption levels (masking ratios of 0.4, 0.6, 0.8). These new results, presented in the table below and the attached PDF, demonstrate our method’s robustness and superior performance across diverse conditions. We will incorporate these findings as well as more experiments on new datasets (e.g., CelebA) in the revision.\n\n| Masking Ratio                     | 0.4   | 0.6   | 0.8   |\n| --------------------------------- | ----- | ----- | ----- |\n| AmbientDiffusion, FID$\\downarrow$ | 18.85 | 28.88 | 46.27 |\n| Ours, FID$\\downarrow$             | 13.75 | 21.08 | 45.24 |\n\n\n&nbsp; \n\n**2. Recent work \"Consistent Diffusion Meets Tweedie\"**\n\n&emsp; We thank the reviewer for pointing out this relevant ICML 2024 work, which cleverly finetunes Stable Diffusion (SD) to leverage the pre-trained knowledge in denoising tasks. We will add it in related work. However, as our method focuses on training diffusion models from scratch using corrupted observations, we feel the two works belong to different categories. A fairer comparison with the ICML work would involve applying an EM framework to finetune SD with noisy observations, which presents intriguing future research. We will explore it in future work.\n\n&nbsp; \n\n**3. The limitation of clean data initialization**\n\n&emsp; Our method relies on a small amount of clean data as a starting point, which we do acknowledge as a limitation, as discussed in the conclusion. However, we have minimized the requirement of the clean data to as few as 10 or 50 clean images (Fig. 5(a)), which we believe is a reasonable assumption for many scientific applications. For instance, in fluorescent microscopy imaging, high SNR images are limited due to long exposure times and potential phototoxicity rather than physical restrictions. Therefore, acquiring a small number of high-SNR images to aid training is feasible.\n\n&emsp; On top of that, we have explored alternative initialization strategies, including using out-of-distribution (OOD) clean images or classical pre-processing methods to initialize EM iterations (see Fig. 5 and Tbl. 4 in the paper). Our preliminary findings suggest that our EM approach can be initialized using low-frequency image statistics (e.g., smoothness) learned from OOD or pre-processed images. Investigating ways to eliminate the need for clean data would be an important future research direction. We will discuss more in the revision.\n\n&nbsp; \n\n**4.DPS does not offer true posterior samples**\n\n&emsp; We acknowledge that DPS-based methods may not always accurately sample the true posterior due to their simplified assumptions. Our key contribution is the EM approach itself, which provides flexibility in choosing the posterior sampling method in its E-step. For example,  it can replace DPS with more advanced posterior sampling techniques with stronger theoretical guarantees, such as plug-and-play Monte Carlo (PMC)[1], which ensures non-asymptotic stationarity. We will discuss it in the revision.\n\n&nbsp; \n\n**5. Training efficiency**\n\n&emsp; Although our method involves multiple training steps,  as discussed in Sec. 4.3, we avoid training diffusion models from scratch at each M-step. Instead, we fine-tune the model in most iterations, only training from scratch in the final 1-3 steps to prevent memorizing poor samples from the initial stages. This ensures rapid convergence in each iteration and significantly reduces training time. In practice, our method has a similar training time as Ambient Diffusion (ours is 2 days with 4 NVIDIA A800 GPUs).\n\n&nbsp; \n\n**6. Discussion on corruption types that are impossible to recover**\n\n&emsp;  We agree that for extremely corrupted images (e.g., >99% pixels masked), reconstructing the true distribution becomes challenging or even impossible due to the loss of spatial relationships among pixels. This is a common issue across all baselines. A potential solution could involve incorporating more prior knowledge, such as fine-tuning SD instead of training from scratch, similar to ICML 2024 work. We will discuss it in the revision.\n\n&nbsp; \n\n**7.Redundancy in the checklist section**\n\n&emsp;  Nice catch! We will correct it.\n\n&nbsp; \n\nWe thank the reviewer again for the valuable insights and suggestions. We hope our explanations have clarified all the concerns. We are happy to address any further questions or discussions the reviewer may have. Thank you!\n\n&nbsp; \n\n**Reference**:\n\n[1] Yu Sun, Zihui Wu, Yifan Chen, Berthy Feng, and Katherine L. Bouman. Provable Probabilistic Imaging using Score-based Generative Priors, arXiv 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DM6vF8oiBz", "reviewer": "Reviewer_HYFc", "summary": "Authors propose using the EM algorithm to train a diffusion model on corrupted data. To initialize the process, the proposed method requires access to a \"limited number of clean samples.\" The authors claim the method allows convergence to the true data distribution.", "questions": "* How do authors quantify \"minimal clean images\"? The authors mention\n  that sparse clean data provides a good initialization of the DM's\n  manifold. Why is this the case? What does \"good\" mean precisely in\n  this context?", "limitations": "* I disagree with the authors that needing access to a minimal number of\n  clean images is a minor limitation. AmbientDiffusion and some other prior\n  work discussed do not assume access to such a dataset, and hence the\n  comparisons might not be entirely fair.", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Originality.** The idea of using EM to train a diffusion model from corrupted data is specifically original; however, training generative models without access to clean data is a well-explored problem, even before the diffusion model era.\n\n**Quality and clarity.** The paper is easy to follow, and the problem is well-motivated.\n\n**Significance.** I do not find the proposed method in its current format significant, as it assumes access to a \"limited dataset of clean images\" without quantifying how small this set can be. Additionally, the paper claims convergence to the true data distribution without providing conditions on the forward operator and noise process, which is an unsubstantiated claim given the limited set of inverse problems explored.", "weaknesses": "* Claiming convergence to the true data distribution is a grand claim, and simply providing examples from relatively simple inverse problems does not substantiate this claim. Therefore, due to unsubstantiated claims and limited quantitative discussion on the required conditions for the forward operator and the number of clean data samples needed, I find this paper not truly usable in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp; \n\nWe thank the reviewer for the thoughtful comments. We believe there may have been some misunderstandings regarding our work, which we will clarify. Below we carefully address all the questions from the reviewer.\n\n&nbsp; \n\n**1. Quantifying the number of required clean images**\n\n&emsp;  We totally agree that quantifying the amount of required clean images is important and we have carefully investigated this problem in Section 5.4, Figure 5(a), and Table 4 of the main paper. Our experiments show that diffusion models trained on as few as 10 clean images provide adequate initialization. Moreover, to further reduce the need for clean data, we also explored using out-of-distribution (OOD) or pre-processed images for initialization, which achieved comparable results. We will highlight these findings more clearly in the revision.\n\n&nbsp; \n\n**2. Convergence claims & conditions for the forward operator**\n\n&emsp;  In our paper, convergence refers to reaching a local minimum of the optimization problem, as established by the properties of the classical EM algorithm and supported by previous theoretical studies [1]. We discuss this briefly in Section 3.3, lines 124-126, of the original paper. The forward operator and noise process do not affect this type of convergence but do influence the convergence speed by determining the amount of information in the corrupted observations. We will clarify the sentences and definitions about convergence in the final version.\n\n&nbsp; \n\n**3. Why sparse clean data provide “good” initialization**\n\n&emsp;  This question has been partially addressed in Section 4.2, lines 160-165, of the original paper. Sparse clean data helps the initial diffusion model learn common structures in natural images, such as continuity, smoothness, and object profiles. In this context, “good” means capturing important low-frequency statistics of natural images. From the EM algorithm perspective, “good” initialization means defining a local minimum solution close to the global minimum.\n\n&nbsp; \n\n**4. Limitation of requiring clean images**\n\n&emsp;  Considering the minimal number of clean images required for initialization and the alternative strategies we've validated for our EM framework (as explained in answer (1)), we respectfully disagree that this is a major limitation of our algorithm. Additionally, in many scientific applications, accessing some clean images is reasonable, as the lack of clean images is often due to cost rather than physical restrictions. For instance, in fluorescent microscopy, while high-SNR images are limited due to the need for long exposure times and the risk of phototoxicity, it is still feasible to obtain some high-SNR images to support the training of clean diffusion models from extensive low-SNR microscopy data.\n\n&nbsp; \n\nWe thank the reviewer once again for the valuable comments. We hope our explanations have clarified all the questions. We are happy to address any further concerns or discussions the reviewer may have. We kindly hope the reviewer could consider raising the score if satisfied with our responses. Thank you!\n\n&nbsp; \n\n**Reference**:\n\n[1] Dempster, A.P.; Laird, N.M.; Rubin, D.B. Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society, 1977", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pkruBZvJII", "reviewer": "Reviewer_Kirh", "summary": "This paper we proposes an expectation-maximization (EM) approach to train diffusion models from corrupted observations. The extensive experiments show the effectiveness of the proposed method.", "questions": "While you mention that only a small number of clean images are needed to initialize the EM iterations, does the number of clean images affect the final model performance in applications? Is it possible to completely eliminate the dependence on clean data?", "limitations": "The authors presented the limitations and proposed the future direction to addressed them.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The experiments are extensive.\n2. The proposed approach addresses a significant challenge in the field where large clean datasets are often unavailable.", "weaknesses": "1. The source code is not provided yet, but the authors promised to make it public in the future.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp; \n\nWe thank the reviewer for the insightful suggestions. We are happy that the reviewer recognizes our approach \"addresses a significant challenge in the field\", and our experiments are \"extensive\" and \"show the effectiveness of the proposed method\". We will release all the code and the checkpoint to the community to stimulate future research. Below we address each question of the reviewer.\n\n&nbsp; \n\n**1. Source code**\n\n&emsp; We are happy to provide a private repository of our code to address your concerns. Following NeurIPS author guidelines, \n\n> If you were asked by the reviewers to provide code, please send an anonymized link to the AC in a separate comment, make sure the code itself and all related files and file names are also completely anonymized.\n\n&emsp; we have sent a link to the AC for anonymity verification and will share it with you once approved.\n\n&nbsp; \n\n**2. Impact of number of clean images on model performance**\n\n&emsp;  Preliminary experiments in Section 5.4 and Figure 5(a) show that using 10 to 500 clean images results in similar posterior sampling outcomes after the first EM stage, indicating that the diffusion model successfully learns important image statistics with limited data. Since clean images initialize the diffusion model by providing low-frequency statistics, the model performance will not be significantly affected by their number.\n\n&nbsp; \n\n**3. Completely eliminating the dependence on clean data**\n\n&emsp; This is indeed an important research direction. In this paper, we have explored strategies to eliminate the dependence on clean data, such as initializing EM iterations with classical pre-processing methods, as detailed in Appendix B. Additionally, leveraging a generative foundation model like Stable Diffusion as the initial model could be a potential alternative solution, as suggested by other reviewers (Consistent Diffusion Meets Tweedie, ICML 2024). We will continue to explore this direction in future work.\n\n&nbsp; \n\nWe thank the reviewer once again for the insightful suggestions. We greatly appreciate the recognition and are more than willing to address any further questions or discussions the reviewer may have. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r4B2TLkjdj", "reviewer": "Reviewer_bEKL", "summary": "The authors present a new method for training diffusion models from corrupted data.\nThe presented method is an EM algorithm that in turn (i) uses the current model to sample clean images given the noisy input and then (ii) use these clean samples to train the diffusion model.\nThe authors show the results for data corrupted by noise, blurring and missing pixels.", "questions": "At the moment the degradation model is known and seems to have to involve Gaussian noise (line 149).\nHow does this work for the blurring operation, which seems to be deterministic? Or is the amount of added Gaussian noise just not visible.\nHow does it work for the in-painting task, which also does not seem to involve Gaussian noise?\n\n\nTypos and language:\n\nLine 103-104 \"we defines\"\nTable 4: \"The results show that EM-Score is insensitivity to initializations\"", "limitations": "Do the authors think that it could be possible to include other degradation models, such as a more realistic combination of Gaussian noise Poisson shot noise? \nWould it potentially be possible to learn the parameters of a degradation model. e.g. the blur kernel of the parameters of the noise distribution?\nA learnable degradation model would drastically increase the applicability in real life imaging applications.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This is a conceptual paper that does not focus on empirically optimised architecture details, but instead shows the applicability of a simple elegant new idea. This is a plus.\n* I can think of many applications in e.g. in scientific imaging (e.g. microscopy or astronomy) that suffer heavily from noise and degradation and for which clean data training is usually not available or very limited. So, I believe such methods can have a substantial practical impact.\n* The paper is well structured, with each building block (Score based diffusion models, posterior sampling, EM algorithm) being carefully introduced.", "weaknesses": "* Most importantly, I think the evaluation is not as convincing as it could be. All qualitative results seem to have artefacts to a degree. This also seems to be true for the baselines. E.g. the ambient diffusion results in Figure 3 suffer from some form of pixel artefacts. This is not the case when looking at the ambient diffusion paper (e.g. Figure 7), where not such artefacts are visible. Granted, this is a different dataset. However, I wonder if there is a problem with the training.\n\n* I think the fact that the method requires some (albeit little) clean data as a first training step is a weakness of the method.\n* My understanding is that the current system assumes Gaussian noise as part of the forward model (line 149). This can be limiting.\n* There are some typos and language problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp; \n\nWe thank the reviewer for the constructive suggestions. We are happy that the reviewer recognizes our idea as \"simple\" and \"elegant\", our method \"can have a substantial practical impact\", and our paper \"is well structured\". We have provided explanations and clarifications regarding all the concerns, as shown below:\n\n&nbsp; \n\n**1. Qualitative results with artifacts**\n\n&emsp; Thanks for your careful review! In Figure 3, we used the officially released Ambient Diffusion checkpoint for image posterior sampling rather than training the SOTA models ourselves. We double-checked our code and found no issues.\n\n&emsp;  However, training clean diffusion models from corrupted images is extremely challenging, and the missing information can cause qualitative results on some datasets to appear artefactual. The good performance in Figure 7 of the Ambient Diffusion paper leverages AFHQ, whereas our results report on CIFAR10. CIFAR10 images contain more high-frequency features than AFHQ images, making inverse problems more challenging and leading to more reconstruction artifacts. The original Ambient Diffusion paper also noted similar issues, stating that learning clean diffusion models on CIFAR10 is harder compared to CelebA-HQ and AFHQ (Sec. 5.1 in their paper) :\n> “for CelebA-HQ and AFHQ, we manage to maintain a decent FID score even with 90% of the pixels deleted. For CIFAR-10, the performance degrades faster, potentially because of the lower resolution of the training images”.\n\n&nbsp; \n\n**2. Dependence on clean data for initialization**\n\n&emsp;  Our method relies on a small amount of clean data as starting point, which we acknowledge as a limitation, as discussed in the conclusion. However, we have minimized this requirement to as few as 10 clean images (Fig. 5(a)). Additionally, we have explored alternative initialization strategies, including using out-of-distribution (OOD) clean images or classical pre-processing methods to initialize EM iterations (see Fig.5 and Tbl. 4 in the main paper). Our preliminary findings suggest that our EM approach can be initialized using low-frequency image statistics (e.g., smoothness) learned from OOD or pre-processed images. Investigating the elimination of the clean data requirement is an important research direction that we leave for future work. We will discuss this further in the revision.\n\n&nbsp; \n\n**3. Gaussian noise in forward models**\n\n&emsp;  All examples in the paper involve Gaussian noise in their degradation models, including inpainting and deblurring, as practical imaging problems always include measurement noise. However, the noise added to masked and blurry images is minimal (std=0.0001 and 0.02, respectively) and may not be visible. We also support different noise models, as discussed below.\n\n&nbsp; \n\n**4. Extending to other types of measurement noise**\n\n&emsp;  Our method can be extended to more realistic noise models. As discussed in the original DPS paper [1], it can handle various noise statistics, such as Gaussian and Poisson. Our EM framework has flexibility in replacing our E-step with DPS under new noise assumptions or even more advanced posterior sampling algorithms such as plug-and-play Monte Carlo (PMC)[2].\n\n&nbsp; \n\n**5. Learning parameters of degradation models**\n\n&emsp; Thank you for suggesting this important topic! Learning degradation models complements our method, which focuses on understanding the relationship between observations and underlying images rather than just image priors. This challenge can also be addressed using the EM framework, where the M-step iteratively estimates the degradation model’s parameters by incorporating the gradient of the data fidelity term with respect to these parameters. We will discuss it in the paper and continue to explore this exciting direction in the future.\n\n&nbsp; \n\nWe thank the reviewer once again for the insightful suggestions. We hope our explanations have clarified all the concerns. We are happy to address any further questions or discussions the reviewer may have. Thank you! \n\n&nbsp; \n\n**Reference**:\n\n[1] Hyungjin Chung, Jeongsol Kim, Michael T. Mccann, Marc L. Klasky, Jong Chul Ye. Diffusion Posterior Sampling for General Noisy Inverse Problems, ICLR 2023.\n\n[2] Yu Sun, Zihui Wu, Yifan Chen, Berthy Feng, and Katherine L. Bouman. Provable Probabilistic Imaging using Score-based Generative Priors, arXiv 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
&nbsp;

We thank all the reviewers for their professional and constructive feedback. We are encouraged by their recognition of our paper's technical importance and novelty (qpvz, Kirh, bEKL), broad applicability (qpvz, bEKL), impressive performance in extensive experiments (qpvz, Kirh), and high-quality writing (HYFc, bEKL).   We have summarized some of the major concerns raised by the reviewers below. Point-to-point responses are also included as a reply to each reviewer. Additionally, we have added a PDF file for further experimental results.    

&nbsp;

**1: Requirement for clean data to initialize the EM training**

&emsp; The introduction of clean images aids the diffusion model in converging to a local minimum close to the true image distribution. We clarify that as few as 10 images may suffice, and alternative initialization strategies also exist, as discussed in Section 5.4 and Appendix B of our original submission. We acknowledge that other strategies suggested by reviewers, such as initializing with generative foundation models (e.g., Stable Diffusion[1]), are promising and will discuss them in the revision. We also discuss scientific imaging applications (e.g., fluorescent microscopy) to explain that access to some clean images is reasonable, as their scarcity is often due to cost rather than physical restrictions.

&nbsp;

**2: More comparisons to baseline methods like Ambient Diffusion**

&emsp;  While our original submission included extensive experiments comparing our method with baselines, as suggested by the reviewers, we added additional experiments with different settings (e.g., various corruption levels) and further verification of our implementation to strength our conclusions. We have included these new results and the corresponding conclusions in the PDF as well as the point-by-point responses.

&nbsp;

**3: The algorithm’s extension to different corruption types, noise models, and unknown degradations**

&emsp;  The EM algorithm is a versatile framework that can be extended to various imaging inverse problems and can also learn unknown forward models or noise statistics. We will include more discussions on these aspects in our revision.

&nbsp;

Please let us know if these clarifications and additional results address your concerns. We are happy to discuss any remaining points during the discussion phase.

&nbsp;
&nbsp;

**Reference**:

[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022.

Meta-review: This paper received mixed ratings: Weak Accept, Strong Reject, Accept, Accept. The authors propose an expectation-maximization (EM) approach to train diffusion models from corrupted observations. AC agrees with some reviewers that the proposed is a simple yet elegant idea to alternate between reconstructing clean images from corrupted data using a known diffusion model (E-step) and refining diffusion model weights based on these reconstructions (M-step). Additional strengths highlighted by reviewers include the proposed method is agnostic to the type of corruption, and potentially applicable to many applications, e.g., in scientific imaging. AC agrees with Reviewer bEKL that "This is a conceptual paper that does not focus on empirically optimised architecture details, but instead shows the applicability of a simple elegant new idea." Reviewer HYFc raises a major concern that "claiming convergence to the true data distribution is a grand claim". The authors provided an acceptable response, and agreed to further clarify the sentences and definitions about convergence in the final version. AC recommends accepting the paper.
Decision: Accept (poster)