An Expectation-Maximization Algorithm for Training Clean Diffusion Models from Corrupted Observations

Authors: weimin  bai
yifei  wang
wenzheng  chen
he  sun
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 14:59:01

Abstract: Diffusion models excel in solving imaging inverse problems due to their ability to model complex image priors. However, their reliance on large, clean datasets for training limits their practical use where clean data is scarce. In this paper, we propose EMDiffusion, an expectation-maximization (EM) approach to train diffusion models from corrupted observations. Our method alternates between reconstructing clean images from corrupted data using a known diffusion model (E-step) and refining diffusion model weights based on these reconstructions (M-step). This iterative process leads the learned diffusion model to gradually converge to a local optimum, that is, to approximate the true clean data distribution. We validate our method through extensive experiments on diverse computational imaging tasks, including random inpainting, denoising, and deblurring, achieving new state-of-the-art performance.

Main paper: 
 introduction: 
 Diffusion models (DMs) (1; 2; 3) have demonstrated remarkable versatility in capturing complex real-world data distributions, excelling in diverse applications like image generation (4; 5; 6; 7; 8) , audio synthesis (9) , and molecular design (10) . DMs approximate distributions by learning their score functions-the gradient of the log-likelihood of the data distribution ∇ x log p data (x). This enables high-quality sample generation by simulating reverse-time stochastic differential equations (SDEs) (2) during inference.Recently, there has been growing interest in leveraging DMs as priors for computational imaging inverse problems (11; 12; 13; 14; 15; 16) , which aim to recover underlying images x from corrupted observations y. The Bayesian framework for computational imaging defines the posterior distribution of images x given observations y:EQUATIONwhere p(y | x) defines the forward model of observations and p(x) defines an image prior. DMs offer efficient, data-driven priors that outperform traditional handcrafted priors prone to oversimplification and human biases, such as sparsity (17) or total variation (TV) (18; 19) . However, a major limitation of DM-based solvers is their reliance on substantial volumes of highquality, clean signals for pre-training-a requirement often infeasible in real-world settings, especially Figure 1 : Overview of EMDiffusion. The paper proposes an expectation-maximization (EM) approach to jointly solve imaging inverse problems and train a diffusion model from corrupted observations. Left: In each E-step, we assume a known diffusion model and perform posterior sampling to reconstruct images from corrupted observations. In the M-step, we update the weights of the diffusion model based on these posterior samples. By iteratively alternating between these two steps, the diffusion model gradually learns the clean image distribution and generates high-quality posterior samples. Right: Raw observations and reconstructed clean images based on the diffusion model learned from corrupted data.for scientific and biomedical imaging. In contrast, corrupted noisy observations with differentiable forward models are easier to acquire, such as blurred images from mobile photography or 2D projections of 3D structures in X-ray computed tomography (CT) (20; 21) and cryogenic electron microscopy (cryo-EM) (22; 23) . Our paper seeks to answer a pivotal question: Can a DM be effectively trained to solve inverse problems primarily using large-scale corrupted observations? This presents a chicken-egg dilemma: training an accurate DM requires clean images, but reconstructing clean images from corrupted observations requires a good DM.Utilizing the Expectation-Maximization (EM) framework, we introduce a novel approach called EMDiffusion. This approach initializes with a diffusion prior trained on a minimal set of clean images, then alternates between two steps across multiple iterations: reconstructing clean images from corrupted observations using the current diffusion prior (E-step), and refining the DM parameters based on these reconstructions (M-step). The sparse clean data provides a good initialization of the DM's manifold, preventing collapse into a distorted or biased distribution characterized solely by corrupted inputs. Each E-M iteration leverages the current diffusion prior to generate cleaner reconstructions from the corrupted data, and these enhanced reconstructions then update the DM, providing an improved prior for the next iteration. This cycle continues, with the generated samples and DM progressively converging toward local optima, which equals to approximate the true clean data distribution. The forward operator and noise process do not affect this type of convergence but only influence the convergence speed by determining the amount of information in the corrupted observations. We validate the generalizability and effectiveness of EMDiffusion through extensive experiments, applying it to diverse imaging inverse problems across various datasets, including random inpainting, denoising, and deblurring, and achieving compelling results. 
related works: 
 Inverse problems in computational imaging. Computational imaging aims to reconstruct underlying signals x ∈ R d from corrupted observations y ∈ R m , where the image formation process is probabilistically modeled as:EQUATIONSince m ≤ d and observation noise is inevitable, inverse problems in computational imaging are ill-posed, with the inverse mapping y → x being one-to-many. To address this complexity, Bayesian inference introduces a prior distribution of underlying images, p(x), to constrain the solution space for the image posterior, p(x|y), as illustrated by Eq. 1. Employing Maximum a Posteriori (MAP) estimation, one can derive a point estimate of the underlying image by maximizing log p(x|y).Alternatively, posterior image samples of reconstructed images can be obtained through methods like Markov Chain Monte Carlo (MCMC) (24) or Variational Inference (VI) (25; 26; 27) . However, the performance of many computational imaging solvers is limited by their reliance on oversimplified, handcrafted priors such as sparsity and total variation (TV). These priors fail to capture the true complexity of natural image distributions, hindering the solvers' ability to achieve high-quality reconstructions.Diffusion models for inverse problems. Diffusion models (DMs) (1; 2; 3) have recently emerged as powerful data-driven priors for solving imaging inverse problems. By mastering the intricate distribution of images through training on extensive image data, DMs facilitate both point estimates via Plug-and-Play (PnP) optimization (28; 29) and posterior sampling through generative PnP (GPnP) (30) , PnP Monte Carlo (PMC) (31) , or Diffusion Posterior Sampling (DPS) (13; 14; 32) . These approaches have demonstrated remarkable efficacy in addressing a broad spectrum of noisy inverse problems, with applications spanning diverse fields, including astronomy (11; 33) and biomedical imaging (15; 34) .Learn diffusion models from corrupted data. In many real-world scenarios, acquiring large-scale clean data is costly or infeasible, motivating efforts to learn DMs directly from corrupted data. Data corruptions stem from under-determined forward models (e.g., 2D projections, inpainting, compressed sensing) and measurement noise. Recent studies have explored various strategies to address these challenges. For instance, in inverse graphics, researchers integrate the forward model into the diffusion process and introduce a view-consistency loss over multiple noiseless projections of the same object to learn a 3D DM from 2D images (35; 36) . In image inpainting, AmbientDiffusion (37) randomly masks additional pixels and forces the DM to restore these deliberate corruptions. Since the model cannot distinguish between original and further corruptions, it effectively learns the uncorrupted image distribution. However, the AmbientDiffusion is limited by the additional masking technique and fails to achieve good performance with noisy observations. ( 38) cleverly finetunes Stable Diffusion (SD) to leverage the pre-trained knowledge in denoising tasks, but does not support training a DM from scratch. Meanwhile, SURE-Score (39) proposes to jointly learn an image denoiser and a score-based DM using Stein's unbiased risk estimate (SURE) loss, where the SURE loss acts as an implicit regularizer on the model weights. Despite its innovative approach, SURE-Score often struggles with significant data corruption, such as inpainting tasks with a large fraction of missing pixels, and tends to produce overly smooth results. A general approach for learning DMs from arbitrarily corrupted data remains an open challenge. 
score-based diffusion models: 
 A diffusion model captures the data distribution by learning a score function, i.e. the gradient of the logarithm of the likelihood of data distribution ▽ x log p data (x). Consequently, a diffusion model generates samples by gradually removing noise from a random input, which is equivalent to a reverse-time stochastic differential equation (SDE) -the solution to a forward-time SDE that gradually injects noise, forward-time SDE:EQUATIONwhere t ∈ [0, T ], f (x t , t) : R d → R d isMany images are consistent with a single observation due to the ill-posed nature of the image formation model. By combining the forward model with the diffusion prior using Bayes' rule, we define a conditional diffusion process that samples the posterior distributionEQUATIONwhere the conditional score function can be further decomposed as:EQUATIONSince the likelihood function is only defined for t = 0, the dependence between y and x t is implicit, making ∇ xt log p t (y | x t ) an intractable integral at each diffusion step. Various techniques have been proposed to address this intractable likelihood function, including exactly computing the probability using an ODE flow (11) , bounding the probability through an evidence lower bound (ELBO) (33) , and approximating the probability using Tweedie's formula (13; 40; 41; 42) . To ensure computational efficiency, we adopt the approximation proposed in (13) ,EQUATIONfor diffusion posterior sampling in all the following sections.The Expectation-Maximization (EM) algorithm (43; 44) is an iterative technique for estimating parameters in statistical models involving latent variables. When the true values of the latent variables are unknown, maximum likelihood estimation (MLE) cannot be directly applied to identify the model parameters. Instead, the EM algorithm maximizes a lower bound of the log-likelihood function, derived using Jensen's inequality:EQUATIONwhere x, y, and θ denote the latent variables, observations, and model parameters, respectively. The algorithm alternates between two steps:• Expectation step (E-step): Sample latent variables from the current estimate of the conditional distribution, x ∼ p θ (x | y), and compute the expected log-likelihood lower bound L(θ).• Maximization step (M-step): Maximize L(θ) = E x∼p θ (x|y) [log p θ (x)] to update parameters θ.This iterative procedure allows the EM algorithm to converge to a local maximum of the observed data log-likelihood, making it a powerful technique for estimation problems involving latent variables, such as Gaussian mixture clustering(45), and dynamical system identification (46) . 
proposed method: 
 Given corrupted observations y and a known forward model p(y | x), learning DMs from corrupted data is a parameter estimation problem involving latent variables. The latent variables are the underlying clean images x, and the goal is to estimate the DM parameters θ that govern the image prior p θ (x). Consequently, we can leverage an iterative EM approach to reconstruct clean images and train the DM using corrupted data jointly, as described in Fig. 1 and Algorithm 1.The Expectation-Maximization (EM) algorithm needs a good initialization to begin its iterative process, as an improper initialization can result in convergence at an incorrect local minimum. While obtaining a large dataset of clean images is difficult, a small set of clean data is often available. This limited clean data can be used to train an initial DM to start the EM iterations. For example, in all The diffusion prior is pre-trained using the 50 clean images shown in (e). When λ is small, there is obvious mode collapse, and all posterior samples come from the training set of 50 clean images, unrelated to the observations. As λ increases, the data likelihood gains more significance, resulting in reconstructed images that are more consistent with the inpainting observations.the following experiments, 50 randomly selected clean images were used to train the initial DM, serving as the starting point for the EM algorithm. As demonstrated in Sec. 5.4, clean images do not need to be from the same dataset; those from out-of-distribution datasets also serve as reasonable initializations.In the E-step, we assume a known diffusion prior and reconstruct the underlying clean images through diffusion posterior sampling. We adopt the standard variance-preserving form of the stochastic differential equation (VP-SDE) (2), which is equivalent to the Denoising Diffusion Probabilistic Models (DDPM) (1) . The drift function f (x t , t) takes the form β(t)x t /2, and the diffusion rate g(t) is β(t). Therefore, the reverse diffusion sampler in Eq. 4 can be represented as:EQUATIONConsidering a known imaging forward model, A, and additive Gaussian noise, p(y | x) ∼ N (y | A(x), σ 2 I), the conditional score function can be represented as:EQUATIONwhereEQUATIONHowever, a naive diffusion posterior sampling approach using Eqs. 8, 9, and 10 often fails to produce high-quality reconstructions. This is because the learned DM is inaccurate during the early EM iterations. We demonstrate this issue with a toy experiment. We performed diffusion posterior sampling (DPS) on randomly masked observations, as shown in Fig. 2 (y, f ) ∼ Dataset(Y, A)4:x T ∼ N (0, I)5:for t = T to 1 do6: ᾱt = t s=1 (1 -β t ) 7: x0 ← 1 √ ᾱt x (i) t + (1 -ᾱt ) s θ (x (i) t , t) 8:z ∼ N (0, I)9:Take reverse-time SDE step on {Sampling in Sec. 4.2}x(i) t-1 ← x (i) t + β(t) x (i) t 2 + s θ (x (i) t , t) -λ 2σ 2 ∇ x (i) t ∥y -f (x 0 )∥ 2 2 + β(t)z 10:end for11:for m = 0 to M do 12:x data ∼ Shuffle(x(i) 0 ∼ p θ (x (i) 0 | y (i) )) 13:t ∼ Uniform({1, . . . , T })14: ᾱt = t s=1 (1 -β t ) 15:ϵ ∼ N (0, I)16:Take gradient descent step on {Optimization in Sec. 4.3}EQUATIONend for 18: end for It does not mean that these low-quality DMs cannot provide any prior information. Although the prior is poor in the early training stages, it has learned common features and structures shared among natural images, such as the continuity and smoothness of natural images and profiles of specific object types. By introducing a hyper-parameter λ to rescale the likelihood term and avoid mode collapse, we find that the low-quality DM can also act as a weak prior for posterior sampling, where the reverse-time SDE can be written as:EQUATIONThe hyper-parameter λ efficiently balances the diffusion prior and the data likelihood, resulting in reliable reconstructed images even when the prior is poor. As demonstrated in Fig. 2 (b), (c), and (d), as λ increases from 1 to 20, the data likelihood term gains more emphasis, making the reconstructed images more consistent with the inpainting observations. The choice of the hyper-parameter λ is automated in each E-step by finding the value that minimizes the data loss,EQUATIONwhere x0,λ represents the diffusion posterior samples of reconstructed images with λ scaling.During the M-step, we update the weights of the score-based models using the posterior samples obtained in the E-step. This resembles training a standard clean DM, s θ , to approximate the timedependent score function, ∇ xt log p(x t | x0 ), through denoising score matching (47) : In each image, 60% of the pixels are masked. As the EM iterations progress, the diffusion model learns cleaner prior distributions, improving the quality of posterior samples. Our method significantly outperforms the baselines, SURE-Score and AmbientDiffusion, achieving reconstruction quality comparable to DPS with a clean prior.EQUATIONTo accelerate training, especially during the early stages when the posterior samples are noisy, the M-step does not always train the score function s θ (•) from scratch. In the initial M-steps, we inherit the DM weights from the previous iteration and fine-tune them only using posterior samples from a subset of observations (e.g., randomly select 10% of total observations). However, once the quality of reconstructed images improves sufficiently, we reinitialize the DM weights and retrain the model with 100% data for a few more iterations. The training strategy transitions when the optimal balancing parameter, λ * , falls below 1, or fails to decrease for more than three consecutive iterations. 
experiments: 
 In this section, we demonstrate the performance of our method in learning DMs from corrupted data and solving inverse problems using these models. We validate the method on three imaging tasks: random inpainting, denoising, and deblurring. Our main results are presented in Fig. 3 , Fig. 4 , and Table 1 , with additional ablation studies in Fig. 5 . Further details on neural network architectures, training settings, and additional reconstruction and generation samples are provided in the appendix.The experiments are conducted on the CIFAR-10 (48) and CelebA (49) datasets at resolutions of 32 × 32 and 64 × 64, respectively. CIFAR-10 consists of 50,000 images across 10 classes for training, while CelebA contains 30,000 images of human faces. At each iteration, 5,000 corrupted images are randomly chosen for posterior sampling and training, and 250 corrupted images from the test set are chosen for evaluation.We evaluate the performance of our method using two groups of metrics. First, we compute the peak signal-to-noise-ratio (PSNR) and learned Perceptual Image Patch Similarity (LPIPS) scores between the reconstructed and ground-truth images, quantifying the accuracy of inverse imaging using learned DMs. Additionally, we compute the Fréchet Inception Distance (FID) between the learned DMs and reserved test data to assess their image generation quality.We compare our method with three related baselines: AmbientDiffusion (37), SURE-Score (39), and DPS with clean prior (13) . AmbientDiffusion and SURE-Score have similar settings to our method, which do not require DMs pre-trained on large-scale clean signals. Considering AmbientDiffusion is well-designed for masked observations, we only use it as the baseline of the image inpainting task.On the other hand, DPS leverages a pre-trained clean diffusion prior for posterior sampling, so it defines the performance upper bound for our method.In our experiments, we randomly select 50 clean images from each dataset to train the initial DMs for the EM iterations. AmbientDiffusion is trained with the standard setting in (37) . The key hyperparameter of SURE-Score, σ ω , is set to the observation noise's standard deviation (0.2 for denoising, Image inpainting. We conduct random inpainting (with mask probability p = 0.6) on CIFAR-10.As shown in Fig. 3 and Table 1 , our method significantly outperforms AmbientDiffusion and SURE-Score, achieving reconstruction quality similar to DPS with a prior trained on the clean CIFAR-10 dataset. The iterative training process is also illustrated in Fig. 3 . Initially, our method performs poorly with the DM trained on only 50 randomly selected clean samples. However, as the E-step and M-step alternate iteratively, the quality of posterior sampling improves. Large-scale posterior samples enrich the priors, leading to enhanced performance at each stage.Image denoising. We perform image denoising on CIFAR-10 with Gaussian noise n ∼ N (0, σ 2 I) and σ = 0.2. The results are shown in Fig. 4 (a) and Table 1 . Our method outperforms SURE-Score, and the self-supervised denoising benchmark, Noise2Self (50) , though it slightly lags behind DPS with clean priors. However, while our method's reconstructions may appear noisier than DPS results, they sometimes reproduce more details, such as the car wheels in the second row and the cat face in the third row of Fig. 4 (a), showcasing the better diversity of our learned DMs.Image deblurring. We validate image deblurring on CelebA using a Gaussian blur kernel with a size of 9 × 9 and a standard deviation of σ = 2 pixels. The results are shown in Fig. 4 (b) and Table 1 . As with the other tasks, our method significantly outperforms SURE-Score in solving imaging inverse problems, recovering fine details of human faces. However, the FID score of our learned diffusion models lags behind the original blurred observations. This is primarily because the FID score measures image similarity mainly through smooth features, making it a less effective metric for deblurring tasks. Comparing the results of all three tasks, we find that AmbientDiffusion only works well for inpainting because its additional masking technique is specifically designed for that purpose. SURE-Score consistently produces over-smoothed results because the SURE loss regularizes the gradient of generated images. As a comparison, our method does not make any special assumptions and provides a general framework applicable to all three tasks. The generation results are in Appendix D.Number of clean images for training initial DMs. Our EM approach starts with DMs trained on a small set of clean images. Fig. 5 (a) shows the PSNR of posterior samples generated by these models in the first E-step, allowing us to evaluate the impact of the number of clean training images on the performance of the initial DMs. Remarkably, DMs trained on as few as 10 clean images (0.02% of the corrupted images) can still act as reasonable priors. For inpainting and denoising tasks, DMs trained on 10 clean images provide nearly the same reconstruction quality as those trained on 500 clean images, as these tasks primarily require priors for low-frequency features, and 10 images suffice for an initial guess. However, the deblurring task benefits from DMs trained on more clean data since deblurring aims to recover high-frequency details where more data helps.Surprisingly, we find that DMs trained on clean images from CelebA (downsampled to 32×32) can also be used to initialize tasks on CIFAR-10. For inpainting and denoising tasks, DMs initialized with 50 clean images from an out-of-distribution (OOD) dataset (CelebA) achieve similar performance to those initialized with in-distribution (ID) data (CIFAR-10). For the deblurring task, DMs trained on OOD data perform better than those trained on a similar amount of ID data, suggesting that OOD data can sometimes serve as stronger priors for guessing high-frequency information.Learned priors through iterative training. Fig. 5 (b) shows the FID scores of the learned DMs in the inpainting task with 50,000 corrupted CIFAR-10 images after each EM stage. The generation ability of the DMs gradually improves as the EM iterations progress. As explained in Sec. 4.3, initially the DM inherits weights from previous steps for fast training and converges at the sixth iteration. After resetting the DM, the training resumes for three more rounds and finally converges to a FID score of approximately 21.08, significantly better than AmbientDiffusion's 28.88, setting a new state-of-the-art. Notably, our method achieves this performance using a DM architecture with far fewer parameters: our method employs a vanilla DDPM with 35.7 million parameters, while AmbientDiffusion uses an improved DDPM++ architecture with over 100 million parameters. Additionally, we compared our method with DMs trained on different amounts of clean data. Our model, learned from 50,000 corrupted images (60% pixels masked) using EM, performs better than a DM trained on around 15,000 clean images.Scaling factor in adaptive diffusion posterior sampling. Fig. 5 (c) shows the PSNR of reconstructed images at each EM stage with different scaling factors, using the inpainting task on CIFAR-10 as an example. We observe that the quality of posterior sampling initially improves and then de-teriorates as the scaling factor increases, confirming the existence of an optimal scaling factor as suggested in Eq. 12. As the EM stages progress, the optimal scaling factor decreases, indicating that the learned priors progressively improve through the EM iterations. This observation justifies the need for adaptive scaling factors in our method.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provide code in supp.Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: Our research explores computational imaging algorithms in which we didn't see any ethics issues.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
conclusion: 
 In this paper, we proposed EMDiffusion, a novel expectation-maximization (EM) framework for training diffusion models primarily from corrupted observations. The key assumption is that it is information-theoretically possible to learn the underlying distribution from measurements. Our method demonstrated state-of-the-art performance in image inpainting, denoising, and deblurring across various datasets. Additionally, an important finding is that a small amount of clean, indistribution data can act as an implicit regularizer, aiding the training of diffusion models from corrupted observations. Future work will aim to 1) extend initialization approaches, potentially by incorporating foundation models or traditional machine learning techniques, such as using preprocessed images from unsupervised inpainting, deblurring, or denoising for initialization, and 2) extend to various imaging inverse problems and learning unknown forward models or noise statistics.We provide the implementation details and more results in the appendix. We first describe our network architecture and training settings in Sec. A, then show initialization details in Sec. B. More results are also provided in Sec. C and Sec. D.A Implementation Details. Noise schedule. We leverage the default settings on VP-SDE ( 2), which uses a linear schedule with timesteps T = 1000, β 1 = 1e -4, and β T = 0.02.Exponential moving average. To stabilize the training process and reduce the color shift of samples generated by trained DMs, we adopt an exponential moving average (EMA) technique with a decay factor of 0.999 for all experiments.Optimizer. We apply AdamW (53) and set the learning rate to 2e -4 for CIFAR-10 and 2e -5 for CelebA.Hyperparameters for the training process. We set the batch size to 512 for CIFAR-10 and 64 for CelebA. We set the dropout rate to 0.1 for CIFAR-10 and 0 for CelebA. As for the learning rate, we adapt 1e -4 for CIFAR-10 and 2e -5 for CelebA, for a larger learning rate will result in unstable training.Dataset argumentation. We only use random horizontal flips to CIFAR-10 during training to achieve better performance. We did not flip CelebA, as the distribution of human faces is quite simple. We provide a table of training hyperparameters in Table 2 . 3 .Training schedule. At each EM iteration, we randomly choose 5,000 corrupted observations for diffusion posterior sampling and then train DMs. We further divide the iterations into two phases:• Phase 1 -Resume training DMs: at the early EM iterations, we inherit weights of DMs from the last iteration for quick convergence. For CIFAR-10, this phase lasts for about 6-8 EM iterations, while for CelebA deblurring, this phase increases to 10 EM iterations. The key insight is that DMs from Phase 1 always have a memory of bad posterior samples, which has a negative effect on the learned distribution.For CIFAR-10, this phase lasts for 3 EM iterations, we found it significantly improves the FID score of DMs. While for CelebA deblurring, this phase lasts for 2 EM iterations until we find the improvement is not obvious. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification:Our research explores computational imaging algorithms in which we didn't see any negative social impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We study computational imaging algorithms and didn't see such risks.Answer: [NA] Justification: We didn't do crowd sourcing with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To verify the sensitivity of EMDiffusion's initial DM training data, we provide more quantitative and qualitative results in this section, as shown in Table 4 . We draw the conclusion that EMDiffusion is not sensitive to the initial data. Apart from evaluating different numbers of in-distribution (ID) images and out-of-distribution (OOD) images for training the initial DMs, we also test the initialization on preprocessed observations, and find all of them converge similarly. Specifically, we preprocess the noisy observations with BM3D (58), the blurry observations with Fast TV Constraint (59) , and leave the masked observations unchanged. DMs initialized on these preprocess samples also perform well on the following E-step. As shown in Table 5 , we include additional comparisons to Ambient Diffusion (37) across various corruption conditions and datasets. The performance gap between the proposed method and Ambient Diffusion narrows under higher levels of corruption, likely due to the simpler DDPM architecture we employ. In contrast, Ambient Diffusion utilizes the improved DDPM++ architecture (54) , which is specifically modified to perform better for high-corruption inpainting-type problems. Nonetheless, the overall performance demonstrates the effectiveness of the proposed method, as we do not focus on empirically optimized architecture details, but instead show the applicability of the new EM idea for training DMs from corrupted observations.EMDiffusion is proposed to learn clean distributions from corrupted observations. In Sec. 5, we present detailed posterior sampling results and FID scores of learned DMs.Our model outperforms baselines by a significant margin in three inverse imaging tasks on two datasets. Though the FID score of our model trained on blurry CelebA is slightly higher than Ambient Diffusion, we argue that FID scores are easily influenced by sharp artifacts introduced by DPS ( 13), which is adopted in our E-steps. However, the distributions EMDiffusion learned from various types of corrupted observations are obviously better than baselines, as shown in Fig. 7 ,6,8.Future work. To achieve a better posterior distribution through the proposed EM framework, an accurate and efficient E-step plays a key role. We adopt the Diffusion Posterior Sampling (13) that could potentially introduce artifacts due to its approximation of the underlying data likelihood term. Therefore, a better FID score could be achieved by designing a principled posterior sampling method.(a) SURE-Score, FID=191.96 (c) Ours, FID=91.89 • If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: This paper received mixed ratings: Weak Accept, Strong Reject, Accept, Accept. The authors propose an expectation-maximization (EM) approach to train diffusion models from corrupted observations. AC agrees with some reviewers that the proposed is a simple yet elegant idea to alternate between reconstructing clean images from corrupted data using a known diffusion model (E-step) and refining diffusion model weights based on these reconstructions (M-step). Additional strengths highlighted by reviewers include the proposed method is agnostic to the type of corruption, and potentially applicable to many applications, e.g., in scientific imaging. AC agrees with Reviewer bEKL that "This is a conceptual paper that does not focus on empirically optimised architecture details, but instead shows the applicability of a simple elegant new idea." Reviewer HYFc raises a major concern that "claiming convergence to the true data distribution is a grand claim". The authors provided an acceptable response, and agreed to further clarify the sentences and definitions about convergence in the final version. AC recommends accepting the paper.

Reviews: review: paper_summary: paper_summary: The authors introduce a new framework for training diffusion models from corrupted data. Prior work on this research topic is based on the Ambient Diffusion framework or Stein's Unbiased Risk Estimate (SURE) idea. The authors propose an alternative methodology based on the Expectation-Maximization algorithm. The algorithm is tested on standard datasets and leads to improved performance over the considered baselines.
strength_weakness: strength_weakness: 1. The comparison to the baselines is somewhat limited/unfair. As the authors mention, Ambient Diffusion has been designed for inpainting. The authors use Ambient Diffusion for denoising and Deblurring. It is not clear how they do this since the original algorithm cannot be trivially extended to these settings. Even for the case of inpainting, the authors could report results for other levels of corruption as well (apart from $p=0.6$) and for other datasets apart from CIFAR-10. Regarding the comparisons with Ambient Diffusion, the only valid data point right now is for $p=0.6$. Providing a more comprehensive analysis would help the readers understand in what regime the proposed method works the best.
2. Specifically for the case of denoising, the authors might want to provide comparisons with the more recent work Consistent Diffusion Meets Tweedie which extends the Ambient Diffusion idea to the setting of noisy samples. 
3. For some scientific applications, there is no available clean data at all. The proposed algorithm needs some clean data to initialize the EM and this could be limiting for such applications.
4. One issue with the proposed method is that it relies on solving inverse problems with diffusion models. This problem is computationally intractable and for that reason, there has been a large body of research papers proposing approximations to the conditional score. The authors rely on such an approximation (DPS) which is known that doesn't truly offer samples from the posterior unless the data distribution is very simple (e.g. Gaussian).
5. One issue with this method seems to be the training speed. Even for converge to a local optimum, EM typically requires many steps. Each step here is a new training of a diffusion model. Prior works on learning diffusion models from corrupted data only required a single training.
6. For some corruption types, it is impossible to ever reconstruct the true distribution from noisy observations. The authors do not discuss how the proposed algorithm would work in such cases.
7. Minor: There is a block that needs to be deleted from the NeurIPS checklist section.
questions: questions: See the Weaknesses Section above. I am willing to further increase my score if my concerns are properly addressed.
limitations: limitations: The authors have adequately addressed the limitations of their method.
paper_summary: paper_summary: Authors propose using the EM algorithm to train a diffusion model on corrupted data. To initialize the process, the proposed method requires access to a "limited number of clean samples." The authors claim the method allows convergence to the true data distribution.
strength_weakness: strength_weakness: * Claiming convergence to the true data distribution is a grand claim, and simply providing examples from relatively simple inverse problems does not substantiate this claim. Therefore, due to unsubstantiated claims and limited quantitative discussion on the required conditions for the forward operator and the number of clean data samples needed, I find this paper not truly usable in practice.
questions: questions: * How do authors quantify "minimal clean images"? The authors mention
  that sparse clean data provides a good initialization of the DM's
  manifold. Why is this the case? What does "good" mean precisely in
  this context?
limitations: limitations: * I disagree with the authors that needing access to a minimal number of
  clean images is a minor limitation. AmbientDiffusion and some other prior
  work discussed do not assume access to such a dataset, and hence the
  comparisons might not be entirely fair.
paper_summary: paper_summary: This paper we proposes an expectation-maximization (EM) approach to train diffusion models from corrupted observations. The extensive experiments show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The source code is not provided yet, but the authors promised to make it public in the future.
questions: questions: While you mention that only a small number of clean images are needed to initialize the EM iterations, does the number of clean images affect the final model performance in applications? Is it possible to completely eliminate the dependence on clean data?
limitations: limitations: The authors presented the limitations and proposed the future direction to addressed them.
paper_summary: paper_summary: The authors present a new method for training diffusion models from corrupted data.
The presented method is an EM algorithm that in turn (i) uses the current model to sample clean images given the noisy input and then (ii) use these clean samples to train the diffusion model.
The authors show the results for data corrupted by noise, blurring and missing pixels.
strength_weakness: strength_weakness: * Most importantly, I think the evaluation is not as convincing as it could be. All qualitative results seem to have artefacts to a degree. This also seems to be true for the baselines. E.g. the ambient diffusion results in Figure 3 suffer from some form of pixel artefacts. This is not the case when looking at the ambient diffusion paper (e.g. Figure 7), where not such artefacts are visible. Granted, this is a different dataset. However, I wonder if there is a problem with the training.

* I think the fact that the method requires some (albeit little) clean data as a first training step is a weakness of the method.
* My understanding is that the current system assumes Gaussian noise as part of the forward model (line 149). This can be limiting.
* There are some typos and language problems.
questions: questions: At the moment the degradation model is known and seems to have to involve Gaussian noise (line 149).
How does this work for the blurring operation, which seems to be deterministic? Or is the amount of added Gaussian noise just not visible.
How does it work for the in-painting task, which also does not seem to involve Gaussian noise?


Typos and language:

Line 103-104 "we defines"
Table 4: "The results show that EM-Score is insensitivity to initializations"
limitations: limitations: Do the authors think that it could be possible to include other degradation models, such as a more realistic combination of Gaussian noise Poisson shot noise? 
Would it potentially be possible to learn the parameters of a degradation model. e.g. the blur kernel of the parameters of the noise distribution?
A learnable degradation model would drastically increase the applicability in real life imaging applications.

Comments: title: Thank you for your reply.
Rebuttal acknowledgement
My concern is resolved
comment: Thank you so much for your response! We appreciate your time and consideration.
&nbsp;

We thank the reviewer for the additional comments. We are pleased that many of your concerns have been addressed. Below, we provide additional clarification regarding the convergence claim and the conditions of the forward operator.

&nbsp;

**1. Convergence Claim**

&emsp; As explained in our rebuttal, the convergence we refer to is the local convergence inherited from the EM framework. We explicitly mentioned this in our original paper (lines 124-126). This claim is further supported by our extensive experimental results across various imaging tasks (see Appendix C). We promise to rephrase our abstract and add further clarifications on this point in the revised manuscript to avoid any confusion.

&nbsp;

**2. Forward Operator**

&emsp; In our previous responses, we misunderstood the reviewer's comments as asking us to discuss the effects of various corruption types (e.g., inpainting, denoising, deblurring) on the method’s effectiveness. We fully agree that there are certain cases where recovering a clean distribution from corrupted data is information-theoretically impossible, such as when the forward operator matrix is entirely composed of zeros. We will include additional discussions on these scenarios in the revised manuscript.

&nbsp;

We hope our explanations address all the concerns of the reviewer. We would greatly appreciate it if the reviewer could reconsider the evaluation of our paper. Thank you!
I thank the authors for their response. I will maintain my score because of one main reason: the claims made in the paper are too general and misleading, e.g., "This iterative process leads 8 the learned diffusion model to gradually converge to the true clean data distribution" (from abstract). In the rebuttal, the authors state that "In our paper, convergence refers to reaching a local minimum of the optimization problem." I do not get that impression from the paper. In addition, the authors, in response to my concern regarding conditions on the inverse problem's forward operator, state that "The forward operator and noise process do not affect this type of convergence but do influence the convergence speed by determining the amount of information in the corrupted observations." I strongly disagree: the forward operator could be an all-zeros matrix. You cannot recover the true data distribution using observations like that.
I don't have any additional questions.
We are very happy that the concerns are addressed!  Thank you once again for your valuable feedback!
We sincerely thank the reviewer for the additional comments. We will incorporate all your invaluable suggestions in our revision. Specifically, we will:

1） Remove AmbientDiffusion’s results on deblurring and denoising.

2） Include experiments on more corruption conditions and datasets.

3） Add discussion on the cases where learning clean diffusion from corrupted data is theoretically impossible.

Thank you once again for your valuable feedback!
I would like to thank the authors for their time and their efforts during the rebuttal.

I believe that the experiments with Ambient Diffusion for tasks beyond inpainting should be completely removed. Extending this method to corruptions beyond random inpainting should be done in a way such that Theorem 4.1. of the paper is satisfied. The current mechanism for further corruption is not valid and hence the results are not valid. I would like to ask the authors to remove the results for such tasks to avoid confusing the readers.

I would like to thank the authors for providing additional comparisons to Ambient Diffusion. I would urge the authors to provide more points and more datasets. This is crucial: different proposed methods should be evaluated in the same setting so that future work can be compared on a common benchmark. From the current results, it looks like the gap between the proposed method and Ambient Diffusion is closing for higher corruption. Results for higher corruption should be provided as well. In my opinion, it is absolutely fine if the proposed method even underperforms a baseline in some settings. The contribution is significant already.

Regarding the ICML work, it provides a general algorithm for training diffusion models from noisy observations than can be applied to training from scratch as well. In any case, this is concurrent work and I agree with the authors that direct comparisons are not needed.

Regarding discussion for other corruption types: this is not what I meant. I was referring to cases where it is information-theoretically impossible to learn the underlying distribution from measurements, e.g. you can't learn the distribution of human faces from a set of training images where the eyes are always missing. 

Overall, I believe this paper has some really nice aspects and hence I will keep my positive score of 6.
Thanks for your reply. My concern is resolved.