Title: An Expectation-Maximization Algorithm for Training Clean Diffusion Models from Corrupted Observations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion model

Inverse Problem

Computational Imaging
Abstract: Diffusion models excel in solving imaging inverse problems due to their ability to model complex image priors. However, their reliance on large, clean datasets for training limits their practical use where clean data is scarce. In this paper, we propose EMDiffusion, an expectation-maximization (EM) approach to train diffusion models from corrupted observations. Our method alternates between reconstructing clean images from corrupted data using a known diffusion model (E-step) and refining diffusion model weights based on these reconstructions (M-step). This iterative process leads the learned diffusion model to gradually converge to a local optimum, that is, to approximate the true clean data distribution. We validate our method through extensive experiments on diverse computational imaging tasks, including random inpainting, denoising, and deblurring, achieving new state-of-the-art performance.

An Expectation-Maximization Algorithm for Training Clean Diffusion Models from Corrupted Observations

Abstract

Diffusion models excel in solving imaging inverse problems due to their ability to model complex image priors. However, their reliance on large, clean datasets for training limits their practical use where clean data is scarce. In this paper, we propose EMDiffusion, an expectation-maximization (EM) approach to train diffusion models from corrupted observations. Our method alternates between reconstructing clean images from corrupted data using a known diffusion model (Estep) and refining diffusion model weights based on these reconstructions (M-step). This iterative process leads the learned diffusion model to gradually converge to a local optimum, that is, to approximate the true clean data distribution. We validate our method through extensive experiments on diverse computational imaging tasks, including random inpainting, denoising, and deblurring, achieving new state-of-theart performance. The code is available at https://github.com/ai4imaging/ EMDiffusoin .

1 Introduction

Diffusion models (DMs) (1; 2; 3) have demonstrated remarkable versatility in capturing complex real-world data distributions, excelling in diverse applications like image generation (4; 5; 6; 7; 8), audio synthesis (9), and molecular design (10). DMs approximate distributions by learning their score functions-the gradient of the log-likelihood of the data distribution ∇ x log p data ( x ) . This enables high-quality sample generation by simulating reverse-time stochastic differential equations (SDEs) (2) during inference.

Recently, there has been growing interest in leveraging DMs as priors for computational imaging inverse problems (11; 12; 13; 14; 15; 16), which aim to recover underlying images x from corrupted observations y . The Bayesian framework for computational imaging defines the posterior distribution of images x given observations y :

where p ( y | x ) defines the forward model of observations and p ( x ) defines an image prior. DMs offer efficient, data-driven priors that outperform traditional handcrafted priors prone to oversimplification and human biases, such as sparsity (17) or total variation (TV) (18; 19).

However, a major limitation of DM-based solvers is their reliance on substantial volumes of highquality, clean signals for pre-training-a requirement often infeasible in real-world settings, especially

Diffusion from

E-steps

1 st  M-step

Diffusion

Observations

Noise

Final

M-step

2 nd  M-step

…

masked data

Diffusion from

blurry data

Diffusion from

noisy data

for scientific and biomedical imaging. In contrast, corrupted noisy observations with differentiable forward models are easier to acquire, such as blurred images from mobile photography or 2D projections of 3D structures in X-ray computed tomography (CT) (20; 21) and cryogenic electron microscopy (cryo-EM) (22; 23). Our paper seeks to answer a pivotal question: Can a DM be effectively trained to solve inverse problems primarily using large-scale corrupted observations? This presents a chicken-egg dilemma: training an accurate DM requires clean images, but reconstructing clean images from corrupted observations requires a good DM.

Utilizing the Expectation-Maximization (EM) framework, we introduce a novel approach called EMDiffusion. This approach initializes with a diffusion prior trained on a minimal set of clean images, then alternates between two steps across multiple iterations: reconstructing clean images from corrupted observations using the current diffusion prior (E-step), and refining the DM parameters based on these reconstructions (M-step). The sparse clean data provides a good initialization of the DM's manifold, preventing collapse into a distorted or biased distribution characterized solely by corrupted inputs. Each E-M iteration leverages the current diffusion prior to generate cleaner reconstructions from the corrupted data, and these enhanced reconstructions then update the DM, providing an improved prior for the next iteration. This cycle continues, with the generated samples and DM progressively converging toward local optima, which equals to approximate the true clean data distribution. The forward operator and noise process do not affect this type of convergence but only influence the convergence speed by determining the amount of information in the corrupted observations.

We validate the generalizability and effectiveness of EMDiffusion through extensive experiments, applying it to diverse imaging inverse problems across various datasets, including random inpainting, denoising, and deblurring, and achieving compelling results.

2 Related Works

Inverse problems in computational imaging. Computational imaging aims to reconstruct underlying signals x ∈ R d from corrupted observations y ∈ R m , where the image formation process is probabilistically modeled as:

Since m ≤ d and observation noise is inevitable, inverse problems in computational imaging are ill-posed, with the inverse mapping y → x being one-to-many. To address this complexity, Bayesian inference introduces a prior distribution of underlying images, p ( x ) , to constrain the solution space for the image posterior, p ( x | y ) , as illustrated by Eq. 1. Employing Maximum a Posteriori (MAP) estimation, one can derive a point estimate of the underlying image by maximizing log p ( x | y ) . Alternatively, posterior image samples of reconstructed images can be obtained through methods like Markov Chain Monte Carlo (MCMC) (24) or Variational Inference (VI) (25; 26; 27). However, the

performance of many computational imaging solvers is limited by their reliance on oversimplified, handcrafted priors such as sparsity and total variation (TV). These priors fail to capture the true complexity of natural image distributions, hindering the solvers' ability to achieve high-quality reconstructions.

Diffusion models for inverse problems. Diffusion models (DMs) (1; 2; 3) have recently emerged as powerful data-driven priors for solving imaging inverse problems. By mastering the intricate distribution of images through training on extensive image data, DMs facilitate both point estimates via Plug-and-Play (PnP) optimization (28; 29) and posterior sampling through generative PnP (GPnP) (30), PnP Monte Carlo (PMC) (31), or Diffusion Posterior Sampling (DPS) (13; 14; 32). These approaches have demonstrated remarkable efficacy in addressing a broad spectrum of noisy inverse problems, with applications spanning diverse fields, including astronomy (11; 33) and biomedical imaging (15; 34).

Learn diffusion models from corrupted data. In many real-world scenarios, acquiring large-scale clean data is costly or infeasible, motivating efforts to learn DMs directly from corrupted data. Data corruptions stem from under-determined forward models (e.g., 2D projections, inpainting, compressed sensing) and measurement noise. Recent studies have explored various strategies to address these challenges. For instance, in inverse graphics, researchers integrate the forward model into the diffusion process and introduce a view-consistency loss over multiple noiseless projections of the same object to learn a 3D DM from 2D images (35; 36). In image inpainting, AmbientDiffusion (37) randomly masks additional pixels and forces the DM to restore these deliberate corruptions. Since the model cannot distinguish between original and further corruptions, it effectively learns the uncorrupted image distribution. However, the AmbientDiffusion is limited by the additional masking technique and fails to achieve good performance with noisy observations. (38) cleverly finetunes Stable Diffusion (SD) to leverage the pre-trained knowledge in denoising tasks, but does not support training a DM from scratch. Meanwhile, SURE-Score (39) proposes to jointly learn an image denoiser and a score-based DM using Stein's unbiased risk estimate (SURE) loss, where the SURE loss acts as an implicit regularizer on the model weights. Despite its innovative approach, SURE-Score often struggles with significant data corruption, such as inpainting tasks with a large fraction of missing pixels, and tends to produce overly smooth results. A general approach for learning DMs from arbitrarily corrupted data remains an open challenge.

3 Preliminary

3.1 Score-based Diffusion Models

A diffusion model captures the data distribution by learning a score function, i.e. the gradient of the logarithm of the likelihood of data distribution ▽ x log p data ( x ) . Consequently, a diffusion model generates samples by gradually removing noise from a random input, which is equivalent to a reverse-time stochastic differential equation (SDE) - the solution to a forward-time SDE that gradually injects noise,

where t ∈ [0 , T ] , f ( x t , t ) : R d → R d is the drift function, g ( t ) controls the rate of the Brownian motion w ∈ R d , and w denotes the Brownian motion running back. A tractable isotropic Gaussian distribution is achieved when t = T , i.e. x T ∼ N ( 0 , I ) , and the data distribution is achieved when t = 0 , i.e. x 0 ∼ p data . x t ∈ R d denotes the image x 0 diffused at time t . ∇ x t log p t ( x t ) is a time-dependent score function, which is usually approximated by a deep neural network, s θ ( · ) , parameterized by θ . The generated data distribution from the reverse-time SDE depends only on this time-dependent score function.

3.2 Diffusion Posterior Sampling

Many images are consistent with a single observation due to the ill-posed nature of the image formation model. By combining the forward model with the diffusion prior using Bayes' rule, we

define a conditional diffusion process that samples the posterior distribution

where the conditional score function can be further decomposed as:

Since the likelihood function is only defined for t = 0 , the dependence between y and x t is implicit, making ∇ x t log p t ( y | x t ) an intractable integral at each diffusion step. Various techniques have been proposed to address this intractable likelihood function, including exactly computing the probability using an ODE flow (11), bounding the probability through an evidence lower bound (ELBO)(33), and approximating the probability using Tweedie's formula(13; 40; 41; 42). To ensure computational efficiency, we adopt the approximation proposed in (13),

for diffusion posterior sampling in all the following sections.

3.3 Expectation Maximum Algorithm

The Expectation-Maximization (EM) algorithm (43; 44) is an iterative technique for estimating parameters in statistical models involving latent variables. When the true values of the latent variables are unknown, maximum likelihood estimation (MLE) cannot be directly applied to identify the model parameters. Instead, the EM algorithm maximizes a lower bound of the log-likelihood function, derived using Jensen's inequality:

where x , y , and θ denote the latent variables, observations, and model parameters, respectively. The algorithm alternates between two steps:

Expectation step (E-step) : Sample latent variables from the current estimate of the conditional distribution, x ∼ p θ ( x | y ) , and compute the expected log-likelihood lower bound L ( θ ) .

Maximization step (M-step) : Maximize L ( θ ) = E x ∼ p θ ( x | y ) [log p θ ( x )] to update parameters θ .

This iterative procedure allows the EM algorithm to converge to a local maximum of the observed data log-likelihood, making it a powerful technique for estimation problems involving latent variables, such as Gaussian mixture clustering(45), and dynamical system identification(46).

4 Proposed Method

Given corrupted observations y and a known forward model p ( y | x ) , learning DMs from corrupted data is a parameter estimation problem involving latent variables. The latent variables are the underlying clean images x , and the goal is to estimate the DM parameters θ that govern the image prior p θ ( x ) . Consequently, we can leverage an iterative EM approach to reconstruct clean images and train the DM using corrupted data jointly, as described in Fig. 1 and Algorithm 1.

4.1 Initialization: Training a Vague Diffusion Model using Limited Clean Images

The Expectation-Maximization (EM) algorithm needs a good initialization to begin its iterative process, as an improper initialization can result in convergence at an incorrect local minimum. While obtaining a large dataset of clean images is difficult, a small set of clean data is often available. This limited clean data can be used to train an initial DM to start the EM iterations. For example, in all

(a) Observations

(b)

λ

= 1

(c)

λ

= 10

(d)

λ

= 20

the following experiments, 50 randomly selected clean images were used to train the initial DM, serving as the starting point for the EM algorithm. As demonstrated in Sec. 5.4, clean images do not need to be from the same dataset; those from out-of-distribution datasets also serve as reasonable initializations.

4.2 E-step: Adaptive Diffusion Posterior Sampling

In the E-step, we assume a known diffusion prior and reconstruct the underlying clean images through diffusion posterior sampling. We adopt the standard variance-preserving form of the stochastic differential equation (VP-SDE) (2), which is equivalent to the Denoising Diffusion Probabilistic Models (DDPM) (1). The drift function f ( x t , t ) takes the form β ( t ) x t / 2 , and the diffusion rate g ( t ) is √ β ( t ) . Therefore, the reverse diffusion sampler in Eq. 4 can be represented as:

Considering a known imaging forward model, A , and additive Gaussian noise, p ( y | x ) ∼ N ( y | A ( x ) , σ 2 I ) , the conditional score function can be represented as:

where

However, a naive diffusion posterior sampling approach using Eqs. 8, 9, and 10 often fails to produce high-quality reconstructions. This is because the learned DM is inaccurate during the early EM iterations. We demonstrate this issue with a toy experiment. We performed diffusion posterior sampling (DPS) on randomly masked observations, as shown in Fig. 2(a), using an initial DM trained on only 50 clean images. The resulting posterior samples, depicted in Fig. 2(b), show mode collapse due to the severely limited prior. All recovered samples come from the training set of 50 clean images and are unrelated to the observations. Similarly, if the DM is trained on blurry, noisy images with artifacts, naive DPS also performs poorly in image reconstruction.

Algorithm 1 EMDiffusion: Learning Score-based Priors for Inverse Imaging

Require: DM s θ , Observations ( Y , A ) , few clean data x , Cycles N , Timesteps T , Epoches M ,

Measurement noise N ( 0 , σ 2 I ) , Diffusion rate { β t } T t =1

1: Initialize s θ on x through denoising score matching (47)

2: for i = 1 to N do

3: ( y , f ) ∼ Dataset( Y , A )

4: x T ∼ N ( 0 , I )

5: for t = T to 1 do

6: ¯ α t = ∏ t s =1 (1 -β t )

8: z ∼ N ( 0 , I )

9: Take reverse-time SDE step on {Sampling in Sec. 4.2}

10: end for

11: for m = 0 to M do

12: x data ∼ Shuffle(ˆ x ( i ) 0 ∼ p θ (ˆ x ( i ) 0 | y ( i ) ))

13: t ∼ Uniform( { 1 , . . . , T } )

14: ¯ α t = ∏ t s =1 (1 -β t )

15: ϵ ∼ N ( 0 , I )

16: Take gradient descent step on {Optimization in Sec. 4.3}

17: end for

18: end for

It does not mean that these low-quality DMs cannot provide any prior information. Although the prior is poor in the early training stages, it has learned common features and structures shared among natural images, such as the continuity and smoothness of natural images and profiles of specific object types. By introducing a hyper-parameter λ to rescale the likelihood term and avoid mode collapse, we find that the low-quality DM can also act as a weak prior for posterior sampling, where the reverse-time SDE can be written as:

The hyper-parameter λ efficiently balances the diffusion prior and the data likelihood, resulting in reliable reconstructed images even when the prior is poor. As demonstrated in Fig. 2 (b), (c), and (d), as λ increases from 1 to 20, the data likelihood term gains more emphasis, making the reconstructed images more consistent with the inpainting observations. The choice of the hyper-parameter λ is automated in each E-step by finding the value that minimizes the data loss,

where ˆ x 0 ,λ represents the diffusion posterior samples of reconstructed images with λ scaling.

4.3 M-step: Optimizing Score-Based Priors

During the M-step, we update the weights of the score-based models using the posterior samples obtained in the E-step. This resembles training a standard clean DM, s θ , to approximate the timedependent score function, ∇ x t log p ( x t | ˆ x 0 ) , through denoising score matching (47):

where t ∼ Uniform ( { 1 , ..., T } ) , ˆ x 0 = ˆ x 0 ,λ ∗ represents the posterior samples from the previous E-step, and x t ∼ p ( x t | ˆ x 0 ) are generated by the forward-time SDE in Eq. 3.

Masked

Observation

SURE-

Score (39)

Ambient

Diffusion (37)

Ours 1st

Iteration

Ours Final

Iteration

DPS (13) w/

Clean Prior

Ground

Truth

To accelerate training, especially during the early stages when the posterior samples are noisy, the M-step does not always train the score function s θ ( · ) from scratch. In the initial M-steps, we inherit the DM weights from the previous iteration and fine-tune them only using posterior samples from a subset of observations (e.g., randomly select 10% of total observations). However, once the quality of reconstructed images improves sufficiently, we reinitialize the DM weights and retrain the model with 100% data for a few more iterations. The training strategy transitions when the optimal balancing parameter, λ ∗ , falls below 1, or fails to decrease for more than three consecutive iterations.

5 Experiments

In this section, we demonstrate the performance of our method in learning DMs from corrupted data and solving inverse problems using these models. We validate the method on three imaging tasks: random inpainting, denoising, and deblurring. Our main results are presented in Fig.3, Fig.4, and Table 1, with additional ablation studies in Fig. 5. Further details on neural network architectures, training settings, and additional reconstruction and generation samples are provided in the appendix.

5.1 Datasets and Evaluation Metrics

The experiments are conducted on the CIFAR-10 (48) and CelebA (49) datasets at resolutions of 32 × 32 and 64 × 64 , respectively. CIFAR-10 consists of 50,000 images across 10 classes for training, while CelebA contains 30,000 images of human faces. At each iteration, 5,000 corrupted images are randomly chosen for posterior sampling and training, and 250 corrupted images from the test set are chosen for evaluation.

We evaluate the performance of our method using two groups of metrics. First, we compute the peak signal-to-noise-ratio (PSNR) and learned Perceptual Image Patch Similarity (LPIPS) scores between the reconstructed and ground-truth images, quantifying the accuracy of inverse imaging using learned DMs. Additionally, we compute the Fréchet Inception Distance (FID) between the learned DMs and reserved test data to assess their image generation quality.

5.2 Baseline and Training Settings

We compare our method with three related baselines: AmbientDiffusion (37), SURE-Score (39), and DPS with clean prior (13). AmbientDiffusion and SURE-Score have similar settings to our method, which do not require DMs pre-trained on large-scale clean signals. Considering AmbientDiffusion is well-designed for masked observations, we only use it as the baseline of the image inpainting task. On the other hand, DPS leverages a pre-trained clean diffusion prior for posterior sampling, so it defines the performance upper bound for our method.

In our experiments, we randomly select 50 clean images from each dataset to train the initial DMs for the EM iterations. AmbientDiffusion is trained with the standard setting in (37). The key hyperparameter of SURE-Score, σ ω , is set to the observation noise's standard deviation (0.2 for denoising,

Noisy

Observation

SURE-

Score (39)

Ours

DPS (13) w/

Clean Prior

Ground

Truth

Observation

SURE-

Score (39)

Ours

DPS (13) w/

Clean Prior

Ground

Truth

(a) CIFAR10, Denoising

(b) CelebA, Deblurring

and 0.01 for inpainting and deblurring). To ensure a fair comparison, we also provide the same 50 clean images for training AmbientDiffusion and SURE-Score. Details are in Appendix A.

5.3 Results

Image inpainting. We conduct random inpainting (with mask probability p = 0 . 6 ) on CIFAR-10. As shown in Fig. 3 and Table 1, our method significantly outperforms AmbientDiffusion and SUREScore, achieving reconstruction quality similar to DPS with a prior trained on the clean CIFAR-10 dataset. The iterative training process is also illustrated in Fig. 3. Initially, our method performs poorly with the DM trained on only 50 randomly selected clean samples. However, as the E-step and M-step alternate iteratively, the quality of posterior sampling improves. Large-scale posterior samples enrich the priors, leading to enhanced performance at each stage.

Image denoising. We perform image denoising on CIFAR-10 with Gaussian noise n ∼ N (0 , σ 2 I ) and σ = 0 . 2 . The results are shown in Fig. 4(a) and Table 1. Our method outperforms SURE-Score, and the self-supervised denoising benchmark, Noise2Self (50), though it slightly lags behind DPS with clean priors. However, while our method's reconstructions may appear noisier than DPS results, they sometimes reproduce more details, such as the car wheels in the second row and the cat face in the third row of Fig. 4(a), showcasing the better diversity of our learned DMs.

Image deblurring. We validate image deblurring on CelebA using a Gaussian blur kernel with a size of 9 × 9 and a standard deviation of σ = 2 pixels. The results are shown in Fig. 4(b) and Table 1. As with the other tasks, our method significantly outperforms SURE-Score in solving imaging inverse problems, recovering fine details of human faces. However, the FID score of our learned diffusion models lags behind the original blurred observations. This is primarily because the FID score measures image similarity mainly through smooth features, making it a less effective metric for deblurring tasks.

(a) Initialization DMs

(b) DMs after each EM iteration

(c) Trend of optimal

λ

32

180

30

28

Inpainting

Denoising

Deblurring

160

140

50k corrupted data

50kclean data

40kclean data

20k clean data

24

22

Iteration0

Iteration 1

Iteration2

26

120

15kclean data

20

Iteration3

10k clean data

Optimal

PSNR

24

100

PSNR

18

22

80

16

60

20

14

18

40

34.30

16

500 ID

100 ID

50 ID

10 ID

50 0OD

20

0

0

2

4

6

21.0

8

24.98

11.09

9.44

7.08

10

12

10

0.11357911

20

50

100

Data

EMIteration

Scalingfactor入

Comparing the results of all three tasks, we find that AmbientDiffusion only works well for inpainting because its additional masking technique is specifically designed for that purpose. SURE-Score consistently produces over-smoothed results because the SURE loss regularizes the gradient of generated images. As a comparison, our method does not make any special assumptions and provides a general framework applicable to all three tasks. The generation results are in Appendix D.

5.4 More Analysis and Ablation Studies

Number of clean images for training initial DMs. Our EM approach starts with DMs trained on a small set of clean images. Fig. 5(a) shows the PSNR of posterior samples generated by these models in the first E-step, allowing us to evaluate the impact of the number of clean training images on the performance of the initial DMs. Remarkably, DMs trained on as few as 10 clean images (0.02% of the corrupted images) can still act as reasonable priors. For inpainting and denoising tasks, DMs trained on 10 clean images provide nearly the same reconstruction quality as those trained on 500 clean images, as these tasks primarily require priors for low-frequency features, and 10 images suffice for an initial guess. However, the deblurring task benefits from DMs trained on more clean data since deblurring aims to recover high-frequency details where more data helps.

Surprisingly, we find that DMs trained on clean images from CelebA (downsampled to 32×32) can also be used to initialize tasks on CIFAR-10. For inpainting and denoising tasks, DMs initialized with 50 clean images from an out-of-distribution (OOD) dataset (CelebA) achieve similar performance to those initialized with in-distribution (ID) data (CIFAR-10). For the deblurring task, DMs trained on OOD data perform better than those trained on a similar amount of ID data, suggesting that OOD data can sometimes serve as stronger priors for guessing high-frequency information.

Learned priors through iterative training. Fig. 5(b) shows the FID scores of the learned DMs in the inpainting task with 50,000 corrupted CIFAR-10 images after each EM stage. The generation ability of the DMs gradually improves as the EM iterations progress. As explained in Sec. 4.3, initially the DM inherits weights from previous steps for fast training and converges at the sixth iteration. After resetting the DM, the training resumes for three more rounds and finally converges to a FID score of approximately 21.08, significantly better than AmbientDiffusion's 28.88, setting a new state-of-the-art. Notably, our method achieves this performance using a DM architecture with far fewer parameters: our method employs a vanilla DDPM with 35.7 million parameters, while AmbientDiffusion uses an improved DDPM++ architecture with over 100 million parameters. Additionally, we compared our method with DMs trained on different amounts of clean data. Our model, learned from 50,000 corrupted images (60% pixels masked) using EM, performs better than a DMtrained on around 15,000 clean images.

Scaling factor in adaptive diffusion posterior sampling. Fig. 5(c) shows the PSNR of reconstructed images at each EM stage with different scaling factors, using the inpainting task on CIFAR-10 as an example. We observe that the quality of posterior sampling initially improves and then de-

teriorates as the scaling factor increases, confirming the existence of an optimal scaling factor as suggested in Eq. 12. As the EM stages progress, the optimal scaling factor decreases, indicating that the learned priors progressively improve through the EM iterations. This observation justifies the need for adaptive scaling factors in our method.

6 Conclusion

In this paper, we proposed EMDiffusion, a novel expectation-maximization (EM) framework for training diffusion models primarily from corrupted observations. The key assumption is that it is information-theoretically possible to learn the underlying distribution from measurements. Our method demonstrated state-of-the-art performance in image inpainting, denoising, and deblurring across various datasets. Additionally, an important finding is that a small amount of clean, indistribution data can act as an implicit regularizer, aiding the training of diffusion models from corrupted observations. Future work will aim to 1) extend initialization approaches, potentially by incorporating foundation models or traditional machine learning techniques, such as using preprocessed images from unsupervised inpainting, deblurring, or denoising for initialization, and 2) extend to various imaging inverse problems and learning unknown forward models or noise statistics.

References

J. Ho, A. Jain, and P. Abbeel, 'Denoising diffusion probabilistic models,' Advances in neural information processing systems , vol. 33, pp. 6840-6851, 2020.

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, 'Score-based generative modeling through stochastic differential equations,' arXiv preprint arXiv:2011.13456 , 2020.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, 'Deep unsupervised learning using nonequilibrium thermodynamics,' in International conference on machine learning . PMLR, 2015, pp. 2256-2265.

A. Q. Nichol and P. Dhariwal, 'Improved denoising diffusion probabilistic models,' in International Conference on Machine Learning . PMLR, 2021, pp. 8162-8171.

A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, 'Zeroshot text-to-image generation,' in International Conference on Machine Learning . PMLR, 2021, pp. 8821-8831.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, 'Hierarchical text-conditional image generation with clip latents,' arXiv preprint arXiv:2204.06125 , 2022.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, 'High-resolution image synthesis with latent diffusion models,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 10 684-10 695.

C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi, 'Palette: Image-to-image diffusion models,' in ACM SIGGRAPH 2022 Conference Proceedings , 2022, pp. 1-10.

Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, 'Diffwave: A versatile diffusion model for audio synthesis,' arXiv preprint arXiv:2009.09761 , 2020.

M. Xu, L. Yu, Y. Song, C. Shi, S. Ermon, and J. Tang, 'Geodiff: A geometric diffusion model for molecular conformation generation,' arXiv preprint arXiv:2203.02923 , 2022.

B. T. Feng, J. Smith, M. Rubinstein, H. Chang, K. L. Bouman, and W. T. Freeman, 'Score-based diffusion models as principled priors for inverse imaging,' arXiv preprint arXiv:2304.11751 , 2023.

G. Zhang, J. Ji, Y . Zhang, M. Yu, T. Jaakkola, and S. Chang, 'Towards coherent image inpainting using denoising diffusion implicit models,' in International Conference on Machine Learning . PMLR, 2023, pp. 41 164-41 193.

H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye, 'Diffusion posterior sampling for general noisy inverse problems,' arXiv preprint arXiv:2209.14687 , 2022.

H. Chung, B. Sim, D. Ryu, and J. C. Ye, 'Improving diffusion models for inverse problems using manifold constraints,' Advances in Neural Information Processing Systems , vol. 35, pp. 25 683-25 696, 2022.

Y. Song, L. Shen, L. Xing, and S. Ermon, 'Solving inverse problems in medical imaging with score-based generative models,' arXiv preprint arXiv:2111.08005 , 2021.

B. Kawar, N. Elata, T. Michaeli, and M. Elad, 'Gsure-based diffusion model training with corrupted data,' arXiv preprint arXiv:2305.13128 , 2023.

W. Zhao, S. Zhao, L. Li, X. Huang, S. Xing, Y. Zhang, G. Qiu, Z. Han, Y. Shang, D.-e. Sun et al. , 'Sparse deconvolution improves the resolution of live-cell super-resolution fluorescence microscopy,' Nature biotechnology , vol. 40, no. 4, pp. 606-617, 2022.

C. Bouman and K. Sauer, 'A generalized gaussian image model for edge-preserving map estimation,' IEEE Transactions on image processing , vol. 2, no. 3, pp. 296-310, 1993.

K. Kuramochi, K. Akiyama, S. Ikeda, F. Tazaki, V. L. Fish, H.-Y. Pu, K. Asada, and M. Honma, 'Superresolution interferometric imaging with sparse modeling using total squared variation: application to imaging the black hole shadow,' The Astrophysical Journal , vol. 858, no. 1, p. 56, 2018.

C. Wang, K. Shang, H. Zhang, Q. Li, Y. Hui, and S. K. Zhou, 'Dudotrans: dual-domain transformer provides more attention for sinogram restoration in sparse-view ct reconstruction,' arXiv preprint arXiv:2111.10790 , 2021.

A. W. Reed, H. Kim, R. Anirudh, K. A. Mohan, K. Champley, J. Kang, and S. Jayasuriya, 'Dynamic ct reconstruction from limited views with implicit neural representations and parametric motion fields,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 2258-2268.

E. D. Zhong, T. Bepler, B. Berger, and J. H. Davis, 'Cryodrgn: reconstruction of heterogeneous cryo-em structures using neural networks,' Nature methods , vol. 18, no. 2, pp. 176-185, 2021.

E. Ye, Y. Wang, H. Zhang, Y. Gao, H. Wang, and H. Sun, 'Recovering a molecule's 3d dynamics from liquid-phase electron microscopy movies,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 10 767-10 777.

S. Brooks, A. Gelman, G. Jones, and X.-L. Meng, Handbook of markov chain monte carlo . CRC press, 2011.

D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, 'Variational inference: A review for statisticians,' Journal of the American statistical Association , vol. 112, no. 518, pp. 859-877, 2017.

H. Sun and K. L. Bouman, 'Deep probabilistic imaging: Uncertainty quantification and multimodal solution characterization for computational imaging,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, no. 3, 2021, pp. 2628-2637.

H. Sun, K. L. Bouman, P. Tiede, J. J. Wang, S. Blunt, and D. Mawet, ' α -deep probabilistic inference ( α -dpi): efficient uncertainty quantification from exoplanet astrometry to black hole feature extraction,' The Astrophysical Journal , vol. 932, no. 2, p. 99, 2022.

A. Graikos, N. Malkin, N. Jojic, and D. Samaras, 'Diffusion models as plug-and-play priors,' Advances in Neural Information Processing Systems , vol. 35, pp. 14 715-14 728, 2022.

Y. Zhu, K. Zhang, J. Liang, J. Cao, B. Wen, R. Timofte, and L. Van Gool, 'Denoising diffusion models for plug-and-play image restoration,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 1219-1229.

C. A. Bouman and G. T. Buzzard, 'Generative plug and play: Posterior sampling for inverse problems,' arXiv preprint arXiv:2306.07233 , 2023.

Y. Sun, Z. Wu, Y. Chen, B. T. Feng, and K. L. Bouman, 'Provable probabilistic imaging using score-based generative priors,' arXiv preprint arXiv:2310.10835 , 2023.

H. Chung, B. Sim, and J. C. Ye, 'Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 12 413-12 422.

B. T. Feng and K. L. Bouman, 'Efficient bayesian computational imaging with a surrogate score-based prior,' arXiv preprint arXiv:2309.01949 , 2023.

H. Chung and J. C. Ye, 'Score-based diffusion models for accelerated mri,' Medical image analysis , vol. 80, p. 102479, 2022.

A. Tewari, T. Yin, G. Cazenavette, S. Rezchikov, J. B. Tenenbaum, F. Durand, W. T. Freeman, and V. Sitzmann, 'Diffusion with forward models: Solving stochastic inverse problems without direct supervision,' arXiv preprint arXiv:2306.11719 , 2023.

T. Anciukeviˇ cius, Z. Xu, M. Fisher, P. Henderson, H. Bilen, N. J. Mitra, and P. Guerrero, 'Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 12 608-12 618.

G. Daras, K. Shah, Y. Dagan, A. Gollakota, A. G. Dimakis, and A. Klivans, 'Ambient diffusion: Learning clean distributions from corrupted data,' 2023.

G. Daras, A. G. Dimakis, and C. Daskalakis, 'Consistent diffusion meets tweedie: Training exact ambient diffusion models with noisy data,' arXiv preprint arXiv:2404.10177 , 2024.

A. Aali, M. Arvinte, S. Kumar, and J. I. Tamir, 'Solving inverse problems with score-based generative priors learned from noisy data,' arXiv preprint arXiv:2305.01166 , 2023.

B. Kawar, M. Elad, S. Ermon, and J. Song, 'Denoising diffusion restoration models,' Advances in Neural Information Processing Systems , vol. 35, pp. 23 593-23 606, 2022.

A. Jalal, M. Arvinte, G. Daras, E. Price, A. G. Dimakis, and J. Tamir, 'Robust compressed sensing mri with deep generative priors,' Advances in Neural Information Processing Systems , vol. 34, pp. 14 938-14 954, 2021.

J. Song, A. Vahdat, M. Mardani, and J. Kautz, 'Pseudoinverse-guided diffusion models for inverse problems,' in International Conference on Learning Representations , 2022.

A. P. Dempster, N. M. Laird, and D. B. Rubin, 'Maximum likelihood from incomplete data via the em algorithm,' Journal of the royal statistical society: series B (methodological) , vol. 39, no. 1, pp. 1-22, 1977.

A. Gao, J. Castellanos, Y. Yue, Z. Ross, and K. Bouman, 'Deepgem: Generalized expectationmaximization for blind inversion,' Advances in Neural Information Processing Systems , vol. 34, pp. 11 592-11 603, 2021.

D. A. Reynolds et al. , 'Gaussian mixture models.' Encyclopedia of biometrics , vol. 741, no. 659-663, 2009.

H. Sun, N. J. Kasdin, and R. Vanderbei, 'Identification and adaptive control of a high-contrast focal plane wavefront correction system,' Journal of Astronomical Telescopes, Instruments, and Systems , vol. 4, no. 4, pp. 049 006-049 006, 2018.

P. Vincent, 'A connection between score matching and denoising autoencoders,' Neural computation , vol. 23, no. 7, pp. 1661-1674, 2011.

A. Krizhevsky, G. Hinton et al. , 'Learning multiple layers of features from tiny images,' 2009.

Z. Liu, P. Luo, X. Wang, and X. Tang, 'Deep learning face attributes in the wild,' in Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

J. Batson and L. Royer, 'Noise2self: Blind denoising by self-supervision,' in International Conference on Machine Learning . PMLR, 2019, pp. 524-533.

O. Ronneberger, P. Fischer, and T. Brox, 'U-net: Convolutional networks for biomedical image segmentation,' in Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 . Springer, 2015, pp. 234-241.

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770-778.

I. Loshchilov and F. Hutter, 'Decoupled weight decay regularization,' arXiv preprint arXiv:1711.05101 , 2017.

D. Kim, S. Shin, K. Song, W. Kang, and I.-C. Moon, 'Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation,' arXiv preprint arXiv:2106.05527 , 2021.

T. Karras, M. Aittala, T. Aila, and S. Laine, 'Elucidating the design space of diffusion-based generative models,' Advances in Neural Information Processing Systems , vol. 35, pp. 26 56526 577, 2022.

J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, 'Free-form image inpainting with gated convolution,' in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 4471-4480.

Y. Song and S. Ermon, 'Improved techniques for training score-based generative models,' Advances in neural information processing systems , vol. 33, pp. 12 438-12 448, 2020.

A. Danielyan, V. Katkovnik, and K. Egiazarian, 'Bm3d frames and variational image deblurring,' IEEE Transactions on image processing , vol. 21, no. 4, pp. 1715-1728, 2011.

A. Beck and M. Teboulle, 'Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems,' IEEE transactions on image processing , vol. 18, no. 11, pp. 2419-2434, 2009.

We provide the implementation details and more results in the appendix. We first describe our network architecture and training settings in Sec. A, then show initialization details in Sec. B. More results are also provided in Sec. C and Sec. D.

A Implementation Details.

Our neural network architecture follows the vanilla denoising diffusion probabilistic model (DDPM) (1). For quick implementation, see https://huggingface.co/google/ ddpm-cifar10-32 and https://huggingface.co/google/ddpm-celebahq-256 .

Model architecture. Our architecture is exactly aligned with DDPM (1), which is a U-Net (51) based on a Wide ResNet (52). Diffusion time t is implemented by adding the Transformer sinusoidal position embedding into each residual block. For CIFAR-10, our 32 × 32 models use four feature map resolutions ( 32 × 32 to 4 × 4 ) and convolutional residual blocks per resolution level. For CelebA, we increase the feature map number for our 64 × 64 to six. We enable the dropout regularization to reduce overfitting. Our CIFAR-10 model has 35.7 million parameters and our CelebA model has 114 million parameters.

Noise schedule. We leverage the default settings on VP-SDE (2), which uses a linear schedule with timesteps T = 1000 , β 1 = 1 e -4 , and β T = 0 . 02 .

Exponential moving average. To stabilize the training process and reduce the color shift of samples generated by trained DMs, we adopt an exponential moving average (EMA) technique with a decay factor of 0.999 for all experiments.

Optimizer. We apply AdamW (53) and set the learning rate to 2 e -4 for CIFAR-10 and 2 e -5 for CelebA.

Hyperparameters for the training process. We set the batch size to 512 for CIFAR-10 and 64 for CelebA. We set the dropout rate to 0.1 for CIFAR-10 and 0 for CelebA. As for the learning rate, we adapt 1 e -4 for CIFAR-10 and 2 e -5 for CelebA, for a larger learning rate will result in unstable training.

Dataset argumentation. We only use random horizontal flips to CIFAR-10 during training to achieve better performance. We did not flip CelebA, as the distribution of human faces is quite simple. We provide a table of training hyperparameters in Table 2.

Baselines. As for Ambient Diffusion, We use the official checkpoint, which adopts the improved DDPM++ architecture (54) and the EDM scheduler design (55). It also modifies the model architecture's convolutions to Gated Convolutions (56), as they are known to perform better for inpainting-type problems. Their 32 × 32 model has 113 million parameters, which is larger than ours. SureScore, on the other hand, uses the deepest NCSNv2 (57) model architecture, which has 95 million parameters. As for DPS with clean priors, we adopt the pre-trained DDPMs provided by https://huggingface.co/google . The details of the architectures of all methods are shown in Table 3.

Training schedule. At each EM iteration, we randomly choose 5,000 corrupted observations for diffusion posterior sampling and then train DMs. We further divide the iterations into two phases:

Phase 1 - Resume training DMs: at the early EM iterations, we inherit weights of DMs from the last iteration for quick convergence. For CIFAR-10, this phase lasts for about 6-8 EM iterations, while for CelebA deblurring, this phase increases to 10 EM iterations.

· Phase 2 - Reset training DMs: at the later EM iterations, we reset the weights of DMs at each M-step, that is, training DMs from scratch. The key insight is that DMs from Phase 1 always have a memory of bad posterior samples, which has a negative effect on the learned distribution. For CIFAR-10, this phase lasts for 3 EM iterations, we found it significantly improves the FID score of DMs. While for CelebA deblurring, this phase lasts for 2 EM iterations until we find the improvement is not obvious.

B Additional Strategies for Training Initial DMs

To verify the sensitivity of EMDiffusion's initial DM training data, we provide more quantitative and qualitative results in this section, as shown in Table 4. We draw the conclusion that EMDiffusion is not sensitive to the initial data. Apart from evaluating different numbers of in-distribution (ID) images and out-of-distribution (OOD) images for training the initial DMs, we also test the initialization on preprocessed observations, and find all of them converge similarly.

Specifically, we preprocess the noisy observations with BM3D (58), the blurry observations with Fast TV Constraint (59), and leave the masked observations unchanged. DMs initialized on these preprocess samples also perform well on the following E-step.

C Additional Results on Random Inpainting

As shown in Table 5, we include additional comparisons to Ambient Diffusion (37) across various corruption conditions and datasets. The performance gap between the proposed method and Ambient Diffusion narrows under higher levels of corruption, likely due to the simpler DDPM architecture we employ. In contrast, Ambient Diffusion utilizes the improved DDPM++ architecture (54), which is specifically modified to perform better for high-corruption inpainting-type problems. Nonetheless, the overall performance demonstrates the effectiveness of the proposed method, as we do not focus on empirically optimized architecture details, but instead show the applicability of the new EM idea for training DMs from corrupted observations.

D Generative Samples

EMDiffusion is proposed to learn clean distributions from corrupted observations. In Sec. 5, we present detailed posterior sampling results and FID scores of learned DMs.

Our model outperforms baselines by a significant margin in three inverse imaging tasks on two datasets. Though the FID score of our model trained on blurry CelebA is slightly higher than Ambient Diffusion, we argue that FID scores are easily influenced by sharp artifacts introduced by DPS (13), which is adopted in our E-steps. However, the distributions EMDiffusion learned from various types of corrupted observations are obviously better than baselines, as shown in Fig. 7,6,8.

Future work. To achieve a better posterior distribution through the proposed EM framework, an accurate and efficient E-step plays a key role. We adopt the Diffusion Posterior Sampling (13) that could potentially introduce artifacts due to its approximation of the underlying data likelihood term. Therefore, a better FID score could be achieved by designing a principled posterior sampling method.

(c) Ours, FID=21.08

(c) Ours, FID=86.47

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We make clear claims that well reflect the paper's contributions and scope Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the limitation in the last.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We show ablation study for the proposed techniques.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide code in supp.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the experiment details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our experiments contain a large amount of examples that don't need the error bar.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide details of the computation resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research explores computational imaging algorithms in which we didn't see any ethics issues.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:Our research explores computational imaging algorithms in which we didn't see any negative social impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We study computational imaging algorithms and didn't see such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use public datasets that are licensed for research purposes.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We did;t use new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We didn't do crowd sourcing with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We didn't do experiments related to human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new method for training diffusion models from corrupted data.\nThe presented method is an EM algorithm that in turn (i) uses the current model to sample clean images given the noisy input and then (ii) use these clean samples to train the diffusion model.\nThe authors show the results for data corrupted by noise, blurring and missing pixels.\n\nSTRENGTHS:\n* This is a conceptual paper that does not focus on empirically optimised architecture details, but instead shows the applicability of a simple elegant new idea. This is a plus.\n* I can think of many applications in e.g. in scientific imaging (e.g. microscopy or astronomy) that suffer heavily from noise and degradation and for which clean data training is usually not available or very limited. So, I believe such methods can have a substantial practical impact.\n* The paper is well structured, with each building block (Score based diffusion models, posterior sampling, EM algorithm) being carefully introduced.\n\nWEAKNESSES:\n* Most importantly, I think the evaluation is not as convincing as it could be. All qualitative results seem to have artefacts to a degree. This also seems to be true for the baselines. E.g. the ambient diffusion results in Figure 3 suffer from some form of pixel artefacts. This is not the case when looking at the ambient diffusion paper (e.g. Figure 7), where not such artefacts are visible. Granted, this is a different dataset. However, I wonder if there is a problem with the training.\n\n* I think the fact that the method requires some (albeit little) clean data as a first training step is a weakness of the method.\n* My understanding is that the current system assumes Gaussian noise as part of the forward model (line 149). This can be limiting.\n* There are some typos and language problems.\n\nQUESTIONS:\nAt the moment the degradation model is known and seems to have to involve Gaussian noise (line 149).\nHow does this work for the blurring operation, which seems to be deterministic? Or is the amount of added Gaussian noise just not visible.\nHow does it work for the in-painting task, which also does not seem to involve Gaussian noise?\n\n\nTypos and language:\n\nLine 103-104 \"we defines\"\nTable 4: \"The results show that EM-Score is insensitivity to initializations\"\n\nLIMITATIONS:\nDo the authors think that it could be possible to include other degradation models, such as a more realistic combination of Gaussian noise Poisson shot noise? \nWould it potentially be possible to learn the parameters of a degradation model. e.g. the blur kernel of the parameters of the noise distribution?\nA learnable degradation model would drastically increase the applicability in real life imaging applications.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper we proposes an expectation-maximization (EM) approach to train diffusion models from corrupted observations. The extensive experiments show the effectiveness of the proposed method.

STRENGTHS:
1. The experiments are extensive.
2. The proposed approach addresses a significant challenge in the field where large clean datasets are often unavailable.

WEAKNESSES:
1. The source code is not provided yet, but the authors promised to make it public in the future.

QUESTIONS:
While you mention that only a small number of clean images are needed to initialize the EM iterations, does the number of clean images affect the final model performance in applications? Is it possible to completely eliminate the dependence on clean data?

LIMITATIONS:
The authors presented the limitations and proposed the future direction to addressed them.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose using the EM algorithm to train a diffusion model on corrupted data. To initialize the process, the proposed method requires access to a \"limited number of clean samples.\" The authors claim the method allows convergence to the true data distribution.\n\nSTRENGTHS:\n**Originality.** The idea of using EM to train a diffusion model from corrupted data is specifically original; however, training generative models without access to clean data is a well-explored problem, even before the diffusion model era.\n\n**Quality and clarity.** The paper is easy to follow, and the problem is well-motivated.\n\n**Significance.** I do not find the proposed method in its current format significant, as it assumes access to a \"limited dataset of clean images\" without quantifying how small this set can be. Additionally, the paper claims convergence to the true data distribution without providing conditions on the forward operator and noise process, which is an unsubstantiated claim given the limited set of inverse problems explored.\n\nWEAKNESSES:\n* Claiming convergence to the true data distribution is a grand claim, and simply providing examples from relatively simple inverse problems does not substantiate this claim. Therefore, due to unsubstantiated claims and limited quantitative discussion on the required conditions for the forward operator and the number of clean data samples needed, I find this paper not truly usable in practice.\n\nQUESTIONS:\n* How do authors quantify \"minimal clean images\"? The authors mention\n  that sparse clean data provides a good initialization of the DM's\n  manifold. Why is this the case? What does \"good\" mean precisely in\n  this context?\n\nLIMITATIONS:\n* I disagree with the authors that needing access to a minimal number of\n  clean images is a minor limitation. AmbientDiffusion and some other prior\n  work discussed do not assume access to such a dataset, and hence the\n  comparisons might not be entirely fair.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a new framework for training diffusion models from corrupted data. Prior work on this research topic is based on the Ambient Diffusion framework or Stein's Unbiased Risk Estimate (SURE) idea. The authors propose an alternative methodology based on the Expectation-Maximization algorithm. The algorithm is tested on standard datasets and leads to improved performance over the considered baselines.\n\nSTRENGTHS:\n1. The research topic of training diffusion from corrupted data is relevant and timely.\n2. The authors propose a fresh idea for this interesting research problem. The proposed idea has several elegant properties: I) it is simple, ii) relatively easy to implement, and iii) the idea is oblivious to the type of corruption.\n3. The fact that the proposed algorithm is agnostic to the type of corruption is important: as the authors correctly mention, prior work, such as Ambient Diffusion, is designed (or at least tested) only for inpainting. \n4. The authors manage to outperform the prior state-of-the-art, Ambient Diffusion in the same setting (CIFAR-10 with 60% corruption). \n5. The authors have a nice way of leveraging clean samples. Prior work assumes that we don't have any access to clean samples. This paper naturally integrates the available clean data by using them to initialize the EM algorithm.\n\nOverall, I believe this submission offers a promising alternative to Ambient Diffusion for learning diffusion models with corrupted data.\n\nWEAKNESSES:\n1. The comparison to the baselines is somewhat limited/unfair. As the authors mention, Ambient Diffusion has been designed for inpainting. The authors use Ambient Diffusion for denoising and Deblurring. It is not clear how they do this since the original algorithm cannot be trivially extended to these settings. Even for the case of inpainting, the authors could report results for other levels of corruption as well (apart from $p=0.6$) and for other datasets apart from CIFAR-10. Regarding the comparisons with Ambient Diffusion, the only valid data point right now is for $p=0.6$. Providing a more comprehensive analysis would help the readers understand in what regime the proposed method works the best.\n2. Specifically for the case of denoising, the authors might want to provide comparisons with the more recent work Consistent Diffusion Meets Tweedie which extends the Ambient Diffusion idea to the setting of noisy samples. \n3. For some scientific applications, there is no available clean data at all. The proposed algorithm needs some clean data to initialize the EM and this could be limiting for such applications.\n4. One issue with the proposed method is that it relies on solving inverse problems with diffusion models. This problem is computationally intractable and for that reason, there has been a large body of research papers proposing approximations to the conditional score. The authors rely on such an approximation (DPS) which is known that doesn't truly offer samples from the posterior unless the data distribution is very simple (e.g. Gaussian).\n5. One issue with this method seems to be the training speed. Even for converge to a local optimum, EM typically requires many steps. Each step here is a new training of a diffusion model. Prior works on learning diffusion models from corrupted data only required a single training.\n6. For some corruption types, it is impossible to ever reconstruct the true distribution from noisy observations. The authors do not discuss how the proposed algorithm would work in such cases.\n7. Minor: There is a block that needs to be deleted from the NeurIPS checklist section.\n\nQUESTIONS:\nSee the Weaknesses Section above. I am willing to further increase my score if my concerns are properly addressed.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}