GSURE-Based Diffusion Model Training with Corrupted Data
Authors: Bahjat Kawar
Noam Elata
Tomer Michaeli
Michael Elad
Venue: Accepted by TMLR ()

Abstract: Diffusion models have demonstrated impressive results in both data generation and downstream tasks such as inverse problems, text-based editing, classification, and more. However, training such models usually requires large amounts of clean signals which are often difficult or impossible to obtain. In this work, we propose a novel training technique for generative diffusion models based only on corrupted data. We introduce a loss function based on the Generalized Stein’s Unbiased Risk Estimator (GSURE), and prove that under some conditions, it is equivalent to the training objective used in fully supervised diffusion models. We demonstrate our technique on face images as well as Magnetic Resonance Imaging (MRI), where the use of undersampled data significantly alleviates data collection costs. Our approach achieves generative performance comparable to its fully supervised counterpart without training on any clean signals. In addition, we deploy the resulting diffusion model in various downstream tasks beyond the degradation present in the training set, showcasing promising results.

Published in Transactions on Machine Learning Research (04/2024)
GSURE-Based Diffusion Model Training with Corrupted Data
Bahjat Kawar"
bejjot..wwrrSss..eemiion...
Deportment of Computer Science
Noam Elata'
Department of Electrical and Computer Engineering
Tomer Michaeli
tomer.m Comeer.@@e.techhioon.coo
Deportment of Electriool and Computer Engineering
Michael Elad
Aacd@ec.teccmion.co
Department of Computer Science
Technion Israel Institute of Technology, Haifa, Israel
Reviewed on OpenReview: https: // openreview. net/ forum? id= BRL JMuaJJ
Abstract
Diffusion models have demonstrated impressive results in both data generation and down- stream tasks such as inverse problems, text-based editing, classification, and more. However, training such models usually requires large amounts of clean signals which are often difficult or impossible obtain. In this work, we propose a novel training technique for generative diffusion models based only on corrupted data. We introduce loss function based on the Generalized Stein's Unbiased Risk Estimator (GSURE), and prove that under some conditions, it is equivalent to the training objective used in fully supervised diffusion models. We demonstrate our technique on face images as well as Magnetic Resonance Imaging (MRI), where the use of undersampled data siguificantly alleviates data collection costs. Our approach achieves generative performance comparable to its fully supervised counterpart without training on any clean signals. In addition, we deploy the resulting diffusion model in various downstream tasks beyond the degradation present in the training set, showcasing promising results!
1 Introduction
Denoising diffusion probabilistic models (DDPMs) (Sohl-Dickstein et al. 2015
Ho et al.
2020 Song & Ermon
2019), or diffussin models for short, are a family of generative models that has recently
risen to prominence.
They have achieved state-of-the-art performance in image generation (Song et al.
2020b
Vahdat et al. 2021
Dhariwal & Nichol
2021 Rombach al.) 2022 Kim ct al., 2022 as well as impressive
generative modeling
capabilitiie in other modalities (Ho et. al. 2022 Singer al.) 2023 Kong et al.
2021
Popov et. al.
2021
Gong et al.
2023
Li et al.
2022
Tevet et al. 2022), including protein structures
Watson et al. 2022
Qia
et al. 2022
Schneuing et al.
2022
Yim et al. 2023 and medical data
(Sog et al.
2023
Chung & Ye
2022
Jalal et al.
2021
Xie & Li
2022
Chung et al. 2023 Li et al. 2023 Adib et al.
2023
The prowess and
flexibility of DDPMs have enabled
their profound impact on downstream applications
Kawar et al.
2022
Theis et al.
2022 Blau et al.
2022
Pinaya et al.) 2022 Wyatt et al. 2022
Kawar al.
2023
Zimmermann
et al. 2021
* EEqual contribution.
1 'Qur code available

---

Published in Transactions on Machine Learning Research (04/2024)
No degradation
p=0.2, 0000
p=0. 4, 0000002
p=0.6, o0=0.03
Figure 1: Training sets samples of the different degradation settings in CelebA (Liu et al. 2015 experiments.
Training a diffusion model to learn an unknown data distribution a complex task. It usually requires training parameter-heavy neural networks on large amounts of pristine data. For instance, diffusion models' success in image generation was in part enabled by large curated datasets, containing millions even billions of images (Deng et al. 2009 Schuhmann et al., 2022). However, such large-scale datasets of pristine samples may often be expensive, difficult, or even impossible to obtain, especially in the medical domain (Mullainathan & Obermeyer 2022). Interest. in the degraded data setting has risen in recent years Xiang et al. (2023); Aali et al. (2023); Daras et al. (2024), ; yet existing solutions address only specific cases. In this work, we present GSURE-Diffusion a method for training generative diffusion models based on data corrupted by linear degradations and Gaussian noise. This setting can make data collection for deep learning significantly faster and less expensive.
DiffR--iff operates on a datasets of noisy linear measurements of signals, and assumes the signal acquisition process is randomized within a fixed general structure, which is the case in many real-world applications. In this setting, we present a novel loss function to learn the underlying data distribution. First, we use the Singular Value Decomposition (SVD) of the degradation operators to decouple the measurement equation, following DDRM (Kawar et al. 2022). This transformation simplifies the degradation into a masking operation. Then, add synthetic noise to the SVD-transformed measurements, likening them to the noisy samples used in the DDPM (Ho et al. 2020 framework. Finally,
the ensemble version of the Generalized Stein's Unbiased Risk Estimator (GSURE) (Aggarwal et al., 2022: Eldar 2008) to learn to denoise samples without access to ground-truth clean signals. Simply put, GSURE's mathematical formulation allows us to makes use the undamaged data within the corrupted image. Our proposed GSURE-basod loss function general to randomized linear measurement settings, and we prove its equivalence to the fully supervised denoising diffusion loss under some conditions.
To empirically evaluate technique, apply it downsized grayscale version of CelebA (Liu al. 2015 a dataset of celebrity face images. We train missing patches, and compare its generative output with an oracle model that trained on the full clean images. We observe that GSURE-Diffusion results in comparable generative performance to the oracle, despite having trained solely on corrupted data.
model noisy images with randomly
Furthermore, we use GSURE-Diffusion for Magnetic Resonance Imaging (MRT), ubiquitous medical imaging modality providing vital diagnostic information. Generative models for MRI are usually trained on datasets of fully sampled MR images, which can be expensive to obtain. In contrast, train generative model noisy undersampled data obtained from accelerated MRI scans. By utilizing GSURE-Diffusion, we train model with performance comparable to an oracle version while significantly reducing the time and resources required to collect the training dataset. Then, we showcase the flexibility of the generalive diffusion model we obtain. We use this model for accelerated MRI reconstruction, extending its applicability to acceleration factors beyond those encountered during training. Morcover, this model may serve as a foundational framework for addressing various tasks. We demonstrate its capabilities in MRI reconstruction for different subsampling strategies and uncertainty quantification, and compare its results with oracle version.
2

---

Published in Transactions on Machine Learning Research (04/2024)
No degradation
p=0.2, 0.001
p=0. 4, 0002
p=0.6, 00=0.03
Figure Generated samples (with 50 DDIM Song al. 2020a) steps) from models trained on different. degradation settings in CelebA (Liu et al. 2015 experiments.
In conclusion, we present GSURE-Diffusio,, novel method for training generative diffusion models based on corrupted measurements the underlying data. We demonstrate the capabilities of the method through several experiments, and show its applicability to real-world problems. We hope that DSURE-Diflusion will facilitate future work on generative modelling for challenging settings, generalizing for more complex scenarios and various modalities.
2
Background
2.1 Denoising Diffusion Probabilistic Models
Denoising Diffusiin Probabilistic Models (DDPMs) (Ho et al. 2020) are a family of generative models that learn a distribution (x), approximating a data distribution (xx) from a dataset of samples. DDPMs follow a Markov chain structure Xy + -x--1 + .... + X1 - Xg that reverses a forward noising process from be x, and the intermediate variables x₁ are defined by
Xg to Xy. In the forward process, Xg is (x, (x,,,,,, usually chosen to be simple Gaussian / (V1 -ix B.T). This leads to useful property, q" (xxx/xx)) = N (Vazx. (1 - -a)) I) with â = IT= (1 -1-1100000 Bc), which facilitates model training. In the reverse and more challenging process, the learned distribution // (xx-1/x;) is also modeled as Gaussian, with a learned mean dependent on neural network f (x:) and fixed (Ho et al. 2020 learned (Nichol & Dhariwal 2021 covariance.
The diffusion model p(() (xx) is trained to optimize an evidence lower bound (ELBO) on the log- likelihood objective (Sohl-Dicksteiin et al., 2015 The ELBO can be simplified into the following denoising objective:
ME 1800 (xx)
(xx()x))
(1)
where the Nt assign weights to different t (different noise levels), and the expectation is over some x, ~ q(x, Xu),
Xo (xx). Please refer to (Ho et al. 2020 Song et al. 2020a) for derivations. After training, diffusion
models synthesize data by starting with sample Xy ~ (O. 00, I), following the learned distributions Pg along
the Markov chain, sampling from each, and outputting Xo the final sample. Diffusion models have had
incredible success in image generation (Song et al., 2020b Vahdat et al. 2021 Dhariwal & Nichol 2021
Rombach et al.) 2022 Kim el al. 2022) ; as well as generation of other modalities (Ho et al. 2022
Singer
et al. 2023 Kong et al. 2021 Popov et al. 2021 Gong et al. 2023 Li et al.
2022
Tevet et al. 2022
They
have also been deployed
myriad related tasks (Kawar et al. 2022 Theis al. 2022 Blau al.
2022
Pinaya al. 2022 Wyatt al.
2022 Kawar al.) 2023 Zimmermann et al, 2021). In this work, we revisit
DDPMs and seek way to train them using only corrupted data.
3

---

Published in Transactions on Machine Learning Research (04/2024)
2.2 Generalized Stein's Unbiased Risk Estimator (GSURE)
Given noisy measurements y x x+2 z (where y, E R²) with noise ~ (0 o²T), and a function f(y) aiming to estimate x from y, Stein's 's unbiased risk estimator (SURE) (Stein 1981] is an unbiased estimator for the mean squared crror (MSE) of f(y), formulated as
E (1s(y) x12
_ = [1/(s) -11] + 20 E [V [(y)] no?
Crucially, SURE provides the ability estimate the MSE of a denoiser f(y) without access to clean signals
x. As result, many rescarchers have used SURE for unsupervised learning of denoisers (Zhang & Desai
1998
Blu & Luisier 2007 Metzler et al.; 2018 Soltanayey & Chun 2018 Nguyen et al., 2020 Jo et al. 2021
even extending to training diffusion models based on (fully sampled) noisy data (Xiang et al. 2023 Aali
et al. 2023).
In the context of inverse problems, SURE has been generalized for corrupted measurements beyond additive white Gaussian noise (Eldar, 2008). The Generalized SURE (GSURE) considers y = Hx + (where x RR" H Rmxn y, IR",, and (0, C)), and a function f(y) estimating x. In this case, GSURE provides unbiased estimate for the projected MSE:
IE (f(x) x)]| E ( XML)1 +E VHT Pf(y)] +c,
where Ht is the Moore-Penrose pseudo-inverse of H, P H' is projection matrix onto the range-space of H, XMI. ( HC and a constant that does not depend on f(y). Several works have utilized GSURE for solving inverse problems by training only on corrupted measurements (Metzler et al. 2018; Zhussip et al., 2019 Liu et al., 2020 Abu-Husseit et al., 2022p. However, when H causes significant information loss, the projected MSE stops being good proxy for the full MSE. Ensemble SURE (ENSURE) (Aggarwal et al. 2022) learns from dataset of measurements, each corrupted by a different operator H. Therefore, the expectation over the projected MSE is taken over H as well as the data and noise. This constitutes a more accurate proxy for the full MSE without relying on clean signals. In this work, we extend the ENSURE framework for training a diffusion model using corrupted data.
3
GSURE-Diffusion: Mathematical Formulation
3.1
Problem Formulation
We are interested in training a generative diffusion model that can sample from unknown data distribution gxx). However, we only have access to dataset D of corrupted measurements
y Hx+
(4)
where E RT, x E R H E Rmx*, and z~ ((, is additive white Gaussian noise (AWGN) Equation 4 refers to a single instance of an ideal image and its corresponding measurement, and more generally, different measurements y the dataset may relate to different signals x, different degradation procedures H, and different noise realizations z. We assume x, z, and H are randomly and independently sampled from their respective distributions.
In order to decouple the mathematical relationship between the observed measurements and the underlying data, we follow [Kwarr et al. 2022) and utilize the singular value decomposition (SVD) of H.
= -UVVV
(5)
where E Rm and V E x are orthogonal matrices, and Σ E R is a rectangular diagonal matrix containing the singular values of H. We define x V x, y EV y, and T.Z Using these definitions and the SVD, Equation becomes
Px +
(6)
2Qur method also handle anisotropic uncorrelated noise. We only consider AWGN simplify notations.
4

---

Published in Transactions on Machine Learning Research (04/2024)
where P= is diagonal subsampling matrix with zeroes and ones, and z ~ NO 00 E constitutes anisotropic uncorrelated Gaussian noise.
We make the following assumptions on the training dataset D: (i) The sampling matrices H and noise levels To are known; (ii) All matrices H share the same right-singular vectors ; and (iii) The different H across the dataset jointly cover the signal space IR", Le., E[ is a positive definite matrix p These assumptions are satisfied in many real-world applications such as Magnetic Resonance Imaging (MRI) measurements are acquired in similar fashion for all data points (upholding (fi)), and the subsampling pattern can be randomly chosen for each point (upholding (iii).. H and do are derived from the physics of the signal acquisition procedure, thus upholding (i).
Under the transformed measurement equation presented in Equation 6 we aim to train a generative model for x, which can easily translate to x through the orthogonal transformation x = Vx.
3.2 GSURE-Based Denoising Diffusion Loss Function
In order to train diffusion model for x, we aim to obtain noisy training samples & that satisfy the marginal distribution q" (x, = N (Vā,x, (1 āe) I), as in traditional diffusion models. However, we only have access to corrupted measurements y as in Equation 6 For a given i, we perturb these measurements with additional noise according to
8 = Vorg + (11 - D''DD) E c
where E, ~ N (0, I) is independently sampled. Intuitively, Vayy includes noise with a diagonal covariance atgEEEET, We increase the noise level each entry by an appropriate amount reach a variance of - az in all entries. Increasing the noise to the desired level is possible as long as ((1- āz) I - ( is a positive-eemi-defiitte (PSD) matrix. Because ā₂ is monotonically decreasing w.r.t t, by setting the beginning of the noise schedule such that ((1 ā,) I a is PSD, we ensure that this covariance matrix is PSD for all timesteps. This way, we obtain samples X, suitable for training a diffusion model, as they follow the marginal distribution
<xxx/x,)) N NN::NN(Vaaa (Va,Px, (1-a)) 1))
This resembles the ideal distribution of training samples 9 (xx|x), differing only in the mean value for entries dropped by P. In the following, we derive loss function that uses x, satisfying Equation 8) and utilizes an expectation over 8, 2, and P yielding samples X, ~ [Ep|g(X//x. P)]. This results in an estimate for denoising ideal samples from q" (xx (x). The estimate assumes that the model has the ability to generalize for samples from q² (x,, x) = (xx X P = I), despite having trained only on signals with an undersampled P. We validate our model's ability to denoise samples from q" (x: in ssubeection D.3
Idcally, we would like to train diffusion model f/())
(X,) (xx) using the traditional denoising diffusion loss function in Equation I However, as we only have access to undersampled measurements, we consider weighted expected projected MSE objective (similar to ENSURE (Aggarwal et al. 2022 ):
Σ E WP (xx) - x) 1){?
where the expectation is taken over X ~ (xx,/x, P), x~ ((x,, P is independently sampled and W E[P]-á 3 > (positive definite). The weight matrix W placed in Equation 9 for balancing the effect of the
projections P, in the case where elements of P do not have equal probability. In practice, this expectation is realized through y sampled from the datasct D and Equation
Proposition 3.1. For x q(x), & = V x, X, sampled from Equation 8 and the diagonal weight matrix
W -E (positive definite), if P and (x)) -x) x) are statistically independent, then Equation 9
equals Equation 1.
UUdder these notations, E(P) is measured for a fixed V and the values E are necessarily ordered.
5

---

Published in Transactions on Machine Learning Research (04/2024)
We place the proof in Appendix A The proof relies the aforementioned assumptions made in ENSURE (Ag garwal et al. 2022]. We assess the validity of this assumption for our trained networks in subsection D).4 and find that the assumption approximately holds for half the range of timesteps used in the denoiser. Nevertheless, even though the assumption is violated for some timesteps, the algorithm's performance remains similar the oracle model, as shown Figure 4 The expected projected MSE term in Equation measures the squared error of the denoiser (x.) only in entries kept by P. This fact makes the loss easier to measure, as we do not have access to the entries dropped by P. However, we still cannol accurately measure this loss, becauss we lack access to noiseless signals Px. To mitigate this, we utilize GSURE to estimate Equation using only X, with the loss
Σ BB
WP
f (xi)
Xx
+ +2 (Vx, PW2 (x))) +
(10)
where is constant that does not depend the same random variables from Equation 9
and x is scalar hyperparameter. The expectation is over
Proposition 3.2. For x g(x), x = V x, X₁ sampled from Equation and X = 1 - ax, it holds that Equation 10 equals Equation g
We place the proof in the Appendix A Proposition 3. and Proposition 3.2 present principled method to train a denoising diffusion model based only on corrupted data y. By minimizing the loss function in Equation 10 we obtain a trained diffusion model that can be utilized in the same fashion as fully supervised one, for both generation and downstream applications.
When our proposed training scheme applied in practice, we apply the following modifications for practical considerations: First, the expectation term in Equation 10 is replaced by an average over training batches. Second, X is replaced by y to alleviate the high variance of the loss function at little to cost in terms of bias. Lastly, the divergence term calculated using the Hutchinson's trace estimator, which an unbiasod Monte Carlo estimator (Ramani et al. 2008 Hutchinson) 1989). While this estimator rotuues some noise into equation, we find that the estimation error is negligible when using numerically stable derivative calculation. We expand upon these pragmatic implementation details in Appendix D
4 Experiments
In the following, we demonstrate the capabilitics of our mothod for training a diffusion model using corrupted data. To obtain corrupted data, we simulate several corruptions on datasets containing clean images. Then, we train a diffusion model based only on the corrupted data, and compare its results against an oracle model (with identical training hyperparamcters) which is trained the pristine data with the traditional diffusion loss function from Equation 1
Table 1: FID Heusel et al. 2017) results for diffusion models trained on increasing levels of degradation for 32 x 32-pixel CelebA (Liu et al. 2015) images, with different DDIM (Song et al. 2020a) steps at generation time. Models were trained with (lop) or without (bottom) GSURE- Diffusion, on degraded data.
Training Scheme Data Degradation
10 Steps
20 Steps
50 Steps
100 Steps
No degradation (oracle)
21.99
13.09
08.15
06.84
p 0.2, σo 0.01
86.25
108.56
119.30
123.08
Regular
p 0.4, do = 0.02
216.74
229.28
236.61
240.03
p = 0.6, do = 0.03
273.62
279.17
280.89
281.60
p 0.2, do 0.01
18.77
12.25
08.84
08.82
GSURE-Diffusion
p = 0.4, do = 0.02
19.26
14.98
14.03
15.14
p = 0.6, do 0.03
34.51
27.74
26.42
28.31

---

Published in Transactions on Machine Learning Research (04/2024)
Mask
Zero-Filled
GSURE-Diffusion
Oracle
Ground-Truth
PSNR: 28.60d
PSNR: 30.09d
PSNR: 30.93dt
PSNR: 32.01dB
PSNR: 32.95dB
PSNR: 93.9
PSNR: 25.90dB
PSNR: 55dB
PSNR: 27.15dB
PSNR: d7
PSNR: 26 33dB
PSNR: 28 23dB
Figure 3: Accelerated MRI reconstruction results for R= and og 0.01.
4.1 Human Face Images
To empirically evaluate GSURE-Diflusion, we apply it on x 32-pixel grayscale face images from CelebA (Liu et al. 2011)). We simulate corrupted measurement process by splitting the images into 4-pixel non- overlapping patches, and randomly crasing each patch with probability p. We also perturb the data with AWGN with standard deviation dg. This degradation matches
assumptions in subsection 3.1 (see We adapt the U-Net. (Ronneberger et al. 2015) architecture from DDPM (Ho et al. [0200] to match the image dimensions, and train diffusion models for increasing levels of degradation on the CelebA training set. Training hyperparameters and more details are provided Appendix C
Appendix B), and we provide samples of it in Figure 1.
After training, we generate images from the models using the deterministie DDIM (Song 2020a sampling schedule. We measure the generative performance using the Fréchet Inception Distance (FID) (Heusel et al. 2017) between 10000 generated images and the CelebA validation set. As can be seen in Table 1 and Figure 2 our GSURE-Diffusion models achieve generative performance comparable to the oracle model, despite having trained only on corrupted data. As expected, the performance deteriorates when more severe degradations are applied to the training data, We note that in the few timesteps regime, the oracle model, possibly due to the regularization effect of training on corrupted measurements. However, once we increase the number of timesteps used for generation, the oracle model benefits from the more fine-grained process, whereas GSURF-Diffusie struggles to do
slightly outperforms
7

---

Published in Transactions on Machine Learning Research (04/2024)
Denoising Mean Squared Error
Noisy
GSURE-Diffusion
Oracle
Oracle GSURE-Diffusion
10
10
1-20
Mean Squared
10-4
10-1
-
10
800
600
400
200
Diffusion Tiieesep
Figure 4: Left: Denoising MSE (on fully sampled noisy images) for the SEURR-Diffusion and oracle models across diffusion timesteps. Right: Qualitative denoising examples for both models.
4.2 Magnetic Resonance Imaging (MRI)
Magnetic Resonance Imaging (MRI) is ubiquitous non-invasive medical imaging modality that can provide life-saving diagnostic information. MRI measurements are obtained in the Fourier spectrum (also called k-space) of an object with magnetic fields. However, measuring the entire k-space can be time-consuming and expensive. Therefore, inforring the scanned object image basod on partial, possibly randomized and noisy, k-space measurements from accelerated MRI scans highly relevant and challenging inverse problem,
drawing considerable research attention (Wang et al. 2016 Hammernik et al. 2018 Lee et al. 2018 Han
et al. 2019 Weiss al. 2021 Wang et al. 2022
The accelerated MRI procedure satisfies the assumpLions we introduced in subsection 3.1 (see Appendix B with the discrete Fourier transform as VT, and the random subsampling mask as Σ. We use this fact and train a generative diffusson model for MRI based solely on accelerated scans. We train on 24, 853 scanned slices from the fastMRI (Knoll et al. 2020 Zbontar al. 2019) single-coil knee MRI dataset, center-cropped to spatial size of 320 320. The accelerated MRT subsampling process is simulated following (Jalal et al., 2021 with an acceleration factor R = 4, randomized subsampling of high frequencies, and AWGN with 0.01. To facilitate training, real and imaginary elements of complex numbers are treated separate input/ /outptt channels. We use the same U-Net (Ronneberger et al. 2015 Ho et al. (020)) architecture from our CclebA (Liu et al.) 2015) experiments, slightly modificd to match the data dinensions, and train a diffussion model on the corrupted measurements. A separate oracle model is trained with the same hyperparameters (detailed in Appendix C) on the fully sampled data.
To evaluate the validity of our approach, we measure the mcan squared error (MSE) of both models in denoising 1024 fully sampled MR images from the fastMRI [Knlll et al. 2020 Zbontar et al. 2019 validation set for different diffusion timesteps. Tn Figure 4 we observe that the denoising ability of GSURF-Diffusin resembles that of the oracle model. This supports our claim that GSURE-Diffusion can be a suitable replacement for the oracle model despite having trained exclusively on corrupted data. To test this claim in real-world applications, we substitute the oracle model for its GSURF-Diffusion counterpart in accelerated MRI reconstruction restoring MR images from noisy subsampled versions. We use the gencral-purposo disfusion--ased inverse problem solver DDRM (Kawar al. 20222 for this task with r 0 and 100 steps. In Figure 3 observe that ESURE-Diffusion achieves similar performance to the oracle for R 4. In all MRI experiments, we also present the "zero-filled" result (showing the MR image with zeroes in the missing frequencies) as a baseline.
Moreover, since we train a foundational MRI generative model, it is not restricted to the degradation setting present in its training data. For instance, the model can generalize well for higher acceleration factors, as we show in Figure 5) GSuRE-Diffusion can also be utilized to reconstruct MR images corrupted by subsampling masks of different characteristics, such as 1-dimensional Gaussian random sampling and variable density
8

---

Published in Transactions on Machine Learning Research (04/2024)
Mask
Zero-Filled
GSURE-Difusion
Oracle
Ground-Truth
PSNR: 26.B
PSNR: 28 23dB
PSNR: 28.64 28.64dB
R=6
PSNR: 76dB
PSNR: 27 2 dB
PSNR: 28.3 dB
R-8
PSNR: 25.57dB
PSNR: 26. 14dB
PSNR: 26.49dB
R=10
PSNR: 21.79UB
PSNR: 21.94IB
PSNR: 22.4 JB
R=12
Figure 5: Accelerated MRI reconstruction results for e {6, 8, 10, 12] and op 0.01. GSURE-Diffusion can generalize well across different acceleration factors.
Poisson disc sampling, as we show in Figure 6. Furthermore, as a generative model, GSURE-Diflusion can provide uncertainty estimates for its outputs. We follow (Chung Ye 2022) and quantify the uncertainty using the standard deviation of 8 stochastic outputs made by the model. We add synthetic Gaussian noise MR images with do = 0.4, and show uncertainty quantification results using
and the oracle model in Figure 7 This uncertainty quantification technique can potentially aid medical practitioners, providing clues towards anomalous regions in MRI scans. These results present evidence that generative model trained on corrupted data can be deployed in various applications. By loosening the requirenents on the quality of the training data, we significantly reduce the cost of data acquisition for model training.
5 Limitations
While GSURE-Diffusion achieves impressive results, it suffers from few limitations. First, the assumptions over the available dataset and its acquisition procedure listed in Isubecction 3.1 and subsection 3.2] which are satisficd in our example for accelerated MRI, may not always hold in other real-world scenarios. Sccond, if the measurement noise has high variance in at least one entry, the minimum noise level in the diffusion process, a1, would also need to be high. This may lead to poor performance, as the final step the generative diffusion process will need to clean significant noisc. Third, if the distribution of P in the dataset is heavily biased towards certain regions, has high variance in terms of P's rank, this can break the diffusion model's ability to infer P from X,, and thus degrade the generative performance.

---

Published in Transactions on Machine Learning Research (04/2024)
Mask
Zero-Filled
GSURE-Difusion
Oracle
Ground-Truth
PSNR 26.6
PSNR: 28.0.dB
PSNR
ddB
1D Gaussian
PSNR: 20.72dB
PSNR: 21.3
PSNR: 21.59BB
Poisson Disc
Figure MRI reconstruction results for 1-dimensional Gaussian random sampling (R= 8) and variable density Poisson disc sampling (R 15), both with og 0.01. Our method generalizes well for different sampling schemes.
Original
Noisy
GSURE Mean
GSURE std
Oracle Mean
Oracle std
0.75
0125
0.075
6.42
9.59
924
06
Figure Uncertainty quantification for MR image denoising with GSURE-Diffusion and oracle models. Means and standard deviations are calculated for 8 stochastic diffusion reconstructions.
The latter two limitations can be mitigated in future work by designing a diffusion model architecture that is explicitly aware of the "mask" P for each input, and can handle X, with diagonal covariance with different values in the main diagonal. Similar ideas (input masks and diagonal covariance) have been suggested fully supervised diffusion modeling literature (Bao al., 2022 Gao al. 2023), and would be interesting explore in the corrupted data setting.
6 Related Work
There has been & rich vein of works on unsupervised learning from datasets of corrupted data (Lehtinen)
et al. 2018 Batson & Royer 2019 Hendriksen et al. 2020 Chen al. 2022). including several SURE- and
SeUREE-based approaches (Soltanayev & Chun 2018 Nguyen et al. 2020 Jo et al. 2021 Metzler et al. 2018
Zhussip et al. 2019 Aggarwal et al.] 2022 Abu-Husscin et al. 2022 Liu et al. 2020). While they achieve impressive results, these efforts are mostly focused on learning a specific task. In contrast, our approach learns a foundational generative model, making it suitable for a wide range applications. While generative modeling mothods which learn only using corrupted data have been proposod in the past (Mattoi Frellsen 2019 Cheng-Xinn Xian Li al.] (2019), our method is the first provide a holistic solution for learning from data corrupted beyond missing pixels, with general linear degradations.
10

---

Published in Transactions on Machine Learning Research (04/2024)
Recent advances in ceffusion-based generative modeling (Ho et al. 2020: Dhariwal & Nichol 2021 have
enabled subsequent work to adapt these models for medical imaging (e.g., MRI) (Song et al.) 2023 Chung
& Ye 2022; Jalal et al. 2021 Xie & Li 2022 Chung et al. 2023 These diffusion models can serve a multitude of tasks, but can be expensive to train as they roquire fully sampled noiscless training data. Notably,
DDM² (iiang et al. 2023 and the concurrent work of SURE-Score (Aali et al., 2023) offer ways to train a diffusion model based on noisy data, which is often the case in practical settings. Similarly, a concurrent work by Daras al. (2024) offers a method for training diffuusion models based on data with missing pixels, showing improved results for downstream inpainting tasks. However, collected data can also be undersampled or corrupted by other transformations. Our proposed framework is more general, as it can handle both Gaussian noise and general linear corruptions.
Diffusion models have permeated various research areas, including online decision making (Hsieh et al. 2023 In that context, obtaining full noiseless data may often be impossible. The authors of (Hsich al. 2023 suggest a diffusion loss function that learns diffusion-bascl prior from noisy data with missing elements, similar to an image inpainting problem. They note that their proposed loss function could be of independent. interest in future work. Our loss function closely resembles theirs, albeit generalizing for linear corruptions beyond inpainting by utilizing the singular value docomposition (SVD).
7 Conclusion
We have introduced GSURE-Diffusion, technique for training generative diffusion models based on data corrupted by linear degradations and additive Gaussian noise. Using the SVD of the degradation operator, GSURE Eldar 2008), and an ensemble of degradations (Aggarwal et al. 2022), we introduce a novel training scheme that approximates the underlying data distribution from corrupted measurements. We perform experiments on CelebA (Liu al. 2015) to demonstrate the validity of our proposed framework. Additionally, we show the applicability of GSURE- to real-world problems by training a model on accelerated (undersampled and noisy) MRI scans from fastMRI (Zbontar et al. 2019 Knoll et al. 2020 We then use the resulting diffusion model to address several downstream tasks.
We hope that our proposed framework will enable future work to train generative models on similar problems such as multi-coil accelerated MRI, sparse-view Computed Tomography (CT), and more. Future work may also address additional challenging measurement acquisition scenarios, as dictated by the data modality. We emphasize the fact that our results are obtained on simulated corruptions. More extensive experiments on real data should be conducted before GSURE-Diffusion can be relied upon for diagnostics in clinical settings.
Acknowledgements
This research was partially supported by the Council For Higher Education Planning Budgeting Committee. Data used in the preparation of this article were obtained from the NYU fastMRT Initiative database Zbbontar ct al. (2019); Knoll et al. (2020). As such, NYU fastMRI investigators provided data but did not participate in analy mnaisss or writing of this report. A listing of NYU Mast investigators, subject to updates, can be found at fastmri.med www.uuiiimdd.......d.... edu The primary goal of fast is to test whether machine learning can aid in the reconstruction of modical images.
11

---

Published in Transactions on Machine Learning Research (04/2024)
References
Asad Aali, Marius Arvinte, Sidharth Kumar, and Jonathan 1 l'Tamir. Solving inverse problems with score-based generative priors learned from noisy data. arviv preprint or arXie:2305.0110 016606 2023.
Shady Abu-Husscin, Tom Tirer, Se Young Chun, Yonina Eldar, and Raja Giryes. Image restoration by deep projected GSURE. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3602-3611, 2022.
Edmond Adib, Amanda Fernandez, Fatemeh Afghah, and John Jeff Prevost. Synthetic ecg signal generation using probabilistic diffusion models. arXiv preprint ar Xrx::22333.02 02430.7....5 2023.
Hemant Kumar Aggarwal, Aniket Pramanik, Mancesh John, and Mathews Jacob. ENSURE: A general approach for unsupervised training of deep image reconstruction algorithms. IEEE Transactions on Medical Imaging, 2022.
Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. In International Conference on Machine Learning, pp- 1555-1584, PMLR, 2022.
Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision. In International Conference on Machine Learning, pp. 524-533. PMLR, 2019.
Tsachi Blau, Roy Ganz, Bahjat Kawar, Alex Bronstein, and Michael Elad. Threat model-agnostic adversarial defense using diffusion models. arXiv preprint arXiw:2207.0808) 2022.
Thierry Blu and Florian Luisier. The sure-let approach to image denoising, IEEE Transactions on Image Processing, 16(11):2778 22778727766 2007.
Dongdong Chen, Julián Tachella, and Mike E Davies. Robust equivariant imaging: fully unsupervised framework for learning to image from noisy and partial measurements. In Proceedings of the IEEE/CVE Conference Computer Vision and Pottern Recognition, pp. 5647 5656, 2022.
Steven Cheng-Xian Li, Bo Jiang, and Benjamin Marlin. Misgan: Learning from incomplete data with generative adversarial networks. arXiv e-prints, pp. arXiv-1902, 2019.
Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated MRI. Medical Image Analysis, 80:102479, 2022.
Hyungjin Chung, Dohoon Ryu, Michael T McCann, Mare L Klasky, and Jong Chul Ye. Solving 3D inverse problems using pre- tre-trained 2D diffusson models. In IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2023.
Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alex Dimakis, and Adam Klivans. Ambient diffusion: Learning clean distributions from corrupted data. Advances in Neural Information Processing Systems, 36, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs image synthesis. Advances in Neural Information Processing Systems, 34:8780 8794, 2021.
Yonina C Eldar. Generalized SURE for exponential families: Applications to regularization. IEEE Transactions on Signal Processing, (7)2):771-48) 2008
Shanghua Gao, Pan Zhou, Ming- Ming Cheng, and Shuicheng Yan. Masked diffusion transformer strong image synthesizer. arXiv preprint Xin:2333 14989, 2023.
12

---

Published in Transactions on Machine Learning Research (04/2024)
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. DiffuSeq: Sequence to sequence text generation with diffusion models. In International Conference on Learning Regresenlations, 2023.
Kerstin Hammernik, Teresa Klatzer, Erich Kobler, Michael P Recht, Daniel K Sodickson, Thomas Pock, and Florian Knoll. Learning variational network for reconstruction of accelerated mri data, Magnetic resonance in medicine, 79(6):3055 3071, 2018.
Yosco Han, Leonard Sunwoo, and Jong Chul Ye. {k}-space deep learning for accelerated mri. IEEE Irunsactions on medical imaging, 39(2):377-38) -86, 2019.
Allard Adriaan Hendriksen, Daniël Maria Pelt, and K Joost Batenburg. Noise2inverse: Self-supervised deep convolutional denoising for tomography. IEFE Transactions on Compulational Imaging, 6:1320-1335, 1335,, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rulc converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, volume 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 13:6840-6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, al. Imagen video: High definition video generation with diffusion models. arviv preprint arXiv:2210. 02303, 2022.
Yu-Guan Hsich, Shiva Prasad Kasiviswanathan, Branislav Kveton, and Patrick Blöbaum. Thompson sampling with diffusion generative prior. arXiv preprint (rXiv:2301.05182, 2023.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in PaatisticsSimulattinn and Computation, 18(3):1059 1006,10766 1989.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 34:14938-14999 14954, 2021.
Yeonsik Jo, Se Young Chun, and Jonghyun Choi. Rethinking deep image prior for denoising, In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5087-5096, 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems, 2022.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Conference Computer Vision and Pottern Recognition 2023, 2023.
Dongjun Kim, Yeongmin Kim, Wanmo Kang, and II-Chul Moon. Refining generative process with discriminator guidance in based diffusion models. arXiv preprint arXiv:2911.1700 2022.
Florian Knoll, Jure Zbontar, Anuroop Sriram, Matthew J. Muckley, Mary Bruno, Aaron Defazio, Marc Parente, Krzysztof J. Geras, Joe Katsnelson, Hersh Chandarana, Zizhao Zhang, Michal Drozdzalv, Adriana Romero, Michael Rabbat, Pascal Vincent, James Pinkerton, Duo Wang, Nafissa Yakubova, Erich Owens, C. Lawrence Zitnick, Michael P. Recht, Daniel K. Sodickson, and Yvonne W. Lui. fastMRI: publicly available raw k-space and DICOM dataset of knee images for accelerated MR image reconstruction using machine learning. Radiology: Artificial Intelligence, 2(1):e190007, 2020.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Represenlations, 2021.
Dongwook Lee, Jaejun Yoo, Sungho Tak, and Jong Chul Ye. Deep residual learning for accelerated mri using magnitude and phase networks. IEFF Transactions on Biomedical Engineering, 65(9):1985-199) 1995, 2018.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. In International Conference on Machine Learning, pp. 2963 2974. PMLR, 2018.
Huayu Li, Gregory Ditzler, Janet Roveda, and Ao Li. DeScoD-ECG: Deep score based diffusion model for ocg baseline wander and noise removal. JEEE Journal of Biomedical and Health Informatics, 2023.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-LM improves controllable text generation. In Advances in Neural Information Processing Systems, 2022.
Jiaming Liu, Yu Sun, Cihat Eldeniz, Weijie Gan, Hongyu An, and Ulugbek S Kamilov. RARE: Image reconstruction using deep priors learned without groundtruth. TEEE Journal of Selected Topics in Signal Processing, 14(6):1088 1099, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3730 3738, 2015.
Alexandee Mattei and Jes Frellsen. Miwae: Deep generative modelling and imputation of incomplete data sets. In International conference on machine learning, pp. 4413-4425 PMLR, 2019.
Christopher A Metzler, Ali Mousavi, Reinhard Heckel, and Richard G Baraniuk. Unsupervised learning with stein's unbiased risk estimator. arXiv preprinl Xiv: 1805. 10531, 2018.
Sendhil Mullainathan and Ziad Obermeyer. Solving medicine's data bottleneck: Nightingale open science. Nature Medicine, 28(5):897-895 899, May 2022.
Han V Nguyen, Magnus O Ulfarsson, and Johannes R Sveinsson. Hyperspectral image denoising using sure-based unsupervised convolutional neural networks. IEEE Transactions on Geoscience and Remote Sensing, (333333369--333 3382, 2020.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Walter HL Pinaya, Mark Graham, Robert Gray, Pedro F Da Costa, Petru-Daniel Tudosiu, Paul Wright, Yee Mah, Andrew D MacKinnon, James Teo, Rolf Jager, al. Fast unsupervised brain anomaly detection and segmentation with diffusion models. In Medical Image Computing and Computer Assisted Intervenlion MICCAI 2022: 25th International Conference, Singapore, Seplember 18 22, 2022, Proceedings, Part VIII. pp. 705 -14. Springer, 2022.
Vadim Popor, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-TTS, A diffusion probabilistic model for text-to-speoch. In International Conference on Machine Learning, pp- 8599 8608. PMLR, 2021.
Zhuoran Qiao, Weili Nie, Arash Vahdat, Thomas F Miller III, and Anima Anandkumar. Dynamic- backbone protein-ligand structure prediction with multiscale generative diffusion models. arXiv preprinl 2rXiv:2209.15111, 2022.
Sathish Ramani, Thierry Blu, and Michael Unser. Monte-carlo sure: A black-box optimization of regularization parameters for general denoising algorithms. IEEE Transactions on image processing, 17(99:1540-1554 2008.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-rcsolution image synthesis with latent. diffusion models. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 10684-10695, 10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assiste Intervention- 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241. Springer, 2015.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blundell, Pietro Lió, Carla Gomes, Max Welling, Michael Bronstein, and Bruno Correia. Structure-based drug design with equivariant diffusion models. arXiv preprint (rXi::2210.13600 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In Thirty-siath Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A A-Video: Text-to-video generation without text-video data. In International Conference on Learning Representations, 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequiliriumm thermodynamics. In International Conference on Machine Learning, pp. 2265.2265. PMLR, 2015.
Shakarim Soltanayev and Se Young Chun. Training deep learning based denoisers without ground truth data. Advances in neural information processing systems, 31, 2018.
Jiaming Song, Chenlin Meng, and Stelano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020b.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems medical imaging with score-based generative models. In International Conference on Leorning Representations, 2023.
Charles M Stein. Estimation of the mean of a multivariate normal distribution. The annals of Statistics, pp- 1135 1151, 1981
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Amit H Bermano, and Daniel Cohen-Oh Or. Human motion diffusion model. arXis preprint axX1:2220.144916 2022.
Lucas Theis, Tim Salimans, Matthew D Hoffman, and Fabian Mentzer. Lossy compression with Gaussian diffusion. arXiv preprint arXiv:2206.08830 2022.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score- generative modeling in latent space. Advances in Neural Information Processing Systems, 34:11287 11302, 2021.
Shanshan Wang, Zhenghang Su, Leslie Ying, Xi Peng, Shun Zhu, Feng Liang, Dagan Feng, and Dong Liang, Accelerating magnetic resonance imaging via deep learning. In IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp. 514-517, IEEE, 2016.
Zi Wang, Chen Qian, Di Guo, Hongwei Sun, Rushuai Li, Bo Zhao, and Xiaobo Qu One-dimensional deep low-rank and sparse network for accolerated mri. IEEE Transactions on Medical Imaging, 42(1):79-90, 2022.
Joseph L Watson David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models. bioRriv, pp. 2022-12 12, 2022.
15

---

Published in Transactions on Machine Learning Research (04/2024)
Tomer Weiss, Ortal Senouf, Sanketh Vedula, Oleg Michailovich, Michael Zibulevsky, Alex Bronstein, et al. Pilot: Physics-informed learned optimized trajectories for accelerated mri. Machine Learning for Biomedical Imaging, 1(April 2021 issue):1-22 23, 2021.
Julian Wyatt, Adam Leach, Sebastian M Schmon, and Chris G Willeocks. AnoDDPM: Anomaly detection with denoising diffusion probabilistic models using simplex noisc. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognilion, pp. 650 656, 2022.
Tiange Xiang, Mahmut Yurt, Ali B Syod, Kawin Setsompop, and Akshay Chaudhari. DDM2: Self-superviodd diffusion MRT denoising with generative diffusson models. In The Fleventh International Conference Learning Representations, 2023.
Yutong Xie and Quanzheng Li. Measurement- cowpurrmmen--coodtiioneeedd denoising diffusion probabilistic model for under- sampled medical image reconstruction. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2022: 25th International Conference, Singapore, Seplember 18 22, 2022, Proceedings, Part VI, pp. Springer, 2022.
Jason Yim, Brian L Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. Se(3) diffusion model with application protein backbone generation. ar preprint arXiv:2302.0220 022777 2023.
Jure Zbontar, Florian Knoll, Amuroop Sriram, Tullie Murrell, Zhengnan Huang, Matthew J. Muckley, Aaron Defazio, Ruben Stern, Patricia Johnson, Mary Bruno, Marc Parente, Krzysztof J. Geras, Joe Katsnelson, Hersh Chandarana, Zizhao Zhang, Michal Drozizal, Adriana Romero, Michael Rabbat, Pascal Vincent, Nafissa Yakubova, James Pinkerton, Duo Wang, Erich Owens, C. Lawrence Zitnick, Michael Recht, Daniel K. Sodickson, and Yvonne W. Lui. fastMRI: An open dataset and benchmarks for accelcrated MRI, 2019.
Xiao-Ping Zhang and Mita Desai. Adaptive denoising based on sure risk. IEEE signal processing leteers, (10):265 2677 1998.
Magauiya Zhussip, Shakarim Soltanayev, and Se Young Chun. Training deep learning based image denoisers from undersampled measurements without ground truth and without image prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogrittin, pp. 10255 10264, 2019.
Roland S Zimmermann, Lukas Schott, Yang Song, Benjamin A Dunn, and David A Klindt. Score-based generative classifiers. arXiv preprint 2xXi::21100044 2021.
16


Reviews:
[{"review_id": "3hqbmvM9ri", "reviewer": "Reviewer_6gUc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As a disclaimer, I would like to stress that I am not on top of the (very large) recent literature on diffusion models.  \n\n## Strength:\n1. The manuscript is well-written and mostly straightforward. The addressed problem—training a diffusion model with degraded samples—is highly relevant, with significant practical implications, notably in medical imaging.  As a non-expert, I have found the literature review adequate for quickly placing the manuscript within the current research landscape.\n2. From my assessment, the proposed method seems to be the first to train diffusion models using degraded samples in the form $H_i x_i + \\textrm{(noise)}$. The most similar existing work is [1], which only deals with additive Gaussian noise (i.e., $H_i = Id$).  \n3. The numerical experiments are robust, and the empirical results are compelling.\n\n## Weaknesses & Questions:\n1. In equation (7), why is the term $(1-\\alpha_t) I - \\alpha_t  \\sigma_0^2 \\Sigma^\\dagger \\Sigma$ semi-definite positive?\n2. One of the conditions for the approach to work is for the set of different $H$ to \"cover\" the signal space, which is formulated as the condition that $\\mathbb{E}[P]$ is positive definite. However, I'm unsure why this condition is considered sufficient. Take the scenario where samples are in $\\mathbb{R}^{2D}$, represented as $[x1,x2]$ with $x_1, x_2 \\in \\mathbb{R}^{D}$, and there are only two possible observation operators: $H_1[x_1, x_2] = x_1$ and $H_2[x_1, x_2] = x_2$. Essentially, we observe either the first $D$ components or the last $D$ components. Here, the matrices associated are $P_1=H_1$ and $P_2=H_2$, and $V=I$, meeting the $\\mathbb{E}[P]$ positive definiteness criterion. Yet, it's unclear how one could estimate the joint distribution of $[x_1, x_2]$—particularly, how to infer the correlation structure between the first and last $D$ components when they're never (noisily) observed together.\n3. divergence term is computed by the Hutchinson estimator: some comments on this estimator that is often extremely noisy wold be appreciated.\n4. this would have been, I think, extremely interesting to run a few simulations on very low dimensional, possibly correlated Gaussian, examples. For example, why not take the clean samples generated from a low-dimensional Gaussian distribution with known covariance structure, and use some simple observation operator such as \"observation of a few coordinates\"-- this setting is entirely tractable. This would allow one to understand a bit more clearly the different trade-offs that are at play.\n\n### Minor issues:\n1. [Figure ??] --> broken ref page 5\n2. page 5: \"$\\frac{1}{ \\sqrt{\\overline{\\alpha}_t} } \\overline{x}_t$ is placed by $\\overline{y}$\": could you please give mode comments and details.\n\n\n\n\n\n\n[1]: Aali, Asad, et al. \"Solving Inverse Problems with Score-Based Generative Priors learned from Noisy Data.\" arXiv preprint arXiv:2305.01166 (2023).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript suggests training a diffusion model with only partial and noisy observations, $y_i = H_i x_i + \\textrm{(noise)}$. Here, the observation operator $H$ varies but is known, and can be represented as $H_i = U_i^\\top \\Sigma_i V$, where $V$ is a constant orthonormal matrix. This setup allows reformulating the problem using known orthogonal projection operators $P_i$, as $\\overline{y}_i = P_i , \\overline{x}_i + \\textrm{(noise)}$.  \n\nFor the diffusion model to be effectively trained, a diverse set of projections $P_i$ is required. The paper introduces a strategy to adapt the conventional training method for diffusion models to this context. A key innovation is replacing the standard mean squared error (MSE) metric, $||\\textrm{(denoiser estimate)} - \\text{(truth)}||^2$, with an unbiased estimate derived from Stein's Lemma.  \n\nThe manuscript presents two set of experiments that demonstrate the effectiveness of the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I have found the text very interesting. I am still not understanding well the point I am raising in the \"weakness\" section above regarding the condition that $\\mathbb{E}[P]$ is positive definite is enough for learning the distribution of clean samples. In any case, I think that this should be clarified more clearly in the manuscript.\n2. While not entirely crucial, I still think that running a few (easy and low-dimensional) experiments in a tractable setting as described in the previous section to be worth it to understand the method better.", "broader_impact_concerns": "no concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z0Zx4oEmXj", "reviewer": "Reviewer_vrwZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The validity of the proposed objective is theoretically justified (Prop 3.1 & 3.2).\n- The empirical results are promising, comparable with the oracle (i.e. full measurement) results on various metrics.\n- The paper clearly discusses limitations:\n    - the data assumptions may not be satisfied in real-world datasets.\n    - the performance may suffer if the noise is high variance (in some coordinates).\n    - the performance may suffer if different parts of the distribution contribute unevenly to the subsampling matrix $P$.\n\nWeaknesses / limitations:\n- For empirical results, my main concern is that the method has limited comparison and evaluation:\n  - The model is evaluated on CelebA and FastMRI, both are small datasets of grayscale images. Results on more complex data will be needed to better evaluate the of the proposed method.\n  - The proposed method is compared with the oracle only, and there are no comparison with methods in prior work. I apologize that I'm not an expert in this field so am not able to provide specific pointers on which papers to compare to. However, I think the related work section should be much more thorough, given the large body of work in diffusion models / image denoising.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes GSURE-Diffusion, a method for learning a generative model from corrupted data. The form of corruption is general, e.g. it could be linear corruption or Gaussian noises.\n\nFormally, the method is based on GSURE (generalize SURE). Let the corrupted measurements be $y := Hx + z$ for some degradation procedure $H$, the paper defines a diagonal subsampling matrix $P := \\Sigma^\\dagger \\Sigma$ where $\\Sigma$ is given by the SVD of $H$. The proposed objective is defined based on $P$.\n- The paper theoretically shows that the proposed objective is equivalent to the fully supervised (i.e. without subsampling) denoising objective (Prop 3.1 & 3.2).\n- The objective is future modified (i.e. eq (10) to eq (15)) to reduce the variance in estimation.\n\nEmpirically, the paper provides results on CelebA and FastMRI.\n- For both dataset, the paper provides the specific choices of $H$.\n- On FastMRI, the paper demonstrates that the method can:\n  - generalize to higher acceleration factors; for example, it's trained on R=4 and can generalize to R = 6,8,10,12.\n  - generalize to other types of masks; for example, it's trained on (todo) and can generalize to 1d Gaussian masks or Poisson masks.\n  - be used for uncertainty quantification, reported spatially.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I'd like to have some clarifications please:\n- Appendix D.3: when checking for generalization of an undersampled $P$ to the identity $I$, why is 45dB chosen as a threshold for PSNR?\n- Table 1: how many runs are there for each setup?\n- Fig 5: why do all masks have a masked out strip at the center?\n- Fig 7: it's unclear how the uncertainty estimates could be helpful. Currently it seems that the uncertainty is high in non-black regions, which however are regions containing information (since black is the background).\n\nChanges:\n- Please add more datasets, preferably with at least one dataset with colored images.\n- Please add comparisons to other methods (e.g. in prior work), rather than just the oracle.\n- Please consider a more thorough literature review, e.g. more discussions on prior methods on inverse problems.", "broader_impact_concerns": "There are no ethical concerns.\nFor broader impact, this work may be used for medical settings, where uncertainty quantification is a desideratum.\nThe paper has a short discussion on potentially providing uncertainty metrics, but more discussions / considerations are needed (as mentioned in my questions above).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OhGHk2mXPs", "reviewer": "Reviewer_HXZD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. Training diffusion models with corrupted data is an important research problem, and of great importance for real applications. \n2. This paper incorporates the GSURE into the diffusion training, which improves the performance of the Regular diffusion on corrupted data.\n\nWeakness: \n1. Some key assumptions are not clearly justified or explained; see details in the comments below. \n2. The matrix P closely resembles, if not exactly interpreted as, the masks for missing values. However, the paper lacks discussion or comparison with works that focus on training generative models with missing values, making it challenging to judge the specific novelty of this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces GSURE-Diffusion, a new training method for generative diffusion models based only on data corrupted by linear degradations and Gaussian noise. It employs a loss function derived from the Generalized Stein’s Unbiased Risk Estimator (GSURE). The paper offers both theoretical guarantees for the loss function and empirical validations of the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In sec 3.2, the authors assume that “The estimate assumes that the model has the ability to generalize for samples from..despite having trained only on signals with an undersampled P” I wonder how this can be verified, and are there any conditions (on date distribution and P) for this?\n\nIn prop 3.1, is W a deterministic weight matrix? If it is, then eq(9) can be simplified or rewritten to emphasize this. The current formulation of eq(9) looks like that W is also random.\n\nThe assumption needed in prop 3.1., i.e., “network’s denoising error is independent of the projection matrix P,” was assessed using Canonical Correlation Analysis (CCA) in the appendix. I am a bit confused about the assumption related to data generation, specifically how the projection matrix P, which represents missing values, is selected in practice. For instance, if the missing scheme implies that P does not depend on x (I think this is the case for sec 4.1), does this naturally imply independence in the assumption? On the other hand, if P depends on data (meaning missing not completely at random), can we still expect such independence?\n\nThe setting in Fig 1 is similar to the case with missing values (although here the authors also impose Gaussian noise). I wonder how would the authors compare with generative models when missing values are present. To name a few:\n\nMattei, Pierre-Alexandre, and Jes Frellsen. \"MIWAE: Deep generative modelling and imputation of incomplete data sets.\" International conference on machine learning. PMLR, 2019.\n\nLi, Steven Cheng-Xian, Bo Jiang, and Benjamin Marlin. \"MisGAN: Learning from Incomplete Data with Generative Adversarial Networks.\" International Conference on Learning Representations. 2018.\n\nMinor: \n- Figure ?? in sec 3.2 is missing.\n- The third paragraph of the introduction is a bit hard to follow, especially for readers not familiar with SVD-transformed measurements, Generalized Stein’s Unbiased Risk Estimator, supervised denoising diffusion loss, etc. I suggest to rewrite it a bit to give better explanations and summaries of the proposed approach.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Please include the latest recommendation of the reviewers:  it may be helpful to include a discussion on [1], which might have been a concurrent work.
[1] Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alexandros G. Dimakis, Adam Klivans. Ambient Diffusion: Learning Clean Distributions from Corrupted Data

The paper is fine otherwise.
Decision: Accept (poster)