Stochastic Forward–Backward Deconvolution: Training Diffusion Models with Finite Noisy Datasets
Authors: Haoye Lu
Qifan Wu
Yaoliang Yu
Venue: ICML 2025 poster (2025)
Keywords: diffusion models
deconvolution
ambient diffusion
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Recent diffusion-based generative models achieve remarkable results by training on massive datasets, yet this practice raises concerns about memorization and copyright infringement. A proposed remedy is to train exclusively on noisy data with potential copyright issues, ensuring the model never observes original content. However, through the lens of deconvolution theory, we show that although it is theoretically feasible to learn the data distribution from noisy samples, the practical challenge of collecting sufficient samples makes successful learning nearly unattainable. To overcome this limitation, we propose to pretrain the model with a small fraction of clean data to guide the deconvolution process. Combined with our Stochastic Forward--Backward Deconvolution (SFBD) method, we attain FID  6.31 on CIFAR-10 with just 4% clean images (and 3.58 with 10%). We also provide theoretical guarantees that SFBD learns the true data distribution. These results underscore the value of limited clean pretraining, or pretraining on similar datasets. Empirical studies further validate and enrich our findings.

Stochastic Forward-Backward Deconvolution: Training Diffusion Models with Finite Noisy Datasets
Haoye Lu" 12 Qifan Wu
Yaoliang Yu
Abstract
Due to some inherent properties, diffussin models are rela- tively easier to train. This unlocks the possibilityy of training very large models on web-scale data, which has been shown to be critical to train powerful models. This paradigm has recently led to impressive advances in image generation, as demonstrated by cutting-edge models like Stable Diffu- sion (-XL.) (Rombach et al., 2022; Podell al., 2024) and DALL-E (2, 3) (Betker et al., 2023). However, despite their success, the reliance on extensive web-scale data introduces challenges. The complexities of the datasets at such scale often result in the inclusion of copyrighted content. Fur- thermore, diffusion models exhibit a greater tendency than earlier generative approaches, such as Generative Adversar- ial Networks (GANs) (Goodfellow et al., 2014; 2020), to memorize training examples. This can lead to the replica- tion of parts or even entire images from their training sets (Carlini et al., 2023; Somepalli et al., 2023).
Recent diffusion beasusioo--baedd generative models achieve remarkable results by training on massive datasets, yet this practice raises concerns about memoriza- tion and copyright infringement. A proposed rem- edy is to train exclusively noisy data with po- tential copyright issues, ensuring the model never observes original content. However, through the lens of deconvolution theory, we show that al- though it is theoretically feasible to learn the data distribution from noisy samples, the prac- tical challenge of collecting sufficient samples makes successful learning nearly unattainable. To overcome this limitation, we propose to pretrain the model with small fraction of clean data to guide the deconvolution process. Combined with Deconvolution
our Stochastic Berward-Backweer (SFBD) method, we attain FID 6.31 on CIFAR 10 with just 4% clean images (and 3.58 with 10%). We also provide theoretical guarantees that SFBD learns the true data distribution. These results un- derscore the value of limited clean pretraining, or pretraining on similar datasets. Empirical studies further validate and enrich our findings.
A recently proposed approach to address memorization and copyright concerns involves training (or fine- fine-tuning) diffu- sion models using corrupted samples (Daras et al., 2023b; Somepalli et al., 2023; Daras Dimakis, 2023; Daras et al., 2024). In this framework, the model is never exposed to the original samples during training. Instead, these samples undergo known non-invertible corruption process, such as adding independent Gaussian noise to each pixel in image datasets. This ensures that the model cannot memorize or reproduce the original content, as the corruption process is irreversible for individual samples.
1. Introduction
Diffusion-based generative models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021a;b; 2023) have gained increasing attention. Nowadays, it is considered one of the most powerful frameworks for learning high- dimensional distributions and we have witnessed many im- pressive breakthroughs (Croitoru al., 2023) in generating images (Ho al., 2020; Song al., 2021a;b; Rombach et al., 2022; Song et al., 2023), audios (Kong et al., 2021; Yang et al., 2023) and videos (Ho et al., 2022).
Interestingly, under mild assumptions, certain non- inverrrrteele corruption processes, such as Gaussian noise injection, cre- ate a mathematical bijection between the noisy and original distributions. Thus, in theory, generative model can learn the original distribution using only noisy samples (Bora et al., 2018 8). Building on this concept, Daras et al. (2024) demonstrated that when an image is corrupted via a forward diffusion to a specific noise level a, diffusion models can recover distributions at noise levels below a by enforcing consistency constraints (Daras et al., 2023a).
'Cheriton School of Computer Science, University of Waterloo, Canada VVector Institute, Canada. Correspondence to: Haoye <haoye. <haoye.n@@waaccloo.ccm uwaterloo.ca..
While Daras et al. (2024) empirically showed that their approach could be used fine-tune Stable Diffusion XL (Podell et al., 2024) using noisy images with a heuristic consistency loss, they did not explore whether diffusion
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
model can be successfully trained solely with noisy images. Moreover, the effectiveness of the consistency loss in such scenarios remains an open question.
Learning generative models from corrupted data poses a sig- nificant challenge, the model must reconstruct the under- lying data distribution from incomplete or noisy information. In their work on AmbientGAN, Bora et al. (2018) showed that it is empirically feasible to train GANs using corrupted images. They also provided a theoretical guarantee that, with a sufficient number of corrupted samples generated by randomly blacking out pixels, the learned distribution converges to the true data distribution. Building this, Wang et al. (2023) demonstrated closely related result: un- der certain weak assumptions, if the model-generated fake samples and the corrupted true samples share the same dis- tribution after undergoing identical corruption, then the fake data distribution aligns perfectly with the true data distribu- tion. Their analysis applies to scenarios where corruption is implemented via a forward diffusion process but does not address cases where the two corrupted distributions are similar but not identical - case we explore in Prop 1 below.
In this paper, address these questions by connecting the task of estimating the original distribution from noisy samples to the well-studied density deconvolution problem (Meister, 2009). Through the lens of deconvolution theory. we establish that the optimal convergence rate for estimating the data density is O(log n) when n noisy samples are generated via a forward diffusion process. This pessimistic rate suggests that while it is theoretically feasible to learn the data distribution from noisy samples, the practical challenge of collecting sufficient samples makes successful learning nearly unattainable. Our empirical studies further validate this theoretical insight and suggest the inefficiency of the current consistency loss outside the regime of fine-tuning latent diffusion models.
To address the poor convergence rate in training diffusion models with noisy data, we propose pretraining models on small subset of copyright-free clean data effective solution. Since the current consistency loss remains inef- fective even with pretraining, we propose a new deconvolu- tion method, Stochastic Borward-Backward Deconvolution (SFBD, pronounced so sofabed), that is fully compatible with the existing diffusion training framework. Experimen- tally, we achieve an FID of 6.31 with jish 4% clean images on CIFAR-I0 and 3.58 with 10% clean images. Our theoret- ical results ensure that the learnt distribution converges to the true data distribution and justify the necessity of pretrain- ing. Furthermore, our results suggest that models can be pretrained using datasets with similar features when clean, copyright-fif data are unavailable. Ablation studies pro- vide additional evidence supporting our claims. Code for the empirical study is available at: github.com/watnie wo.SW...
Inspired by the success of training GANs using corrupted data, Daras et al. (2023b); Aali et al. (2023); Daras & Di- makis (2023); Bai et al. (2024); Daras et al. (2024) demon- strated the feasibility of training diffusion models with cor- rupted data, Notably, Daras et al. (2024) showed that when corruption is performed by a forward diffusion process, the marginal distribution at one time step determines the distri- butions at other time steps, all of which must satisfy certain consistency constraints. Building on this, they showed that if a model learns distributions above the corruption noise level, it can infer those below the noise level by adhering to these constraints. To enforce this, they introduced a consistency loss to improve compliance with the constraints, though its effectiveness was demonstrated only in fine-tuning latent diffusion models.
Outside the field of machine learning, the problem esti- mating the original distribution from noisy samples has traditionally been addressed through density deconvolu- tion (Meister, 2009). This research area aims to recover the distribution of error-free data from noise-contaminated observations. Most existing deconvolution methods are lim- ited to the univariate setting (Carroll & Hall, 1988; Zhang, 1990; Fan, 1991; Cordy & Thomas, 1997: Delaigle & Hall, 2008; Meister & Neumann, 2010; Lounici & Nickl, 2011 1; Guan, 2021), with only a few approaches extending to the multivariate case. These multivariate techniques typically rely on normal mixture models (Bovy et al., 2011: Sarkar et al., 2018) or kernel smoothing methods (Masry, 1993; Lepski & Willer, 2019). Integrating these theoretical in- sights into modern generative model frameworks remains a significant challenge. However, by reinterpreting generative models trained on noisy data through the lens of deconvo- lution theory, we can gain deeper understanding of their fundamental limitations and capabilities, as they inherently address the deconvolution problem.
2. Related Work
The rise of large diffusion models trained on massive datasets has sparked growing concerns about copyright in- fringement and memorization of training data (Carlini et al., 2023; Somepalli et al., 2023). While differential privacy (DP) has been explored as mitigation strategy (Abadi et al., 2016; Xie et al., 2018; Dockhorn et al., 2023), it often presents practical challenges. Notably, DP can require users to share their original data with a central server for training unless local devices have sufficient computational power for backpropagation.
In contrast, training on corrupted data provides compelling alternative, allowing users to contribute without exposing their original data. By sharing only non-invertible, corrupted versions, sensitive information remains on users' devices, eliminating the need to transmit original data.
2

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
A very recent study by Daras et al. (2025), using Gaussian Mixture Models, also highlights the challenge of training diffusion models with only noisy samples and shows that adding a few clean samples can significantly improve perfor- mance. The convergence of conclusions from fundamentally different approaches reinforces the findings of both works. Methodologically,, Daras et al. (2025) apply Tweedie's for- mula (consistency constraint) to recover the clean distribu- tion. In contrast, ours introduces novel deconvolution strategy, offering fresh perspective without the heavy computational cost of enforcing consistency.
has a transition kernel that matches the posterior distribu-
tion of the forward process, Paje (X,(xx) = Pt. a (Xeeeeee,. (x. (x))).
P(()))
for S é in [0,7], 7) Thus, the backward SDE preserves the same marginal distributions the forward process. Here, we represents a standard Wiener process with time flowing backward from T to O, while V log Pe (xx) denotes the score function of the distribution pr(x,) With a well-trained net- work S& (x,, 1) R lo we substitute it into Eq (3) and solve the SDE backward from XT ~ (N T). The resulting Xo then serves as an approximate sample of po-
To train Se estimate the score, let T be a predefined sampler of f E [0, T] and w(t) be a weight function. The network So can be effectively trained via the conditional score-matching loss (Song et al., 2021b):
Beyond diffusion model training, Bie et al. (2022) and Ben- David et al. (2023) demonstrate that even a small amount of clean public data can substantially reduce the sample com- plexity in differentially private (DP) estimation when learn ing from sensitive data. Although motivated by different goal, Nie al. (2022) propose DiffPuree an algorithm that leverages a pretrained diffusion model to remove adversar- warkwarrdd diffusion process.
(b))
E E E Pa Pajo
[s(()sss (x.. V log (x)
Instead, we may first train denoiser D& (x(x,, t) to estimate by minimizing (Karras et al., 2022)
ial perturbaations via While DiffPure and SFBD) share similar structure, they serve distinct purposes: DifiPure assumes well-trained model to purify data, whereas SFBD is designed to train the diffusion model itself.
(a(
E E E [w(t)]] (x,, 1) xx xoo1
(4)
I~T Po Pe
then estimate
E. [[x - X
D 1) Xt
V (x)) =
2
(5)
of
o
3. Prelimilaries
In this section we recall diffusion models, the density de- convolution problem and the consistency constraints.
3.2. Density Deconvolution Problems
Classical deconvolution problems arise in scenarios where data corrupted due to significant measurement errors, and the goal is to estimate the underlying data distribution. Specifically, let the corrupted samples y = ( be generated by the process:
3.1. Diffusion Models
Diffusion models generate data by progressively adding Gaussian noise input data and then reversing this process through sequential denoising steps to sample from noise. Given distribution Po on Rd, the forward perturbation is specified by stochastic differential equation (SDE):
+
(6)
where x and e are independent random variables. Here, x(i is drawn from unknown distribution with density Pdata and ( is sampled from a known error distribution with density h. It can be shown that the corrupted samples y() follow a distribution with density Pdata * h, where * denotes the convolution operator. We provide more details in Appx A.
dx,- = (t) dw,, t E TO.
(1)
Xp ~ Po. T is a fixed positive constant and g(t) is a scalar function. {Weheejorry is the standard Brownian motion.
Eq (1) induces a transition kernel Pt (xe xx.) for 0 S 8 S t S T, which is Gaussian and its mean and covariance matrix can be computed in closed form (Särkkä & Solin, 2019, Eqs 4.23 and 5.51). In particular, for 8 = 0, we write (x (xo) o')
The objective of the (density) deconvolution problem is to estimate the density of Paata using the observed data V. which is sampled from the convoluted distribution Pdata * h. In essence, deconvolution reverses the density convolution process, hence the name of the problem.
(2)
for all / E [0, T], where we set ,({) 2 1// When of is very large, Xy can be approximately regarded as a sample from NN, o Let (xx)) = PPjo (x, (xo) Po(xo) dxg denote the marginal distribution of xs, where we have pr RV N/ I). Anderson (1982) showed that backward SDE
To assess the quality of an estimator p(: of Pdata based on V the mean integrated squared error (MISE) is com- monly used. MISE is defined as:
MISE(P, Pdan) = By S [((x))) - Pdan(()) dx. (7)
dx, = = V (xgg((xx) dt + g() dw,, XT ~ PT
3

---

SFBD: Training Diffussin Models with Finite Noisy Datasets
In this paper, we focus on a corruption process implemented via forward diffusion described in Eq (1). Consequently, unless otherwise stated, in the rest of this work, we assume the error distribution h is Gaussian (000, If with given and fixed ( E (0, T))
It can be shown that if D& minimizes the consistency loss for all r, 8 and perfectly learns the score function for > Ç. then (x,), oi
becomes an exact estimator of the score function for all t E [0, T). Consequently, the distribution Po Plata can be sampled by solving Eq (3).
To see why we could identify an original distribution p through p* h, let (u)( = p (exp(i u x)] for u E IRd be the characteristic function of p. Then,
Daras et al. (2024) demonstrated the effectiveness of this framework only in fine-tuning latent diffusson models, leav- ing its efficacy when training from scratch unreported. More- over, as sampling from Pr|s depends on the model's approx- imation of the score (which is particularly challenging to estimate accurately for t < {) rather than the ground truth, there remains gap between the theoretical framework and its practical implementation. This gap limits the extent to which the algorithm's 's effectiveness is supported by their thcoretical results.
Proposition 1. Letp p and q be two distributions defined on Rd. For all u E Rd.
Sillsexp exp (( (" - [|ll)) 2 PRI. (p= hllg
( (p(()) - (1))
(All proofs are deferred to the appendix.) This result shows if two distributions p and q are similar after being convoluted with h, they must have similar characteristic functions and thus similar distribution. In particular, when p h q* h, then p = g, the case also discussed in Wang et al. (2023, Thm 2). As a result, whenever we could find satisfying Pdata* * h- = h, we can conclude Pdata = q.
4. Theoretical Limit of Deconvolution
In this section, we evaluate the complexity of a deconvolu- tion problem when the data corruption process is modelled using forward diffusion process. Through the framework of deconvolution theory, we demonstrate that while Daras et al. (2024) showed that diffusion models can be trained us- ing noisy samples, obtaining sufficient number of samples to train high-quality models is practically infeasible.
3.3. Deconvolution through the Consistency Constraints
While Prop shows it is possible to train a generative model using noisy samples, it remains difficult question of how to use noisy samples to train diffusion model to generate clean samples effectively.
The following two theorems establish that the optimal con- vergence rate for estimating the data density is O(log n) 2 These results, derived using standard deconvolution the- ory (Meister, 2009) under Gaussian noise assumption, highlight the inherent difficulty of the problem. We present the result for d = 1, which suffices to illustrate the challenge.
The question was partially addressed by Daras et al. (2024)
through the consistency property (Daras et al., 2023a). In particular, since we have access to the noisy samples x from Pusu * h, we can use them to train a network s (x,, t) to approximate (x for > Ç through a modified score matching loss, which referred as ambient score matching (ASM), denoted by (Ø) In their implemen- tation, S (x,, 1) is parameterized by Da(x,,)xx where (x(x) 1) is trained to approximate [[
Theorem 1. Assume y is generated according to (6) with e~ (0, o.) and Pa is univariate distribution. Under some weak assumplions on Paam for sufficiently large sample size n, there exists an estimator p(-; (<) such that
of
of
MISE(f, Pdauu) SC. (log 2
(10)
In contrast, for f S Ç, score-matching is no longer appli- cable. Instead, Daras al. (2024) propose that (x.xx)))
where C is determined by Pdata-
should obey the consistency property:
Theorem 2. In the same setting Thm 1, for arbitrary estimator p(::3) of Pauta based on y
E Xoo x.] Bp-,. [[]xo|x=]], for 0 s5rSasT (8)
MISE( Pdmma) 2 K -logg)
(11)
by jointly minimizing the consistency loss:
where K > 0 is determined by Pdoto and error distribution h
Lcon (Ø, r, Ep. (D(,x,)) | - [D((x, I) (9)
The optimal convergence rate Olog n) -2 indicates that reducing the MISE to one-fourth of its current value re- quires an additional n - samples. In contrast, under the error-free scenario, the optimal convergence rate is known to be O(n-4/5) (Wand, 1998), where reducing the MISE to one-fourth of its current value would only necessitate approximately 4.657n additional samples.
where r and are sampled from predefined distributions. Sampling from Pr|a is implemented by solving Eq (3) back- ward from x,, replacing the score function with the network- estimated one D& via Eq (5). For sampling from P., we first sample x, for T > s and f > Ç, then sample from Ps/r in manner analogous to sampling from Pr|s-
4

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
The pessimistic rate indicates that effectively training gen- erative model using only corrupted samples with Gaussian noise is nearly impossible. Consequently, this implies that training from scratch, using only noisy images, with the consistency loss discussed in Sec 3.3, is infeasible. Notably, as indicated by Eq (10), this difficulty becomes significantly more severe with larger oz. while a large of typically required to alter the original samples significantly address copyright and privacy concerns.
Algorithm Stochastic forward-Backwant Deconvolution. (Given sample set PD denotes the corresponding empiri- cal distribution.)
Input: clean data: Delean = {x(*) J noisy data: Pnoisy
{yr
(0) 1 Ji=1, number of iterations: K.
// Initialize Denoiser
Øu + Pretrain Dø using Eq (4) with po PD
for k = 1 to K do
// Backward Sampling
To address the pessimistic statistical rate, we propose pre- training diffusion models small set of copyright- terreght-free samples. While this limited dataset can only capture subset of the features and variations of the full true data distribu- tion, we argue that it provides valuable prior information, enabling the model to start from a point much closer to the ground distribution compared to random weight initializa- tion. For example, for image generation, pretraining allows the model to learn common features and structures shared among samples, such continuity, smoothness, edges, and general appearance of typical object types.
3
& < [y
Vy!" E Danisy, solve backward SDE
(6)
Eq (3) from T to 0, starting from yr where the score
Doc-, (x,,t)-xe
function is estimated as
of
// Denoiser Update Φx < Train D& by minimizing Eq (4) with po = Pe.
Output: Final denoiser DøR
backward SDE Eq (3). In each iteration, we use the best estimation of the score function so far induced by Dor_, through Eq (5).
Unfortunately, our empirical study in Sec 6 will show that
the consistency loss-based method discussed in Sec 3.3 can-
2. (Denoiser Update) Fine-tur denoiser De to obtain Don, by minimizing Eq (4) with the denoised samples obtained in the previous step.
not deliver promising results even after pretraining. We
suspect that this is caused by the gap between their theo-
retical framework and the practical implementation. As a result, we propose SFBD in Sec 5 to bridge such gap.
The following proposition shows that when Droiey contains sufficiently many samples to characterize the true noisy distribution Pdata * h, when K -> oc, the diffusion model implemented by denoiser Døx has the sample distribution converging to the true Piata-
5. Stochastic Forward-Backward Deconvolution
In this section, we introduce a novel method for solving the deconvolution problem that integrates seamlessly with the existing diffusion model framework. As our approach in- volves iteratively applying the forward diffusion process described in Eq (I), followed by backward step with an optimized drift, we refer to this method as Stochastic Deconvolution (SFBD), described in
Proposition 2. Let Pt be the density of xt obtained by solving the forward diffusion process Eg (1) with Xo Paato- where we have p Paata * h. Consider a modified Alg 1, where the empirical distribution Pr replaced with the ground truth Pt. Correspondingly, P&, becomes Po the distribution of Xo induced by solving:
(k)
dx: - -g(()) Soz ,(xx, 1) dl + g(1) (W),, X ~ D3 (12)
Alg 1.
from Ç to O. where Soz (x,,t) = Du (x g(t) =
The proposed algorithm begins with a small set of clean data, Detean. for pretraining, followed by iterative optimization us- ing a large set of noisy samples. As demonstrated in Sec 6, decent quality images can be achieved on datasets such as CIFAR-10 (Krizhevsky & Hinton, 2009) and CelebA (Liu et al., 2015) using as few as 50 clean images. Dur- ing pretraining, the algorithm produces neural network denoiser, Deb. which serves the initialization for the subsequent iterative optimization process. Specifically, the algorithm alternates between the following two steps: for k = K.
of
( y and Dæn is obtained by minimizing (4) accord- ing to Alg 1. Assume Døs reaches the optimal for all k. Under mild assumptions, for l: 2 O, we have
(pa( |l (k)) 2 DRL (Déau Il Po
(k+1))
(13)
In addition, for all K and E RRd, we have
2Mo
min k=I....K
[8 psu (u) - +b u (w) & exp 2
[|ll))
Po
K
where
1. (Backward Sampling) This step can be intuitively seen as a denoising process for samples in D Dasicy using the
Mo 1/2 1. '(t))Ep; | Vlogpī(xx) - So (x,, t]|l dt.
5

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
Prop 2 shows that, after sufficiently many iterations of back- ward sampling and denoiser updates, the distribution of denoised samples produced by the backward sampling step converges to the true data distribution at a rate of O K)) in terms of the characteristic function. This convergence implies that the corresponding densities and thus the distri- butions also become close. Consequently, fine-tuning the denoiser on these denoised samples during the Denoiser Up- date step enables the diffusion model to generate samples that approximately follow the true data distribution, thereby solving the deconvolution problem.
Table I: Performance comparison of generative models. When oç > O, the models are trained on noisy images corrupted by Gaussian noise NO of after rescaling pixel values to [-1, 1]. For pretrained models, 50 clean images are randomly sampled from the training datasets for pretraining.
Underscored results are produced by this work. Bolded values indicate the best performance.
CIFARIO (32 x 32)
CelebA (64 64)
Method
Pretrain
FID
9c
Pretrain
FID
DDPM al., 2020)
No
4.04
0.0
No
3.26
DDIM (Song al., 2021a]
0.0
No
4.16
0.0
No
6.53
FDM (Karras et at., 2022)
No
1.97
One might argue that the norm in the convergence bound de- pends not only on the iteration count K but also on the norm of u, suggesting the existence of nontrivial approximation gaps when |u| is large, regardless of how large K is. We clarify that in practice, the contribution of the characteristic function at large |u| is typically negligible. For distributions with smooth and bounded densities, characteristic functions decay rapidly, often at exponential or super-polynomia rate. We elaborate further on this point in Appx C, following the proof of the proposition.
SURE-Score (Aali et al., 20233
0.2
Yes
132.61
EMDiff (Bai et al., 2024)
0.2
Yes
86.47
TwaxdiDiff (Daras et al., 2024)
No
167.23
0.2
No
246.95
TweedieDiff (Duras et al., 2024)
0.2
Yes
65.21
0.2
Yes
58.52
TifedieDif'' (Dans al., 2025)
Yes
8,05
0.2
Yes
SPBD (Ours)
0.2
Yes
13,53
0.2
Yos
6.49
15
13:53
TweedieDiff+ SFBD
FID Score
10
8.05
6.75 6.31
5
2.81 2.58
Lastly, we note that this convergence result assumes access to infinitely many noisy samples and should be distinguished from the sample efficiency bounds discussed in Sec 4.
0
50 images
10% clean
4% clean
0=0.2
0-0.2
d=0.59
The importance of pretraining, Prop 2 also highlights the critical role of pretraining, as it allows the algorithm to begin fine-tuning from a point much closer to the true data distribution. Specifically, effective pretraining ensures that $o,, closely approximates the ground-truth score, leading to smaller Mo in Prop 2. This, in turn reduces the number of iterations K required for the diffusion model to generate high-quality samples.
Figure 1: TweedieDiff+ vs SFBD on CIFAR10
approaches require fewer network evaluations and offer im- proved error control for imperfect score estimation and step discretization. As the algorithm generates & that contains samples closer to Pdata with increasing k. clean images used for pretraining can be incorporated into Eg to accelerate this process. In our empirical study, this technique is ap- plied whenever clean samples and noisy samples (prior to corruption) originate from the same distribution.
The practical limits of increasing K. While Prop 2 sug- gests that increasing the number of iterations K can contin- uously improve sample quality, practical limitations come into play. Sampling errors introduced during the backward sampling process, as well as imperfections in the denoiser updates, accumulate over time. These errors eventually off- set the benefits of additional iterations, as demonstrated in Sec 6. This observation further highlights the importance of pretraining to mitigate the impact of such errors and achieve high-quality samples with fewer iterations.
Relationship to the consistency loss. SFBD can be seen as an algorithm that enforces the consistency constraint across all positive time steps and time zero. Specifically, we have
Proposition 3. Assume that the denoising network D& is implemented to satisfy Dæ 0) = =Id(-). When r = 0, the consistency loss in Eq (9) is equivalent to the denoising noise in Eq (4) t s.
Alternative methods for backward sampling. While the backward sampling in Alg is presented as a naive solution to the backward SDE in Eq (3), the algorithm is not limited to this approach. Any backward SDE and solver yielding the same marginal distribution as Eq (3) can be employed. Alternatives include PF-ODE, the predictor-corrector sam- pler (Song et al., 2021b), DEIS (Zhang & Chen, 2023), and the 2nd order Heun method used in EDM (Karras et al., 2022). Compared to the Fuler-Maruyama method, these
The requirement that Do(: Id(.) is both natural and intuitive, as (Xo, 0) approximates [[xx|Xx] = -x-- This fact is explicitly enforced in the design of the EDM frame- work (Karras et al., 2022), which has been widely adopted in subsequent research.
A key distinction between SFBD and the original consis- tency loss implementation is that SFBD does not require sampling from Pria or access to the ground-truth score func-
6

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
Naky
SURE
Are
EMEDIT
Teadlee
Tucello
SEBO
Gread
tance (FID), computed between the reference dataset and 50,000 images generated by the models. Generated samples for FID computation are presented in Appx D.
Models and other configurations. We implemented SFBD algorithms using the architectures proposed in EDM (Karras et al., 2022) as well as the optimizers and hyperparameter configurations therein. All models are implemented in an unconditional setting, and we also enabled the non-leaky augmentation technique (Karras et al., 2022) to alleviate the overfitting problem. For the backward sampling step in SFBD, adopt the -rddrr Heun method (Karras et al., 2022). More information is provided in Appx E.
Noisy Ubserv.
Tweedie Diff w/o PT
Tweedio Diff w/
SPBD (Ours)
Ground Truth
6.1. Performance Comparison
In Table 1, we compare SFBD with representative models for training on noisy images. SURE-Score (Aali et al., 2023) and EMDiff (Bai al., 2024) tackle general inverse prob- lems using Stein's 's unbiased risk estimate and expectation- maximization, respectively. TweedieDiffusion (Daras et al., 2024) applies the original consistency loss from Eq (9), while Daras et al. (2025) introduce TweedieDiff+ with a simplified implementation that improves performance.
Figure 2: Denoised samples of CIFAR-10 (up) and CelebA
(down). (Noise level oe = 0.2)
tion induced by the unknown data distribution Pdata- This is because, in the original implementation, Po Paata whereas in SFBD, pa = () as defined in Prop 2, and is obtained iteratively through the backward sampling step. As k in- creases, Po converges to Pdata ensuring that the same con- sistency constraints are eventually enforced. Consequently, SFBD bridges the gap between theoretical formulation and practical implementation that exists in the original consis- tency loss framework.
Following the experimental setup of Bai et al. (2024), im- ages corrupted by adding independent Gaussian noise with standard deviation of 0ç = 0.2 to each pixel after rescaling pixel values to 1-,, For reference, we also include results for models trained on clean images (ac 0). In cases with pretraining, the models are initially trained on 50 clean images randomly sampled from the training datasets. For all results presented in this work, the same set of 50 sampled images is used.
6. Empirical Study
As shown in Table 1, SFBD consistently produces higher- quality images than all baselines except TweedieDiff+, with further visual evidence provided in Fig 2, where denoised samples are generated by evaluating the backward SDE from noisy training images. Notably, on CelebA, SFBD achieves performance comparable to DDIM trained entirely clean data. While TweedieDiff benefits from pretrain- ing, its performance remains inferior to SFBD. In fact, we observe that the original consistency loss (9) yields only limited improvement post-pretraining with FID scores dete- riorating soon after the loss is applied. TweedieDiff+, which adopts a simplified version of the consistency loss, outper- forms SFBD when only very limited amount of clean data is available, likely due to overfitting during our model's s pretraining. However, this effect diminishes as more clean data is introduced, allowing SFBD to surpass TweedieDiff+, as shown in Fig 1. We attribute this to SFBD's 's more sta- ble fine-tuning via the score-matching loss, which avoids the additional constraints imposed by consistency- based methods.
In this section, we demonstrate the effectiveness of the SFBD framework proposed in Sec 5. Compared to other models trained on noisy datasets, SFBD consistently achieves superior performance across all benchmark set- tings. Additionally, we conduct ablation studies to validate our theoretical findings and offer practical insights for ap- plying SFBD effectively.
Datasets and evaluation metries. The experiments are conducted on the CIFAR-10 (Krizhevsky & Hinton, 2009) and CelebA (Liu al., 2022) datasets, with resolutions of 32 x 32 and 64 x 64, respectively. CIFAR-10 10 consists of 50,000 training images and 10,000 test images across 10 classes. CelebA, a dataset of human face images, includes a predefined split of 162,770 training images, 19,867 valida- tion images, and 19,962 test images. For CelebA, images were obtained using the preprocessing tool provided in the DDIM official repository (Song et al., 2021a).
We evaluate image quality using the Frechet Inception Dis-
7

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
14
Cleart Image Ratio
+ 0.64
14
Mese Lovel
250
Image
automoble
ship
10
ED14
0.20
10
2:25
FID Scare
150
200
weto protrain
2OM RUM
FID
FID
50
Itar
iter
Iterations (k)
Iterations (k)
Itcrations (k)
(a) Clean Image Ratio
(b) Noise Level
(c) Pretraining on Similar Datasets
Figure 3: SFBD performance on CIFAR-10 under various conditions. Unless specified, the clean image ratio is 0.04 and the noise level og is 0.59. In (a) and (b), FID at iteration O corresponds to the pretrained model. In (c), models are pretrained on clean images from the "truck" class, with FID at iteration O measuring the distance between these clean images and those used for fine-tuning, For the w/o pretraining setting, models are trained on the full CIFAR-10 dataset with og = 0.59.
dc.
0.00
0.20
0.30
0.58
1.00
L.9
0.00
0:20
1.38
Table 2: Comparison of SFBD and Restormer (7amir et al., 2022) on denoising tasks.
CIFAR- (455 clean images)
ColehA
CIFAR-10
ColebA
Satting
0.30
SFBD
6.16
3.42
2.68
2:35
S0ciese inge
SEBD
47.69
10.55
5.63
3.85
Figure 4: Noisy images with different de.
Restener
53.87
0.2
Rotenner
18.50
0.59
SFBD
10:35
7.47
6.31
6.54 1.5 trns
SPID
9,05
5.76
4.56
3.98
Restemer
99
Restmmer
73741
Restemvrr
132.99
6.2. A Ablation Study
In this section, we investigate how SFBD's performance varies with clean image ratios, noise levels, and pretraining on similar datasets. The results align with our discussion in Sec 4 and Sec 5 and provide practical insights. Ex- periments are conducted on CIFAR-10, with the default 9, = 0.59. This noise level significantly alters the original images, aligning with our original motivation to address potential copyright concerns (see Fig 4).
the values from 2nd order Heun sampling in EDM (Karras et al., 2022). The impact of noise on the original images is visualized in Fig 4. As shown in Fig 3(b), increasing oç sig- nificantly degrades SFBD's performance. This is expected, as higher noise levels obscure more features in the original images. Furthermore, as suggested by Thm higher og demands substantially more noisy images, which cannot be compensated by pretraining portantly, this performance drop is a mathematical limitation discussed in Sec 4, rather than an issue solvable by better deconvolution algorithms. In Sec 6.3, we show that slightly increasing pretraining clean image set can yield strong re- sults, even at reasonably high noise levels on CelebA.
small clean image set. Im-
Clean image ratio. Fig 3(a) shows the FID trajectories across fine-tuning iterations for different clean image ra- tios. With just 4% clean images, SFBI) achieves strong performance (FID: 6.31) and outperforms DDIM with 10% clean images. While higher clean image ratios further im- prove performance, the gains diminish as small amount of clean data already provides sufficient high-frequency features (e.g., edges and local details) to capture feature variations. Since these features are shared across images, additional clean data offers limited improvement.
Pretraining with clean images from similar datasets. Fig 3(c) evaluates SFBD's performance when fine-tuning image sets from different classes, with the model ini- tially pretrained that the closer the noisy dataset is to the truck dataset (as indicated by the FID at iter 0), the better the model performs after fine-tuning. This is expected, as similar datasets share common features that facilitate learning the target data dis- tribution. Interestingly, even when the pretraining dataset differs significantly from the noisy dataset, the model still outperforms the version without pretraining. This is because unrelated datasets often share fundamental features, such as edges and local structures. Therefore, practitioners should always consider pretraining before fine-tuning on target noisy datasets, while more similar pretraining datasets yield better final sampling performance.
clean truck images. The results show
These findings suggest that practitioners with limited clean datasets should focus on collecting more copyright- free data to enhance performance. Notably, when clean images are scarce, the marginal gains from additional fine-tuning itera- tions A are greater than when more clean data is available. Therefore, in scenarios where acquiring clean data is chal- lenging, increasing fine-tuning iterations can be an effective alternative to improve results.
Noise level. Fig 3(b) shows SFBD's ' sampling performance across fine-tuning iterations for different noise levels, using
8

---

SFBD: Training Diffussin Models with Finite Noisy Datasets
Noiay Olarrv
Tuur
Torr
Iter
Ree
Goound Truth
Output
Nekey Observ.
Ouspas
Experiment Configurations
40
cly de 4y 1.ss; Juan ing. elg di oen 3.30: 50 clean mgg <ig co oy uut s0 dean imgg
FD Scorn
10
bor0
tor Tor
merations (4)
Ioor
1.38
A
cg
0:
0.2
Figure 5: (Leftt) SFBD performance on CelebA under three configurations, with FID iteration 0 for the pretrained model. (Right) Denoised samples generated by the backward SDE, starting from noisy image in the training dataset. For cfg A, results are shown after each fine-tuning iteration, while cfg B and cfg C are shown at their minimum FID iterations.
6.3. Further Discussions
denoising and fine-tuning it does not function like tradi- tional denoiser that reconstructs exact clean samples. In- stead, it learns to match the full data distribution, allowing greater flexibility and producing more realistic outputs. In Table 2, we compare the FID of SFBD-denoised samples at each iteration (Eg in Alg 1) against those denoised by Restormer (Zamir et al., 2022), strong off-the-shelf de- noiser. SFBD consistently yields significantly lower FIDs even after the first iteration, and the sample quality improves steadily with more updates.
Additional results on CelebA. Fig 5 presents SFBD per- formance trajectories on CelebA under three configurations. While Table reports results using configuration (cfg) C to align with benchmarks, this setup is impractical due to its low noise level, which fails to address copyright and privacy concerns. As illustrated in Fig 5 (right), the low noise level allows human observers to identify individuals and recover image details, with model-denoised images nearly identical to the originals. To address this, we report results for cfgs A and B with oc= = 1.38, concealing most original image information. While pretraining on 50 clean images performs poorly, increasing the size to 1.5k (still < 1% of the train- ing dataset) achieves impressive results. At iteration 3, the model reaches FID 5.91, outperforming DDIM trained on clean images. This supports our discussion in Sec 6.2: col- lecting more clean data significantly boosts performance when the clean dataset is small.
These results also caution against replacing SFBD's denois- ing process with a classical denoiser. Since models trained on denoised samples cannot surpass their targets in FID, the values in Table 2 represent upper bounds. Notably, final SFBD models achieve lower generative FIDs (Fig 3(b)) than those from Restormer-denoised data, making it unlikely that competitive model could be trained on such samples.
Data leakage and sample memorization. We note that SFBD is not intended to prevent leakage of the clean sam- ples used for pretraining. As these samples are assumed to be public and copyright-free, leakage from this subset is not concern. Instead, SFBD is specifically designed to protect sensitive data. By construction, the model accesses only single corrupted version of each sensitive sample during the entire training process. This design inherently limits the model's ability reconstruct copyrighted or private content, making SFBD privacy--rrseenvy we adopt the methodology of Daras et al. (2024) to evalu- ate privacy risks: we plot similarity score distributions and identify the sensitive sample most similar to any generated output. The results confirm that SFBD does not reconstruct sensitive data, supporting our privacy claims.
Features learned from noisy images. As shown in Fig 5, when 0g- = 1.38, almost all information from the original images is obscured, prompting the question: can the model learn from such noisy inputs, and how does this happen? In Fig 5, we plot the model's denoised outputs in cfg A after each fine-tuning iteration. These outputs serve as samples for the next iteration, revealing what the model learns and adapts to in the process. For the first row, the pretrained model (iter 0) produces a face very different from the origi- nal, failing to recover features like a headband. This occurs because the clean dataset for pretraining lacks similar faces with headbands. Instead of random guesses, the model com- bines local features (e.g., face shapes, eyes) learned from the clean data with the global structure from the noisy im- ages. This process combines previously learned features in new ways, helps the model better generalize, and gradually improves its ability to approximate the true distribution, as supported by Prop 2. Similarly, in the second row, the model learns to attach goatee to the face despite the correspond- ing region in the original image being a microphone.
by nature. In Appx F,
7. Conclusion
In this paper, we presented SFBD, new deconvolution method based on diffusion models. Under mild assump- tions, we theoretically showed that our method could guide diffusion models to learn the true data distribution through training on noisy samples. The empirical study corroborates our claims and shows that our model consistently achieves state-of-the-art performance in some benchmark tasks.
How SFBD differs from standard denoising algorithms. While SFBD is intuitively described as alternating between

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
Acknowledgement
Ben-David, S., Bie, A., Canonne, C. L., Kamath, G., and Singhal, V. Private distribution learning with public data: The view from sample compression. In Thirty- Conference on Neural Information Processing Systems, 2023. URL https::///// pttttss://ppereee//// oppnrevieweeeeeeeeeeeeeeee net/ forum? id-nDIrJmKPd5.
We gratefully acknowledge funding support from NSERC, the Canada CIFAR AT Chairs program and the Ontario Early Researcher program. Resources used in preparing this re- search were provided, part, by the Province Ontario, the Govermment of Canada through CIFAR, and companies sponsoring the Vector Institute.
Betker, J., Goh, G., Jing, L.,, Brooks, T., Wang. J., Li, L., Ouyang, L.,, Zhuang, J., Lee, J., Guo, Y. et al. Im- proving image generation with better captions, OpenAl, 2023. URL https: http:://: //cdn openai comppapers/ da dall-e-3 .pff.
Impact Statement
This paper introduces SFBD, framework for effectively training diffusion models primarily using noisy samples. Our approach enables data sharing for generative model training while safeguarding sensitive information.
Bie, A., Kamath, G., and Singhal, V. Private estimation with public data. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
For organizations utilizing personal or copyrighted data to train their models, SFBD offers a practical solution to mitigate copyright concerns, as the model never directly accesses the original samples. This mathematically guaran- teed framework can promote data-sharing by providing a secure and privacy-prese preeeyypress training method.
Bora, A., Price, E., and Dimakis, A. G. AmbientGAN: Generative models from lossy measurements. In- lernational Conference on Learning Representations, 2018. URL
However, improper implementation poses a risk of sensi- tive information leakage. A false sense of security could further exacerbate this issue, underscoring the importance of rigorous validation and responsible deployment.
net forum?
id= fDogOb.
Bovy, J., Hogg, D. W., and Roweis, S. T. Extreme deconvolution: Inferring complete distribution func- tions from noisy, heterogeneous and incomplete obser- vations. The Annals of Applied Statistics, 562B) 1677, 2011. URL https: /tt///diii org/oo/gg//ooooooooo 1214/ 1 10-A0AS439.
References
Aali, A., Arvinte, M., Kumar, S., and Tamir, J. I. Soly- ing inverse problems with score-based generative pri- ors learned from noisy data. In 57th Asilomar Con- ference on Signals, Systems, and Computers, pp. 837- 843, 2023. URL pertps://doos orooorgg//// 110./11919/// IEEECONF DEFECONF5922.2023. 10477044.
Carlini, N., Hayes, J.. Nast, M., Jagielski, M., Sehwag, V., Tramer, F., Balle, B., Ippolito, D., and Wallace, E. Extracting training data from diffusion models. In 32nd USENIX Security Symposium, pp. 5253-5270, www.ix.eni......... www.:u.ww.........eeeoe...eeee.e
2023
2023. BtLLttps::/ files htte://ssenkkeeuurtt//aaaaaaa pdf.
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learn- ing with differential privacy. In Proceedings of the ACM SIGSAC Conference on Computer and Commu- nications Security, pp. 308-318, 2016. URL https: //doi, wwg/10.///.ooooooooooooooo..// /..g//..11550 00..0..15/9977777 2978331..
Carroll, R. J. and Hall, P. Optimal rates of con-
vergence for deconvolving a density. Journal of the American Statistical Association, 83(404):1184- 1186, 1988. URL https: /tte:://///.iie//e http:/oi.oo....o///// ott./0.10/1/ 01621459. 10478718.
Anderson, B. D. O. Reverse-time diffusion equation mod- cls. Stochastic Processes and their Applications, 12 (3):313-326, 1982. URL https: /tl:://dooi/ or0/10. 1016/0304 4149 (82) 90051-5.
Cordy, C. B. and Thomas, D. R. Deconvolution of a distribution function. Journal of the American Sta- tistical Association, 92(44)):155-11665 1997. URL https: / /tw.i org/ 10.
Bai, W., Wang. Y., Chen, W., and Sun, H. An expectation- maximization algorithm for training clean diffusion mod- els from corrupted observations. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL httpp::://ppeeeview..rr Forum? d= 3=jURRh4V9NE
Croitoru, F.-A., Hondru, V., Ionescu, R. T., and Shah, M. Diffusion models in vision: A survey. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 45 2023. URL https: //pp:///d.... .rg/ 10.1109/TPAN 2023. 326199 9888.
10

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
Daras, G. and Dimakis, A. Solving inverse problems with ambient diffusion. In NewrlPS 2023 Workshop on Deep Learning and Inverse Problems, 2023. URL https: wwweevviiwnnee/ooooiieee //ppeeeeview.neeeeeeeeeeeeeeeeee
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B.,
Warde-Farley, D., Ozair, S., Courville, A., and Ben- gio, Y. Generative adversarial networks. Commu-
d==mGwg1 ObgHk.
nications of the ACM, 63(11):13-144, 2020. URL https: 1tt../..rrr111111/42222200000
Daras, G., Dagan, Y., Dimakis, A., and Daskalakis, C. Consistent diffusion models: Mitigating sampling drift by learning to be consistent. In Advances in Neu- ral Information Processing Systems, pp. 42038-42063, 2023a. URL https: httttttss://pperreeee http///wwwwwew.//et/tp//e net/ forum? id-Gf2GdJHj27.
Guan, Z. Fast nonparametrie maximum likelihood den- sity deconvolution using bernstein polynomials. Sta- tistica Sinica, 31(2):891-908, 2021. URL https:
/ //dpi.org////o/... /0. 5705/ / /ss. 202018.0173.
Ho, J., Jain, A., and Abbeel, P. Denoising diffu- sion probabilistic models. Advances in Neural Information Processing Systems, pp. 6840-6851, 2020. URL https::/prroeeeinng.neese cc/ cc/paper_fiie 4 ( pdf.
Daras, G., Shah, K., Dagan, Y., Gollakota, A., Dimakis, A., and Klivans, A. Ambient diffusion: Learning clean distributions from corrupted data. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL wwwph:ttts//ocm http::///pppeeeeeeeeeeee net/ forum? id 1d=wBJBLy9KBX
Daras, G., Dimakis, A., and Daskalakis, C. C. Consistent diffusion meets tweedie: Training exact ambient diffu- sion models with noisy data. In Forty-first International Conference on Machine Learning, 2024. URL https: / //openreview. net. forum? wiiiidddllyyaFatt
Ho, J., Salimans, T., Gritsenko, A. A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models. In Advances in Neural Information Processing Systems, 2022. URL https: /wtp//pppweeeeeeeeeeeeeee wttp:e/iiiwweeeffruuueeee forum? Id=f3zNggg__e..
Daras, G., Cherapanamjeri, Y., and Daskalakis, C. C. How much is noisy image worth? data scaling laws for am- bient diffusion. In The Thirteenth International Confer- ence on Learning Reprexentations, 2025. URL https: //openreview net forum? wwt.miiiddiiwweeeeeo.eee wtt.:ew.eee/ootneeeeeeeeee
Kailath, T. The Structure of Radon-Nikodym Deriva- tives with Respect to Wiener and Related Measures. The Annals of Mathematical Statisties, 42(3):1054 1067, 1971. URL https:tt///// http://oo..oorrooo 1.110.121111 2////1//cmm/ acms/ 1177693332
Delaigle, A. and Hall, P. Using SIMEX for smoothing- parameter choice in errors-in-variables problems. Jour- nal of the American Statistical Association, 103(481): 2008. URL tups: /tw.i/oiii.//// http:/....o//1..1...... 1198/ 01 621 000000000011
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-basse generative models. In Advances in Neural Information Processing Systems, 2022. URL http:::://ppeeeveeeww net forum? id=k 7FUTOWMOC7.
Dockhorn, T., Cao. T., Vahdat, A., and Kreis, K. Dif- ferentially private diffusion models. Transactions on Machine Learning Research, 2023. URL https:// openreview http:/oorummm/ppp00--0
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference for Learning Representations, 2015. URL http::// httpp:://aaxvv.oo abs/ 1112/ 698C.
Fan, J. On the optimal rates convergence for nonpara- metric deconvolution problems. The Annals of Statistics,
Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. URL https: httpas:///ppeeeeeee /openrevieeeeeeeeeeeeeeeee net/ forum? id=a XFK8YmZ5J.
193):1257-1272, 1991. URL https:///// www. jstor. org/sstable/224199
Feller, W. An Introduction Probability Theory and Its Applications, Volume 2. Wiley, 1971.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Techni- cal report, University of Toronto, 2009. URL https://www.co cs torcnto.oooooooooo edu.triizzu// Learniig-features--000 pd.
Goodfellow, I. Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Ad- vances in Neurul Information Processing Systems, 2014. URL htttp:://proceeding..eee.. ccc/appe__Lleee/ppperr///////// caaeebl122fa1eer0sssseetttteeeed Paper pdf.
Lepski, O. and Willer, T. Oracle inequalities and adaptive
estimation in the convolution structure density model.
The Annals of Statistics, 47(1):233-28) 287, 2019. URL https: /tap::///oi.../ http//iiiooo.../////// 0.110..1114 121.111111111 111///11--1111111111111111111111 187.

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning generate and transfer data with rectified flow. In The Eleventh International Conference Learning Representations, 2022. https: //pppepeeeee.e.eeeee/ee net/ net/forumt forum? d=XViTTInnss
Pavon, M. and Wakolbinger, A. On free energy, stochastic control, and Schrödinger processes. In Modeling, Estima- tion and Control of Systems with Uncertainty: Proceed- ings of a Conference held in Sopron, Hungary, September 1990, pp- 334 348, 1991. URL https: //oi//do.//// .oooo.oor..... 10. 1001000//91
4612-6122662 0443-443400 5_22.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learn- ing face attributes the wild. Proceedings of In- ternational Conference on Computer Vision (ICCV), 2015. URL http: http://mmmllbb.. ieb.ie... cuhk, edu .k// projects/Celeba. httm. html.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dock- horn, T., Müller, J., Penna, J., and Rombach, R. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / /openreview. net/ forum?id=dis 52222md.....
Lounici, K. and Nickl, R. Global uniform risk bounds for wavelet deconvolution estimators. The Annals of Statistics, 3911)2201-231,001 2011. URL https://doo. 2tg/10..214/10--
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695 2022. URL https:ttt////..... //opi: org 2rg/10..
Masry, E. Strong consistency and rates for deconvolu- tion of multivariate densities of stationary processes. Stochastic Processes and Their Applications, 47(1): 53-74, 1993. URL https:/ /top:://doo..oo http://o.orrg//110000001 0304 1204-4100 4149 (93) 90094-K.
/doi.
2022. 01042.
Sarkar, A., Pati, D., Chakraborty, A., Mallick, B. K., and
Carroll, R. Bayesian semiparametric multivariate den- sity deconvolution. Journal of the American Statistical Association, 113(521):401-411 2018. URL https: doi doi.org/ 1080 00000000011110000 2016 20....2..1.......1.1 1260467.
Meister, A. Deconvolution Problems in Nonparametric Statistics. Springer, 2009. URL https:ttp/////... /ttp://ooooo/////o//// 10. 1001/007/78 3-540- 87557-4.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, Deep unsupervised learning using nonequi- librium thermodynamies. In Proceedings of the 32nd International Conference on Machine Learning, pp. 2256- 2265, 2015. URL https: https:://rroceddii press / v7 / sttp:sshllllaoes www.llldicssttiinnn..1.mmmm html.
Meister, A. and Neumann, M. H. Deconvolution from non-standard error densities under replicated measurements. Statistica Sinica, 20:1609-1636, 2010. URL https://wwwwt wtwsssteestttaa///2/e2 /ttp//222///2221222222222222
edu.
.mlr.
Nie, W., Guo, B., Huang. Y., Xiao, C., Vahdat, A., and Anandkumar, A. Diffusion models for adversarial pu- rification. In Proceedings of the 39th International Conference on Machine Learning, pp. 16805-16827, 2022. URL httph:ttp/// httpas://pccceee/eeeeeeee www.m.r.......... press/ v162/nie224 html.
Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T. Diffusson art digital forgery? investi- gating data replication in diffusion models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6048-6058 2023. URL https: / /doi/ooooo//////oo wttp./org/...11///////1// 1ttp//...119///P2277777 2023.00586.
Oksendal, B. Stochastic Differential Equations: An
Introduction with Applications. Springer, 6th edi- tion, 2003. URL http:://bi..rrg/10.00007 978- 118-3-642-14394-6
Song, J., Meng. C., and Ermon, S. Denoising diffu- sion implicit models. In International Conference on Learning Representations, 2021a. URL https:// openreview net wwr.m//rummmmmmm www.rronnniissttiiiiiiiaooim
Oquab, M.. Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W. Howes, R., Huang, P.-... Li, S.WW., Misra, I. Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Je- gou, H., Mairal, J., Labatut, P., Joulin, A., and Bo- janowski, P. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8856. URL https: openreview net. forum? d 58SUt 62Ft. Fea- tured Certification.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative mod- eling through stochastic differential equations. In In- ternational Conference on Learning Representations, 2021b. URL.https::/oppnreeee net. forum? id-PxTIG12RRHS.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. In Proceedings of the 40th International Conference on Machine Learning, pp. 32211-3333 132252,
12

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
2023. URL https: http://pooeendisss/e.e. ngs www.........r...mmm..........wee mlr v202/ 2002/song2a html.
prreessss press/
Zhang, Q. and Chen, Y. Fast sampling of diffusion mod- els with exponential integrator. In The Eleventh In- ternational Conference Learning Representations, 2023. URL
Stefanski, L. A. and Carroll, R. J. Deconvoly- ing kernel density estimators. Statistics, 21(2):169- 184, 1990. URL https://abi.org: 0233188900880233.
net forum?
rd=Loek7hfb446
1080/
Särkkä, S. and Solin, A. Applied Stochastic Differential Equations. Cambridge University Press, 2019. URL attps::/dbl..org11...tt /ttp///..rgg11.11117771111111111
Tsybakov, A. Introduction to Nonparametric Estima- tion. Springer, 2009. URL httpp::///o...ooo
1007/b13794.
Vargas, F., Thodoroff, P., Lamacraft, A., and Lawrence, N. Solving schrodinger bridges via maximum likelihood. En- tropy, 23(9), 2021. URL https: /wwww//wwww///////// www...ppp.. com/
1099-4300/23/9//
Wand, M. Finite sample performance of decon- volving density estimators. Statistics & Prob-
URL ability Letters, 37(2):131-139, 1998.
https: http:://wwwwwwee www.w//ienneeireetccomm
Wang, Z., Zheng, H., He, P., Chen, W., and Zhou, M. Diffusion-GAN: Training GANs with diffusion. In The Eleventh International Conference on Learning Represen- tations, 2023. URL https: /tttps://peenrevieeee www.oeviee.wn....m. net./ forum? HZ HZE70bpHHUA.
Xie, L., Lin, K., Wang, S., Wang, F., and Ren, J. Differ- entially private generative adversarial network. arXiv preprint 1rX.v:1802.06639 2018. URL https:// arxiv. htgg/abs/1802...
Yang, D., Yu, J., Wang. H., Wang, W., Weng, C., Zou, Y., and Yu, D. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:1720-1733, 2023. URL https: / /ttp///...//..11///////11//// 2023. 3268730.
Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., and Yang, M.-H. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5728-5739, June 2022.
Zhang, C.-H. Fourier methods for estimating mixing den- sities and distributions. The Annals of Statistics, (2):806 831, 1990. URL https://dda wtt.do.oooo/.....ooo 1214/8os/1176347622
13

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
A. A Brief Introduction to the Density Convolutions
In this section, we give brief discussion on the density convolution and how it is related to our problem.
For simplicity, stick to the case when d = 1. Consider the data generation process in Eq (6). Let Py denote the density of the distribution of the noisy samples ()), Then we have
Fact For R
Py(w) = Paotaa (1) hew I.) dx
(Páata * (:)
(14)
Proof. This is because, for all measurable function vh, we have
I (w) dw = / ob + c) Pama (a)a(())(()))))))))))))))))((( dx / (amaa)aa( x) de
= -[w(w) ((w)) [f paaa((t)) nlw - x)) dx dw.
As the equality holds for all bb, have py (w) (1)) M(w x)) dr (Pdata * (.)
As result, according to Fact I, the density convolution is naturally involved in our setting.
Then, we provide an alternative way show why we can recover Pdata given Py and h. (Namely, need to deconvolute Py to obtain Pdata.) Our discussion can be seen complement of the discussion following Prop 1. Let Φp denote the characteristic function of the random variable with distribution p such that
(t) - cxp(itw) p(w) du.
(15)
note that the characteristic function of a density p is Fourier transform. As result, through the dual relationship of multiplication and convolution under Fourier transformation (Meister, 2009, Lemma A.5), we have
Op, (t) Open (t) (t).
(16)
As result, given noisy data distribution Py and noise distribution h, we have
pum (1) Op (t)
(17)
Finally, we recover Piata through an inverse Fourier transform:
Pditta (x) = (2x)-1 f exp((iiax) &ram. (1) di = (2x)-1 / expl expl-itx)
Pr. (t) dc
(18)
&n. (t)
We conclude this section by summarizing the relationship between data and noisy sample distributions Fig 6.
convolution
Pdata
Py = Pdata * h
deconvolution
x()) add (4) ~ y(0 20(0) + (())
irreversible
Figure 6: While the corruption process is irreversible at the sample level, bijective relationship exists between the clean and noisy data distributions.
14

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
B. Proofs Related to Deconvolution Theory
We first show the result suggesting it is possible to identify a distribution through its noisy version obtained by corrupting its samples by injecting independent Gaussian noises.
Proposition 1. Let and q be two distributions defined on R.. For all u E R
(0p(u) (u((u)sSexp |ull)) DRi (pt
Lemma 1. Given two distributions and q on Rd. Let Pp(u) and (u) be their characteristic funtattonsticcunniiins. Then for all Rdd we have
(p(()) +g(u)| s V2 (p g).
(19)
Proof. We note that
(p(u) = Eylexp(iu x)], Py(u) = Exlexp(tuu) x)].
Then for any u Rd, we have
/ exp(in" x))((x) dx / exp(in' xx)(()) dx
$ (u) (u) s
J/c ad
exp(iu x)(xx) o(x)) dx s $ lexp(iu x)) lp(xx q(x)) dx
{ [p(x) - q(x)) dx
2 |pp g||TV,
where the last equality is due to Scheffe's theorem (Tsybakov, 2009, Lemma 2.1, p. 84)). Then, by Pinsker's inequality (Tsybakov, 2009, Lemma 2.5. p. 88), we have
[p)()) - - < 2 ||p rl s (P
which completes the proof.
Proof of Prop 1. Note that, by the convolution theorem (Meister, 2009, A.4), for all u E I2,, have
of
pa (u) (u) (u) (u exp ||ll3),
as h ~ o21) having 2 (u) exp (- (1ull)) Applying Lem 1, we have
exp
- 2
|ull)) (u) 2 (u)
1 prr (ppr(()) (u) (y) / Dku
(20)
Rearranging the inequality completes the proof.
We then derive the proofs regarding the sample complexity of the deconvolution problem.
Theorem Assume y generated according (6) with e ~ (0, o²) and Pdot is a univariate distribution. Under some weak assumptions on Páat for sufficiently large sample size n, there exists an estimator p(:;)) such that
MISE(j, Pdino) SC
(10)
(log )
where C determined by Pduta-
15

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
Proof. The result constructed based on the work by Stefanski & Carroll (1990). In particular, assuming that Pasta continuous, bounded and has two bounded integrable derivatives such that
f Paama (x) dx < 00,
(21)
we can construct kernel based estimator of Pdata of rate
X4
/ Piata (x) dx,
4 1kk22
(22)
where 2²,2 is constant determined by the selected kernel x. and X is function of number of samples gradually decreasing -> It is required that satisfies
B²o
exp(
(23)
2Tn
X²
as n oc, where B >> 0 is constant depending on the picked kernel Here, we assume we picked a kernel with B 1.
To satisfy the constraint, we choose X(n)
Plugging it into Eq (23), we have
vlog
B² lim n=000
Vlogn n nog
Vlog
lim 100 nd
exp(
exp (B² (ggn) = lim
(24)
22
n=000 n 1--B'o 0c
To show limn-+oc
ing B2 og
suffices to show lim
Ing n n² 28 2
= 0. By L'Hopital's rule, we have
log lim 2-28 o (2 2B²) n 28
lim
(25)
As result, X(n)
oc is a valid choice, which gives the convergence rate [Tog n Vlgg n
Theorem 2. In the same setling as Thm for an arbitrary estimator p(=;3) of Pdana based on y,
MISE(P, Pama) 2 K (log n) -2
(11)
where > O is determined by Paoto and error distribution
Proof. This result is special case of Theorem 2.14 (b) in (Meister, 2009). When the eπror density is Gaussian, we have y= 2. In addition, the proof Thm 1, we assumed that Pasta has two bounded integrable derivatives, which equivalently assumes Piata satisfies the Soblev condition with smoothness degree (see Eq, A.8, Meister 2009), Then the theorem const (log
shows MISE(p, Pama)=
const (log n) 28/7
C. Proofs Related to the Results of SFBD
We first prove Prop 2, which we restate below:
Proposition 2. Let Př be the density of x4 obtained by solving the forward diffusion process Eq (I) with Xy ~ Pdita- where we have p-= = Pdata hh. Consider a modified Alg 1, where the empirical distribution Pewc, is replaced with the ground truth
Vc. Correspondingly, PE, becomes Po (k), the distribution of Xa induced by solving:
dx; = gg()) 200) SoL (x,, t) dt + g(t) dw,, X; pt
(12)
y and Do, is obtained by minimizing (4) according Alg I. Assume D&z reaches the optimal for all k. Under mild assumptions, for 2 O. we have
from Ç to O, where SΦ. (xt, De, (xx,))-xx g(t) do? of
DRL. (Dáma || (00)
(páora || Po (k+1)).
(13)
16

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
In addition, for all K 2 1 and Rdd, have
2 2Mo - exp
min
d pams (u) - # (0)
2 ||11)))
K
where
Mo = //-0000 s(t)) (ogpi( Sœv (X1, t)|l dt.
To facilitate our discussions, let
2% :: the path measure induced by the backward process Eq (12). In general, use To denote the path measure when the drift term is parameterized Φ
(k), the path measure induced by the forward process Eq (1) with Po (R) u:ç
" definnd in Prop 2. The density of its
marginal distribution at time / is denoted by Pi (k)
P .. the path measure induced by the forward process Eq (1) with po = Pdata-
note that, according to Alg I, the marginal distribution of 2 de-
at has density Po
(k)
0:5
The following lemma allows us to show that the training of the diffusion model can be seen as process of minimizing the KL divergence of two path measures.
Lemma (Pavon & Wakolbinger 1991, Vargas al. 2021). Given two SDEs:
dx, = fx(x,, f) dt + g(t) dw,, No Po (0) (x) t E [0
(x)
(26)
for = 1,2. Let p for 1,2, be the path measure induced by them, respectively. Then we have,
DEL (P O.T
0:T = (KL(P) p)))))
+ E
(x,,,
fz(xe, t) dt
(27)
P(1
gg())²
In addition, the same result applies to a pair of backward SDEs as well, where Po is replaced with pT
Proof. By the disintegration theorem (e.g., see Vargas et al. 2021, Appx B), we have
dp(1) (- (xx))
(RI || P2) = (KL(Po) (1) || po ()) + E
Ein log
(28)
dp(() 0.7 (x))
where p(()) (- xx) is the conditioned path measure of p(() given the initial point xo. Then, applying the Girsanov theorem (Kailath, 1971; Oksendal, 2003) on the second term yields the desired result.
By Lem 2, we can show that the Denoiser Update step in Alg finds Φe minimizing DKL ((O) that
To see this, note
0:ς
Φ argmin DKL ((0) || T
g(1)2 2 | log (K) (xx) s (x t)l2 dt
argmin (p (K)
pp)
B drc
(29)
where Pt is the marginal distribution induced by the forward process (1) with the boundary condition Po at t = 0. Note
that, we have applied Lem to the backward processes inducing (()) and Q0:-- Thus, the drift term of P (k) is not zero
0:5
0.5
but (t) log Pt (xx) according to Eq (3). Since the first term of Eq (29) a constant, the minimization results in
V log (k) (xx) Sot (xe,
(30)
17

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
for X2 E Rd and (0, C. addition, we note that, the denoising loss Eq (4) is minimized when log (k) (x, t) for all t > 0; as result, ΦR minimizes DKL. (k) O:C |I 20:2 claimed.
(xt)
Now, we ready prove Prop 2.
Proof of Prop 2. Applying Lem to the backward process
+BT o.c % git)
DKL Tox-1) = DKL
|1 P
V Vlogp( - spc_, (xe, t)ll dt
1-- o(t)²
=Bp,
V log (xx) SØL (xe,, t) dt
(31)
dic
Likewise,
DKI
0.c)
DKL(Pe ll 1D5 (K)
+ Epp ais
o(t)
V logpi (xt) v log (8) (x+)1l dt
= PRL |1 11pp()) + Eopp.
V
logg?
SoL (x1,
2
3D (PL |l Pc (k))
+ DKL ( Poxc
||
Q 000
(32)
d:c
where the second equality is due to the discussion on deriving Eq (30).
Lem 2 also implics that
%o 1 (lbb(k-1) (xe, (x d
DKL ") DKL (Pdata || 1 + E E
(33)
where (a- (x1-1)(x(,, t) is the drift the forward process inducing
Q
ti-1 Dzc
In addition,
1/00 1110 o/I2 at = (Peata ll Po (x)))
DKL (Pāc 0.c || B(0) = (Pean |l ) + E pp
(34)
0:c
As result,
DKI(Pama || (1)) (34)
DKL Po. | F(())
(32)
DEL(PE 11 (1)) + DKL. Po || T
> DKL Po:c 11
oz
(33)
DRL DKI(Paaa || Po (k+1)) + Be
1o0
DKL. (psma Po
(k+1))
which is (13). addition, we have
DKL(P 0x ll 2ok-1)
(33) DRL |1 (2) + BE-1
DKL
Po: (Pi
P((d) + B
(32) DKL |l (k) + DKL( T + Bx-1
(Pi @00) + [Dk || p(()) + Bx-1]
As result, applying this relationship recursively, we have
DRa. 1100 Σ DRa (0) + Σ Ba- DKa. (P T0x)
(35)
18

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
Since DKI. (Pi dcç (00
Mo. we have
DeL (Pana p())) h)) (R ll (x s Mo.
(36)
for all K 2 1. This further implies,
Mo K
min kk.,,......)
DEL (Pamu * h p()) 11) S
(37)
Applying Prop 1, we obtain,
- pa (u) S exp - [|ul))
2Mo
min kE(.......K
Ppmm (u)
(38)
K
Additional Comments on the Convergence Guarantee of Prop In the main text, we noted that although the bound appears to grow with ||u]], the behaviour of characteristic functions at large || is typically negligible in practice. Characteristic functions of distributions with smooth, bounded densities tend to decay rapidly often exponentially or a super-polynomial rate.
To make this precise, consider the case: the characteristic function ((u) Fourier transform the density. the density is k-times differentiable, then it is well known (c.g.. Lemma p. 514, Feller 1971) that lo(u) o(u -k. This implies that for sufficiently large ju], the characteristic function becomes negligible in magnitude.
(k) Thus, assuming both Pdata and Po are smooth with bounded support, it suffices to match their characteristic functions over a compact domain |u| < for some U > 0. Such local agreement in the Fourier domain implies closeness of the
corresponding densities, and hence the distributions.
We complete this section by showing the connection between our framework and the original consistency loss. Proposition 3. Assume that the denoising network D& is implemented to satisfy Do(., 0) Id(-). When r 0, the consistency loss in Eq (9) is equivalent to the denoising noise in Eq (4) for 8.
Proof. When t denoising noise in Eg (4) becomes
E E DI (X,, 8) xol13 Ey, E Epop- (x,, x x=
Po Pajo
EEp, Epoi. [1] D (x,,
Epor. [Xu] + Epgr. [xol xoll 2?
=Ep, Epn, D (X,, Epp,, [xo]||]] ||]] + Ep, Eyoi= [[| [xn] xall
+ 2 Ep, Epoir (xe(x).)) Epoix (xol. Epoj- [2o]] - Xo.
(IDD Ep.. [ Const.
(1 (x, Epp. DD Xo, 0)][ + Const.,
which is the consistency loss Eq (9) when r = 0.
19

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
D. Additional Sampling Results
In this section, we present model--eenrruie samples used for FID computation in Sec 6. The samples are taken from the models at their fine-tuning iteration with the lowest FID.
D.I. CIFAR-10
Samples for computing FIDs in Fig 3(a) Clean Image Ratio
Figure 7: Clean image ratio = 0.04 - FID: 6.31
Figure 8: Clean image ratio = 0.1 3.58
Figure 9: Clean image ratio -ID: 2.98
20

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
Samples for computing FIDs in Fig 3(b) Noise Level
Figure 10: Noise level og = 0.30 FID: 3.97
Figure II: Noise level og 0.59 FID: 6.31
Figure 12: Noise level 0ç 1.09 - -.09-FID:000 9.43
21

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
Figure 13: Noise level oę = 1.92 - FID: 10.91
Samples for computing FIDs in Fig 3(c)- Pretraining on Similar Datasets
Figure 14: Class for fine-tuning fne-tuning: automobile FID: 10.39
Figure 15: Class for fine-tuning: ship Ahi: 19.19
22

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
Figure 16: Class for fine-tuning: horse FID: 48.11
Figure 17: Class for fine-tuning: no pretrain - FID: 155.04
D.2. CelebA
Figure 18: cfg A: oç 1.38; 1,500 clean images for pretraining FID: 5.91
23

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
Figure 19: cfg B: og 1.38; 50 clean images for pretraining FID: 23.63
2
Figure 20: efg C: aç 0.20; 50 clean images for pretraining - FID: 6.48
24

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
E. Experiment Configurations
E.1. Model Architectures
We implemented the proposed SFBD algorithm based on the following configurations throughout our empirical sudies:
Table 3: Experimental Configuration for CIFAR-10 and CelebA
Parameter
CIFAR-10
CelebA
General
Batch Size
512
256
Loss Function
EDMLoss (Karras et al., 2022)
EDMLoss (Karras et al., 2022)
Sampling Method
2dd order Heun method (EDM)
2nd order Heun method (EDM)
(Karras al., 2022)
(Karras al., 2022)
Sampling steps
18
40
Network Configuration
Dropout
0.13
0.05
Channel Multipliers
[2,2,2]
(1,2,2.2)
Model Channels
128
128
Resample Filter
{1,1}
(1,3,3,1) 1)
Channel Mult Noise
2
Optimizer Configuration
Optimizer Class
Adam (Kingma & Ba, 2015)
Adam (Kingma & Ba, 2015)
Learning Rate
0.001
0.0002
Epsilon
10-8
10-8
Betas
(0.9, 0.999)
(0.9, 0.999)
E.2. Datasets
All experiments CIFAR-IO (Krizhevsky Hinton, 2009) use only the training set, except for the presented in Fig 3(c). merge the training and test
For this specific the FID computation measures the distance between clean images of trucks and those from the classes which the model is fine-tuned. For subsequent iterations, FID is calculated in the same manner as in other experiments. Specifically, the model first generates 50,000 images, and the FID computed between the sampled images and the images from the fine-tuning classes. All experiments on CelebA (Liu et al., 2015) are performed on its training set.
that each class contains total of 6,000 images. At iteration
25

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
F. Data Leakage and Sample Memorization
As discussed in the main text, SFBD does not aim to protect the clean pretraining data from leakage. However, since these clean samples are assumed to be publicly available and free of copyright restrictions, their potential exposure poses no privacy concerns. SFBD is instead explicitly designed to safeguard sensitive data. The key privacy- preeeccie mechanism that each sensitive sample is only presented to the model a single, corrupted form throughout training. This restriction prevents the model from memorizing or reconstructing the original content, thus reducing the risk of reproducing private or copyrighted material.
To empirically validate this claim, adopt the evaluation protocol of Daras al. (2024), which involves analyzing similarity score distributions and identifying the most similar sensitive sample for each generated image. We report and discuss the results for both CIFAR-10 10 and CelebA. Overall, the findings indicate that SFBD does not regenerate sensitive examples, supporting its privacy-reeeervig properties.
E.1 CIFAR-10
In Fig 21, we show the distribution of maximum similarity scores for models trained with SFBD under various noise levels. The model with noise level = 0 is trained on the full, uncorrupted dataset. To compute similarity, we embed both the generated images (50k samples) and the sensitive dataset images into the DINOv2 (Oquab al., 2024) latent space. For each generated image, record the maximum inner product (i.e., similarity score) with its closest sensitive neighbour. As illustrated in Fig 22, images with similarity scores below 0.93 are visually distinct. Since almost all samples fall below this threshold, Fig 21 indicates that SFBD effectively avoids memorization of sensitive data while promoting sample diversity. Additionally, the figure shows that similarity scores steadily decrease as the noise level increases, supporting the privacy-preserving nature of SFBD and indicating tradeoff between image quality and data leakage risk.
Distribution of Similarity Values
3000
Noise Level
0.93
Number of generated samples
2500
0.5853
1.0882
2000
1500
1000
500
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Cosine Similarity
Figure 21: Distribution of maximum similarity scores for models trained with SFBD under varying noise levels. models are pretrained using 4% clean samples. while the remaining (sensitive) data are corrupted with noise. (The model with noise level 0 is trained the full, uncorrupted dataset.)
F.2. CelebA
Since human faces share highly similar structures, their similarity scores are generally much higher than those observed in CIFAR-10. Consequently, instead of showing full similarity distributions, we directly present the top matching pairs between the generated images (50,000 samples) and their most similar counterparts in the sensitive dataset (used to create
26

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
0.980
0.969
0.975
0.973
0.957
0.949
0.948
0.940
0.930
0.912
0.918
0.902
0.901
0.900
Figure 22: Generated images from the pretrained EDM on CIFAR-10 10 (top row) and their most similar images from the CIFAR-10 training set (bottom row). Images appear visually distinct when the similarity score falls below 0.93.
the noisy training set).
The results for cfg A and cfg C are shown in Fig 23 and Fig 24, respectively, Visual inspection confirms that the model trained with SFBD does not memorize the noisy training data.
0.980
0.978
0.978
0.978
0.978
0.977
0.977
0.977
0.977
0.977
0.977
0.976
0.976
0.976
0.976
0.976
Figure 23: Top matching image pairs from the CelebA dataset. The model is trained in efg. A specified in the submission (1,500 clean images, a 1.38). Each column shows a pair: the generated sample (top) and the most similar sensitive image (bottom) to generate noisy samples. Scores represent similarity computed via DINOv2 (Oquab et al., 2024) and we list the top 18 pairs that give the largest similarity score.
27

---

SFBD: Training Diffusion Models with Finite Noisy Datasets
0.981
0.980
0.979
0.979
0.979
0.979
0.978
0.978
0.978
0.978
0.978
0.978
0.978
0.978
0.977
0.977
Figure 24: Top matching image pairs from the CelebA dataset. The model is trained in cfg C specified in the submission (50 clean images a = 0.2). Each column shows pair: the generated sample (top) and the most similar sensitive image (bottom) to generate noisy samples. Scores represent similarity computed via DINOv2 (Oquab et al., 2024) and we list the top 18 pairs that give the largest similarity score.
28


Reviews:
[{"review_id": "EjIQUlelgN", "reviewer": "Reviewer_MuhL", "summary": "The authors consider the problem of training diffusion models with a small set of clean data and a large set of noisy data. This follows a line of recent works on developing techniques for training diffusion models under corruption in the training set. The main finding of this work is that without clean data performance is fundamentally limited for datasets of finite sizes. The authors develop a technique that significantly improves the performance by leveraging a small set of clean samples.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for the detailed feedback. \n\nBefore addressing specific points, we would like to clarify that the remark stating \"the benchmarking of this work is unfair/incomplete\" is, in our view, **inaccurate**. Specifically,\n\n1. We were not aware of the missing baseline [1] before our submission. We hope you would understand since [1] is very recent and there is no way we could have compared to it. We did our best to discuss its relation to our method below (in **A1**), which will also appear in our revision.\n2. Regarding the claim of unfair implementation: our version of TweedieDiff is not the same as in Daras25 [1] but faithfully follows the original paper (see  **A4** for details). While we’re happy to include results from [1] and discuss the differences, we do not have any intention to “paints an unfair and very pessimistic picture of the baselines.”\n\n**Q1.** Regarding a related work [1] published in ICLR 2025\n\n**A1.** We agree that this very recent work [1] is relevant. We will include a detailed discussion and benchmark comparison in the revision. In particular:\n\nBoth papers show that learning the true data distribution from only noisy samples is theoretically possible but practically requires an infeasible number of samples. To address this, both incorporate a small amount of clean data to guide training.\n\nHowever, they differ in approach. Our method uses the density deconvolution, while Daras et al. build on Gaussian Mixture Models (GMMs), providing more precise modeling under stronger distributional assumptions. We believe the fact that both methods reach similar conclusions—despite their different foundations—reinforces each other.\n\nMethodologically, [1] applies Tweedie’s formula (consistency constraint) to recover the clean distribution. In contrast, ours introduces a novel forward-backward deconvolution strategy, offering a fresh perspective without the heavy computational cost of enforcing consistency.\n\n(Benchmark Comparison)\nWe will include the following CIFAR-10 FID results in the revision:\n- 50 clean images, σ = 0.2 (Table 1 setting): Daras25: 8.05; SFBD: 13.53\n- 10% clean images, σ = 0.2: Daras25: 2.81; SFBD: 2.58\n- 4% clean images, σ = 0.59: Daras25: 6.75; SFBD: 6.31\n\nDaras25 performs better with very limited clean data, likely due to overfitting during our model’s pretraining on small datasets. This overfitting is lessened as more clean data becomes available, allowing SFBD to outperform. We attribute this to SFBD’s stable fine-tuning via score-matching loss, which avoids the extra constraints of consistency-based methods.\n\n**Q2.** Regarding training models in a way similar to EMU\n\n**A2.** We illustrate this on CIFAR-10 by pretraining a diffusion model on noisy images (σ = 0.2), then finetuning on either 50 or 4% clean images. FID trajectories [[link](https://shorturl.at/7B0gq)] show an initial drop followed by a rise—sharper with 50 clean images, slower with 4%. This is expected: finetuning on clean data improves performance at first but eventually causes the model to forget useful noisy features, leading to degradation.\n\n**Q3.** Regarding ambient diffusion\n\n**A3.**  The reported value is from the arXiv version of [2], which states that \"AmbientDiffusion is trained with the standard setting.\" Since it was omitted from the NeurIPS final version and its reliability is uncertain, we will remove it in the revision.\n\n**Q4.** Regarding TweedieDiff\n\n**A4.** **Our implementation closely follows the original TweedieDiff paper**. Comparing Daras25 [1] and Daras24 [3], we think the difference might be caused by the new implementation in Daras25 [[link](https://shorturl.at/226Rv)], which we were not aware of before our submission. We will include results from both versions and clarify their differences in the revision. We are happy to include the new result of Daras25 in our revision and add explanation on the different results. We want to assure the reviewer we tried our best to compare all baselines fairly (to be best of our knowledge).\n\n**Q5.** Regarding memorization\n\n**A5.**  See Reviewer kYMA-**A6**.\n\n**Q6.** Regarding MISE\n\n**A6.** MISE is the standard metric in density estimation that evaluates error uniformly across all x, encouraging agreement both within and outside the support of $p_{\\rm data}(x)$. Weighting the error by $p_{\\rm data}(x)$ instead reduces the influence of low-likelihood regions, under-penalizing high-density estimates in areas where the true density is low. This can make models that generate unlikely or absurd samples appear to perform well, contradicting our goal of accurately modeling the full distribution.\n\n**Q7.** Regarding computation requirement\n\n**A7.** See Reviewer kYMA-**A2**.\n\n[1] How much is a noisy image worth? Data Scaling Laws for Ambient Diffusion. Daras et al., 2025.\n\n[2] An EM Algorithm for Training Clean Diffusion Models from Corrupted Observations. Bai et al., 2024.\n\n[3] Consistent Diffusion Meets Tweedie. Daras et al., 2024.", "claims_and_evidence": "Yes, the claims in this work are supported by clear evidence.", "methods_and_evaluation_criteria": "I find the benchmarking of this work unfair/incomplete.\n\n* Missing baselines: \n   * The work [How much is a noisy image worth? Data Scaling Laws for Ambient Diffusion](https://arxiv.org/abs/2411.02780) (ICLR 2025) proposes the same idea (leveraging a few clean examples) and reaches the same finding (performance with noisy data only is limited but a few clean samples can lead to a dramatic increase in performance). Despite that, the paper is not discussed/cited and there is no benchmarking against it. I believe the authors were probably unaware of this work, but it is super relevant and should be extensively discussed and benchmarked against.\n   * Another useful baseline to compare against would be the training on noisy data first and then fine-tuning on clean data. This is in some sense what is happening post-training in the foundational models (e.g. see the [EMU](https://arxiv.org/abs/2309.15807) paper).\n* Unfair implementation of baselines:\n    * Ambient Diffusion is a method developed for training diffusion models with linear corruptions (e.g. random inpainting). It is unclear how the authors use this method for the denoising case.\n    * The authors report that TweedieDiff achieves FID of 167.23 without clean data and FID of 65.21 with some clean data points on CIFAR-10 for $\\sigma=0.2$. However, the work \"How much is a noisy image worth\" reports: a) FID 12.12 without clean data and without consistency, b) FID 11.93 without clean data and consistency, and iii) FID 60.73 with naive full sampling on a model that was just trained with times $t: \\sigma_t>=0.2$. Using 10% of clean data further drives FID to 2.50. The reported FIDs of 167.53 and 65.21 in the submission paint an unfair and very pessimistic picture for the baselines. \n\n\nI also believe that for the evaluation to be more rigorous, the pair of (FID, memorization) needs to be reported.", "theoretical_claims": "I checked the proofs about MISE, Proposition 1 and the proposed method. The MISE proofs seem to be heavily based on prior work. I also have a question regarding MISE; wouldn't it make more sense if we weight the integral with p_{data}(x)? Why are errors in very unlikely datapoints (according to $p_{data}(x)$) important?", "experimental_designs_or_analyses": "I have several concerns regarding the experimental validation, as listed above. Repeating here:\n\nI find the benchmarking of this work unfair/incomplete.\n\n* Missing baselines: \n   * The work [How much is a noisy image worth? Data Scaling Laws for Ambient Diffusion](https://arxiv.org/abs/2411.02780) (ICLR 2025) proposes the same idea (leveraging a few clean examples) and reaches the same finding (performance with noisy data only is limited but a few clean samples can lead to a dramatic increase in performance). Despite that, the paper is not discussed/cited and there is no benchmarking against it. I believe the authors were probably unaware of this work, but it is super relevant and should be extensively discussed and benchmarked against.\n   * Another useful baseline to compare against would be the training on noisy data first and then fine-tuning on clean data. This is in some sense what is happening post-training in the foundational models (e.g. see the [EMU](https://arxiv.org/abs/2309.15807) paper).\n* Unfair implementation of baselines:\n    * Ambient Diffusion is a method developed for training diffusion models with linear corruptions (e.g. random inpainting). It is unclear how the authors use this method for the denoising case.\n    * The authors report that TweedieDiff achieves FID of 167.23 without clean data and FID of 65.21 with some clean data points on CIFAR-10 for $\\sigma=0.2$. However, the work \"How much is a noisy image worth\" reports: a) FID 12.12 without clean data and without consistency, b) FID 11.93 without clean data and consistency, and iii) FID 60.73 with naive full sampling on a model that was just trained with times $t: \\sigma_t>=0.2$. Using 10% of clean data further drives FID to 2.50. The reported FIDs of 167.53 and 65.21 in the submission paint an unfair and very pessimistic picture for the baselines. \n\n\nI also believe that for the evaluation to be more rigorous, the pair of (FID, memorization) needs to be reported.", "relation_to_broader_scientific_literature": "This paper follows a line of recent works on training diffusion models with corrupted data and its implications for performance, copyright and memorization. The authors do a very good job in describing the state of the literature in Section 2 (Related Work). The main finding of this work is very interesting, and it is that a small set of clean data is essential for performance when the assumption of infinite data is violated (i.e. in all practical settings).", "essential_references_not_discussed": "As mentioned above, the authors miss the work [How much is a noisy image worth? Data Scaling Laws for Ambient Diffusion](https://arxiv.org/abs/2411.02780) (ICLR 2025) which proposes the same idea (leveraging a few clean examples) and reaches the same finding (performance with noisy data only is limited but a few clean samples can lead to a dramatic increase in performance).", "other_strengths_and_weaknesses": "I have already listed the main Weaknesses of the work. On the strengths side:\n- The paper is clearly written.\n- The topic is very interesting and the findings very intuitive.\n- The finding confirms a recent finding from another work, which is in some sense, a positive thing since it shows its importance and its reproducibility.\n- The connection to density deconvolution is novel and the sample-complexity results are interesting.\n- The proposed framework, a sort of Expectation-Maximization applied to diffusion modeling, is very niche and it shows that there are other ways beyond consistency to extrapolate beyond the training data.", "other_comments_or_suggestions": "Please update the running title as it is currently listed as \"Submission and Formatting Instructions for ICML 2025\".", "questions_for_authors": "Please include a discussion of the missed work and if possible, benchmark against it.\nPlease also clarify the questions I asked regarding how you implemented the baselines and if possible update the Tables using the (significantly improved) numbers from the related work. I understand that the rebuttal time is limited, but if you could also provide a comparison with the baseline of first training on everything and then fine-tuning on only the clean data points, that would be really useful.\n\nI would also like to ask about the computational requirements for the method, as it seems to require K finetunings.\n\nIf my concerns are properly addressed, I will raise my score since I like the idea, the paper, and the analysis.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7eViRDWyc2", "reviewer": "Reviewer_ftq1", "summary": "This paper addresses the challenge of training diffusion-based generative models using datasets that are intentionally corrupted with noise to mitigate concerns around memorization and copyright infringement.  \nHowever, the authors show that in practice, the convergence rate for learning from noisy samples is extremely poor—on the order of $O((\\log n)^−2)$—making effective training from only noisy data infeasible.  \nThe key insight is to pretrain a diffusion model on a small set of clean (copyright-free) data and then iteratively refine it using the large noisy dataset. The algorithm alternates between backward sampling (a denoising step using the current model) and updating the denoiser with these generated samples. Over time, the model’s outputs converge to the true data distribution even though it only sees a tiny fraction of clean data.  \nThe authors theoretically validate SFBD’s convergence and demonstrate its practicality via empirical studies on CIFAR-10 and CelebA. Impressively, SFBD achieves competitive image quality with only 4\\% clean images (FID of 6.31 on CIFAR-10), outperforming other methods designed to train on noisy data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments. We will first clarify the distinction between our framework and standard denoising methods, and then address your comments point by point.\n\n**How our method differs from a standard denoising algorithm.**  SFBD alternates between denoising samples and fine-tuning the denoiser, allowing it to blend high-frequency details from clean data with global structure from noisy samples (see Section 6.3). Unlike standard denoising methods that reconstruct exact clean inputs, SFBD trains the denoiser to recover the full data distribution, resulting in more realistic samples.\n\nTo highlight SFBD’s advantages, we compare the FID of its denoised samples at each iteration ($\\mathcal{E}_k$ in Algorithm 1) with the full training set, alongside results from a state-of-the-art off-the-shelf denoiser, Restormer [1].\n\n**CIFAR-10 (4% clean images)**\n\n| $\\sigma$ | Model | Iter 1 | Iter 2 | Iter 3 | Iter 4 \n|-|-|-|-|-|-\n| 0.30 | SFBD | 6.16 | 3.42   | 2.68   | 2.35   \n| | Restormer | 53.87  \n| 0.59 | SFBD | 10.23 | 7.47 | 6.31 | 6.54   \n| | Restormer | 99.99\n| 1.09 | SFBD  |12.68  | 9.39 | 9.08 | 10.14\n| | Restormer | 132.69\n\n**CelebA**  \n\n| Setting | Model | Iter 1 | Iter 2 | Iter 3 | Iter 4 \n|-|-|-|-|-|-\n| 50 clean images, $\\sigma = 0.2$ | SFBD | 47.69 | 10.05 | 5.63 | 3.93\n| | Restormer | 18.90\n| 1,500 clean images, $\\sigma = 1.38$ | SFBD | 9.05 | 5.76 | 4.56 | 3.98\n| | Restormer | 227.91\n\nThe results show that SFBD can produce significantly more realistic samples than regular denoisers. \n\nBelow, we provide point-by-point responses to your comments.\n\n**Q1.** How does the method perform when you use a off-the-shelf denoiser trained on generic data?\n\n**A1.** As shown above, Restormer-denoised images consistently yield much higher FIDs than SFBD (reported in the paper). Since generative models trained on these images can’t surpass the FID of their targets, using off-the-shelf denoisers like Restormer results in significantly worse performance. We will include this discussion in the revision.\n\n**Q2.** Regarding Experimental Designs Or Analyses\n\n**A2.** The upper block of Table 1 assumes full access to both clean copyright-free and sensitive samples, so the results naturally outperform those from models trained on denoised data—making our method appear weaker, not stronger.\n\nFor the algorithms in the lower block, their original designs either assume only noisy data or use clean samples in their own specific way. Adapting them to leverage denoised samples would require substantial changes and could also raise fairness concerns. To maintain a consistent and fair comparison, we used available clean data for pretraining when applicable.\n\nAs noted in **A1**, the reported results in the above tables serve as upper bounds for models trained on denoised samples—whether from a pretrained denoiser (SFBD-Iter 1) or an off-the-shelf one (Restormer). Importantly, final SFBD models (after multiple training iterations) still outperform these upper bounds. We will clarify this in the revision to avoid potential concern.\n\n**Q3.** Regarding the case that noise variance is unknown. \n\n**A3.**   In our setting where noise is **intentionally** added to protect sensitive samples, it is reasonable to assume known noise variance (either directly controlled by the model developer or communicated to it securely).\n\nTo make the proposed framework compatible with scenarios where the noise level is unknown, it could be extended by incorporating noise level estimation techniques-such as those used in blind denoising methods. We consider this an interesting direction for future work.\n\n**Q4.** How does using a denoiser trained on multiple noise levels (blindly) affect the performance?\n\n**A4.**  The denoiser Restormer, discussed in the tables above as well as in **A1** and **A2**, is designed to handle multiple noise levels in a blind manner. As shown in our results and discussions, generative models trained on images denoised by Restormer cannot achieve performance comparable to those trained using SFBD.\n\n**Q5.** Regarding \"The approach assumes that the noise level of the noisy data is within the trajectory of the diffusion process, ...\"\n\n**A5.** In theory, diffusion models can handle noise levels ranging from zero to infinity. We would appreciate it if you could provide more details or clarification on this concern so we can better understand and address the issue.\n\n**Q6.** Regarding \"The performance of the approach degrade significantly when the clean data is out of the distribution\"\n\n**A6.** We included these ablation studies to support our theoretical claims and offer additional insights into the framework’s behavior. While we agree that using in-distribution clean data yields better results, the main takeaway is that pretraining on out-of-distribution data still provides a clear benefit compared to no pretraining at all.\n\n[1] Restormer: Efficient Transformer for High-Resolution Image Restoration. S. W. Zamir et al. CVPR 2022", "claims_and_evidence": "Yes.  \nThe approach is justified using theoretical guarantees, specifically the convergence rate to the true clean distribution is analyzed and bounded.  \nThe claims are supported by the empirical studies, showing that the approach is effective also in practice.", "methods_and_evaluation_criteria": "Almost.\n\nThe approach assumes that the noise level of the noisy data is within the trajectory of the diffusion process, which in many practical cases does not hold.", "theoretical_claims": "Yes. They seem correct.  \n- Proposition 1  \n- Theorem 1  \n- Theorem 2", "experimental_designs_or_analyses": "Yes, the method is compared to other methods designed to generate images given access only to noisy images. This comparison is not fair. The other methods do not require having a set of clean images, for the comparison to be fair one needs to test the methods after denoising the dataset with a denoiser network trained using the given clean images.", "relation_to_broader_scientific_literature": "The task discussed in the paper is very important, specifically when one does not have access to the true data and only the noisy version of the data is accessible. The approach shows that even in this case one can still learn the true data distribution in a very effective way.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "Strengths:  \n- The paper is well written.  \n- The approach is novel and interesting.  \n- Compared to other methods this approach shows great improvement.  \n- The method is supported by theoretical justification.\n\nWeaknesses:\n- The approach assumes that the noise level of the noisy data is within the trajectory of the diffusion process, which in many practical cases does not hold.  \n- The method requires access to clean data at the initial stage.  \n- The method is compared to other approaches but not fairly. The other methods do not require having a set of clean images, for the comparison to be fair one needs to test the methods after denoising the dataset with a denoiser network trained using the given clean images.  \n- The performance of the approach degrade significantly when the clean data is out of the distribution (FIgure 2C, when the model is pretrained on truck images, it performs poorly on horse images).", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- How does the method perform when you use a off-the-shelf denoiser trained on generic data?  \n- What will happen in the blind case when the noise variance is unknown?  \n- How does using a denoiser trained on multiple noise levels (blindly) affect the performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r9Gob8dkDB", "reviewer": "Reviewer_kYMA", "summary": "This work put forward a new training framework, dubbed Stochastic Forward–Backward Deconvolution (SFBD), a method for training diffusion models on noisy datasets while mitigating data memorization and copyright concerns.\nMoreover, the theoretical analysis demonstrated that training solely on corrupted data is inefficient, whereas pretraining on a small fraction of clean data significantly improves performance.\nImportantly, experimental results showed state-of-the-art sample quality with as little as 4\\% clean data in certain settings.\nBy bridging the gap between deconvolution theory and practical generative modeling, the proposed SFBD provides a scalable solution for privacy-preserving training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for reviewing our paper. Below are our responses to your comments:\n\n**Q1.** How many training iterations are required?\n\n**A1.** In most of our experimental settings, SFBD converges within four iterations. (see [[link](https://shorturl.at/I71Vq)] for results from additional iterations.) This is why we originally report the FIDs over the first four iterations.\n\nWhen the noise level is fixed, SFBD converges faster with more clean samples, as the pretrained model starts closer to the true data distribution (see also Sec 5: The Importance of Pretraining). Conversely, with a fixed number of clean samples, higher noise levels also speed up convergence. We hypothesize this is because higher noise levels obscure more information in the noisy samples, prompting SFBD to rely more on the clean data and thus more rapidly adapt by combining complementary features from the noisy sources.\n\n**Q2.** Regarding \"training efficiency, computational overhead, or scalability compared to standard DM\"\n\n**A2.** While SFBD involves multiple fine-tuning steps, each step has a similar cost to standard diffusion training, as it minimizes the regular conditional score-matching loss.\n\nMost training time is spent on pretraining and the first fine-tune, together matching the cost of training a regular diffusion model. Later fine-tuning steps are much faster—each taking less than 1/4 of standard training time in our setup. Overall, using SFBD with 4 fine-tunes takes about 1.75× the time of training a regular diffusion model.\n\nBackward sampling adds cost but is parallelizable. With 8 RTX 6000 GPUs, each sampling step takes under 30 minutes.\n\n**Q3.** Baseline model selections\n\n**A3.**  Our implementation follows EDM [1] with identical hyperparameters (using the FFHQ‑64×64 config for CelebA). As noted in EDM Appx F.3, the UNet backbones are nearly identical to those in DDPM and DDIM. AmbDiff, EMDiff, and TweedieDiff also use EDM backbones, so their architectures match ours. For SureScore, we use results from [2], which employs a slightly different UNet but with a similar parameter count.\n\n**Q4.** The theoretical analysis seems no help for the model convergence, especially the derived lower bound.\n\n**A4**. Thms 1 and 2 are not meant to describe SFBD’s convergence, but to illustrate the difficulty of estimating clean distributions from noisy data. Based on density deconvolution theory, they provide matching upper and lower bounds, implying a sample complexity of $\\mathcal{O}((\\log n)^{-2})$. This poor rate shows that training solely on noisy data is impractical, justifying SFBD’s use of limited clean samples for pretraining.\n\n**SFBD’s actual convergence guarantee is given in Prop 2**, which shows a rate of $\\mathcal{O}(1 / \\sqrt{K})$, where $K$ is the number of iterations in Algorithm 1.\n\n*Note*: The $K$ in Prop 2 and Alg 1 refers to the same quantity. In contrast, the $K$ in Thm 2 is an unrelated constant. To avoid confusion, we’ll revise the notation in Thm 2 to use $C’$ in the updated version.\n\n**Q5.** Can you explain more about the $K$ in $\\mathcal{O}(\\log n)^{-2}$?\n\n**A5.** In Thm 2, given $p_{\\rm data}$ and the noise level $\\sigma_\\tau$, the constant $K$ is a fixed positive value. (**Note**: This quantity will be renamed to $C'$ in the revised version to avoid confusion, as it is only used in Thm 2 and is unrelated to the iteration count $K$ in Alg 1 and Prop 2.)\n\n**Q6.** Regarding the data leakage issue. \n\n**A6.** The SFBD algorithm does not address potential leakage from the clean images used in pretraining. However, since these images are public and copyright-free, such leakage is not a concern in our setting. We also note that:\n- Some clean data is essential, as the problem is otherwise intractable;\n- Clean images can be copyright-free or obtained with user consent;\n- Pretrained models may be sourced from public datasets, though with potential quality trade-offs (as shown in ablation studies).\n\nSFBD is specifically designed to prevent leakage from sensitive data by exposing the model to only one corrupted version of each sensitive sample during training. This inherently limits memorization and reconstruction of private or copyrighted content. Notably, SFBD does not require clean images or the pretrained model to be released or shared through secure channels.\n\nTo further support our privacy claims, we follow [3] by computing similarity scores between generated and sensitive samples [[link](https://shorturl.at/19EOf)]. Results show no reconstruction of sensitive content. These findings will be included in the revision.\n\n**Q7.** Regarding addtional results on bigger datasets\n\n**A7.** See Reviewer BKmX - **A2**.\n\n[1] T. Karras et al. Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS 2023.\n\n[2] An Expectation-Maximization Algorithm for Training Clean Diffusion Models from Corrupted Observations. Bai et al. 2024.\n\n[3] Consistent Diffusion Meets Tweedie. Daras et al., 2024.", "claims_and_evidence": "The paper claims that training solely on noisy data is inefficient due to slow convergence rates but can be significantly improved by pretraining on a small fraction of clean data. While the theoretical analysis supports this claim, the extent to which pretraining on limited clean data generalizes to diverse datasets or real-world scenarios is not entirely explored. Additional experiments on different datasets and noise levels could further strengthen this conclusion.", "methods_and_evaluation_criteria": "This paper suggests that the proposed method may help address potential copyright issues, as it enables the recovery of noisy images without accessing their complete clean information.", "theoretical_claims": "For Theorem 1 & 2:\n\nThe theorem claims that the optimal convergence rate for estimating the data density from noisy samples is $\\mathcal{O}(\\log n )^{-2}$ , which implies that training diffusion models solely on corrupted data is nearly infeasible.\nHowever, it derived that the lower bound of the discrepancy between the modeling distribution and the target data distribution is  $\\mathcal{O}(\\log n )^{-2} \\cdot K$.\nSince this is a lower bound, then how can restrain the modeling discrepancy the modeling distribution and the target data distribution?\nMoreover, how $K$ determines the bound? is this very sensitive or just a constant.", "experimental_designs_or_analyses": "The comparisons with the baseline models are all based on the same model backbone? or otherwise it if unfair.\nIt seems two datasets are insufficient verification of model performance.", "relation_to_broader_scientific_literature": "The key contributions of this paper build on existing research in diffusion models, deconvolution theory, and generative modeling with noisy data. By connecting ideas from these fields, it offers new theoretical insights and practical techniques to enhance training efficiency, improve sample quality, and support privacy-preserving generative modeling.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "weaknesses:\n\n1. While the method improves sample quality, the paper does not provide a detailed analysis of training efficiency, computational overhead, or scalability compared to standard diffusion model training.\n\n2. Although the paper claims that training on noisy data can mitigate copyright risks, it does not provide a formal analysis or guarantees regarding information leakage or potential reconstruction of copyrighted content.\n\n3. The theoretical analysis seems no help for the model convergence, especially the derived lower bound.", "other_comments_or_suggestions": "Evaluating the proposed method on a broader range of datasets would significantly enhance the assessment of its robustness. \nWhile CIFAR-10 and CelebA provide useful benchmarks, incorporating larger and more diverse datasets, such as ImageNet for high-resolution images or LSUN for complex scenes, would better demonstrate the model’s generalizability.", "questions_for_authors": "1. How many training iterations are required for the proposed method?\n\n2. can you explain more about the $K$ in $\\mathcal{O}(\\log n )^{-2}$?\n\n3. Since the training process still requires some clean images, does it truly help resolve the data leakage issue?\n\nIf the authors can address most of my concerns, I will improve the initial score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RZVvWGlSK2", "reviewer": "Reviewer_BKmX", "summary": "This paper tackles an important practical issue: how to train diffusion models without directly accessing large volumes of clean (and potentially copyrighted) data. The authors propose a novel method called Stochastic Forward–Backward Deconvolution (SFBD). The approach begins with pretraining on a small set of clean images and then leverages a large noisy dataset through a forward–backward iterative process. The work is grounded in a solid theoretical framework based on density deconvolution, where the authors derive that the optimal convergence rate for learning from noisy samples is $O((\\log n)⁻²)$. This theoretical insight motivates the use of even a modest amount of clean data to guide the deconvolution process. Extensive experiments on CIFAR-10 and CelebA demonstrate that SFBD outperforms several baselines, including methods such as TweedieDiffusion and SURE-Score, in terms of FID scores.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the review. Below are our responses to your comments:\n\n**Q1.** Regarding the non-trivial gap when $|\\mathbf{u}|$ is large in Prop 2. \n\n**A1.** Thank you for this insightful comment. We agree that the bound appears to grow with $\\|\\mathbf{u}\\|$, and we would like to clarify that in practice, the behavior of characteristic functions for large $\\|\\mathbf{u}\\|$ is typically negligible. Specifically, characteristic functions tend to decay rapidly—often at an exponential or super-polynomial rate—for distributions with smooth and bounded densities.\n\nTo make the discussion concrete, consider the 1D case: the characteristic function $\\phi(u)$ is the Fourier transform of the probability density function. When the density is k-times differentiable, it is well known (e.g., Lemma 4 on page 514 of [2]) that the characteristic function satisfies $|\\phi(u)| = o(|u|^{-k})$. This implies that for sufficiently large $|u|$, the magnitude of the characteristic function becomes negligible.\n\nTherefore, under the assumption that both $p_{\\rm data}$ and $p_0^{(k)}$ are smooth and have bounded support, it suffices to ensure that their characteristic functions are close within a compact domain $|u| < U$ for some $U > 0$. This local closeness in the Fourier domain translates to closeness in the original densities, and hence the distributions. We will include this clarification in the final version of the manuscript.\n\n**Q2.** Regarding addtional results on bigger datasets\n\n**A2.** While we acknowledge that further experiments on larger datasets would strengthen the empirical study, our ability to scale up is constrained by the limited computational resources available at our academic institution. For instance, according to EDM’s official repository [1], training a diffusion model on ImageNet requires 32×A100 GPUs for 13 days—resources that are currently beyond our reach.\n\nTo partially address your comment, we conducted an experiment on the Tiny ImageNet dataset, which contains 100,000 images. We selected 1,000 clean images as copyright-free samples for pretraining and set the noise level to 0.2. Using a reduced batch size of 256 (compared to the recommended 4096 in EDM), the pretrained model achieved an FID of 36.74. After the first fine-tuning iteration, the FID dropped to 20.41. This result reflects a trend consistent with our findings on CIFAR-10 and CelebA, further corroborating our claims.\n\n**Q3.** The results are only shown for at most 4 iterations, it is better to show more iterations and specifically, how fast the results would converge to the optimal state.\n\n**A3.** In most of our experimental settings, SFBD converges within four iterations. (see  [[link](https://shorturl.at/I71Vq)] for results from additional iterations.) This is why we focus on reporting the FID trajectories over the four few iterations in our original submission. \n\nMoreover, when the noise level is fixed, SFBD tends to converge more quickly as the number of clean samples used for pretraining and fine-tuning increases. A larger clean dataset allows the model to start closer to the true data distribution (see also Section 5 – The Importance of Pretraining). Conversely, when the number of clean samples is fixed, increasing the noise level also accelerates convergence. We hypothesize this is because higher noise levels obscure more information in the noisy data, prompting SFBD to rely more on the clean samples, enabling faster adaptation through the fusion of complementary information from noisy samples. We will incoperate this discussion in the revision.\n\n[1] T. Karras et al. Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS 2023.\n\n[2] Feller, W. An Introduction to Probability Theory and Its Applications, Vol. 2. Wiley. 1971", "claims_and_evidence": "Most claims are supported by either theoretical analysis and experimental results well. However, I have one question:\n\nProposition 2 was presented in terms of the infinity norm between the chracteristic function between the data distribution and the iterative distribution. The results show that the norm not only depend on the iteration numbers K but also the l2 norm of the $u$ as well, which implies there are always non-trivial gap when $\\|u\\|$ is large enough no matter how large K is. In this sense, the theorem should be augmented by how the original distance/KL-divergence of the two distributions would be if the gap between the chracteristic function are only large when $\\|u\\|$ is large.", "methods_and_evaluation_criteria": "Experimental results for larger dataset, like imagenet, better to be demonstrated.", "theoretical_claims": "Correct.", "experimental_designs_or_analyses": "The results are only shown for at most 4 iterations, it is better to show more iterations and specifically, how fast the results would converge to the optimal state.", "relation_to_broader_scientific_literature": ".", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "1. The derivation of the convergence rate $O((\\log n)⁻²)$ under a Gaussian noise model provides deep insight into the limitations of learning from solely corrupted data. This analysis justifies the necessity of even a small amount of clean data.\n2. The experimental results, though limited, provide good insights.", "other_comments_or_suggestions": ".", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is a very interesting paper on how to train diffusions when given both clean data and noisy data. There is an interesting result, that even with a small amount of clean samples, the noisy data can help in learning the true (unknown) distribution. Overall, the paper is very interesting, studies an important problem and positions itself well compared to recent work in this very active space. 

In the discussions the authors also agreed to compare with recent related work which seems very closely related but indeed different. 
All additional concerns by the reviewers were sufficiently addressed by the author's rebuttals. 
Overall, this work seems satisfactory and ready for publication.
Decision: Accept (poster)