How Much is a  Noisy Image Worth? Data Scaling Laws for Ambient Diffusion.

Authors: Giannis Daras, Yeshwanth Cherapanamjeri, Constantinos Costis Daskalakis
Primary area: generative models
Keywords: ambient diffusion, noisy data, dataset design, gaussian mixtures

Abstract:
The quality of generative models depends on the quality of the data they are trained on. Creating large-scale, high-quality datasets is often expensive and sometimes impossible, e.g.~in certain scientific applications where there is no access to clean data due to physical or instrumentation constraints. Ambient Diffusion and related frameworks train diffusion models with solely corrupted data (which are usually cheaper to acquire) but ambient models significantly underperform models trained on clean data. We study this phenomenon at scale by training more than $80$ models on data with different corruption levels across three datasets ranging from $30,000$ to $\approx 1.3$M samples. We show that it is impossible, at these sample sizes, to match the performance of models trained on clean data when only training on noisy data. Yet, a combination of a small set of clean data (e.g.~$10\%$ of the total dataset) and a large set of highly noisy data suffices to reach the performance of models trained solely on similar-size datasets of clean data, and in particular to achieve near state-of-the-art performance. We provide theoretical evidence for our findings by developing novel sample complexity bounds for learning from Gaussian Mixtures with heterogeneous variances. Our theoretical model suggests that, for large enough datasets, the effective marginal utility of a noisy sample is exponentially worse that of a clean sample. Providing a small set of clean samples can significantly reduce the sample size requirements for noisy data, as we also observe in our experiments.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730003390294

thread_type:
official_review

<thread_object>
creator: reviewer_kj8h

<content>
summary:
This paper tackles a critical issue in training diffusion models with corrupted data when clean data is scarce or costly. The authors explore the effectiveness of a combination of a small clean dataset and a large noisy dataset, demonstrating that this approach can yield performance close to models trained solely on clean data. Theoretical analysis based on Gaussian Mixture Models (GMMs) with heterogeneous variances and experimental validation across multiple datasets support these claims. Additionally, the paper proposes a budget allocation mechanism for more efficient dataset curation by balancing the use of noisy and clean data.

strengths:
* The paper is well-written and logically structured, making the content easy to follow.
* The theoretical analysis is rigorous and provides good intuition behind the proposed methods. The experiments are thorough, involving the training of more than 80 models across various datasets and noise levels.
* The paper’s conclusions offer significant practical value for dataset curation and budget allocation strategies in real-world applications, where collecting clean data can be expensive.

weaknesses:
* The analysis is limited to Gaussian noise corruption and discrete distributions with finite support. Expanding this framework to continuous distributions and non-Gaussian noise would increase its impact.
* The proposed method assumes that the noise level is known in advance, which may not be the case in real-world scenarios. Incorporating techniques and analysis to handle unknown noise levels would make the approach more practical.

questions:
It remains unclear why an exponentially larger amount of corrupted data is necessary to compensate for the lack of clean data. Can the authors provide more clarification for this point?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730470049851

thread_type:
official_review

<thread_object>
creator: reviewer_vssn

<content>
summary:
First, the paper empirically shows that training on entire noisy data will lead to bad performance, but as long as a few clean data (~10%) are available, models trained by the proposed algorithm are comparable to models trained on clean data. Second, the paper derives minimax optimal estimation bounds to explain the phenomenon. Third, the paper pinpoints the price of noisy images for different datasets and noise levels, which has practical implications.

strengths:
1. The paper is written clearly.
2. In practice, we have small sets of clean data and large sets of noisy data, so the paper considers an important setting.
3. The experiments on different realistic image datasets are sufficient to support the claim in the paper.
4. The established estimation bounds are minimax optimal and discussed clearly. 
5. The price of noisy images is interesting.

weaknesses:
1. The obtained $\hat{D}$ is the ERM result for the Wasserstein distance loss, which is mismatched with the loss of diffusion models. Thus, there exists a gap between theory and practice.
2. The proposed algorithm and proof technique are mainly based on existing works, which may decrease the novelty of this paper.

questions:
1. How can we know the value $\sigma_{t_n}$ in the inputs of Algorithm 1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730569981251

thread_type:
official_review

<thread_object>
creator: reviewer_zaes

<content>
summary:
Quick summary: This paper studies the performance of the diffusion model in the ambient diffusion scenario. Previous work shows that the diffusion model trained by noisy data is inferior to the model trained on clean data, and the consistent diffusion model requires the noisy samples to be infinity from the asymptotic view. This paper proposes to use the mixture of clean data (high-frequency feature) and corrupted data (low-frequency feature) to assist the ambient diffusion model to obtain a very close performance to the state-of-the-art and provide the theoretic analysis to support the experiment and evaluate the corresponding value of noise image using the inspiring sampling bounds from the theoretical results.
Quality: The paper is well-written and well-motivated, and the proposed method is good with a theoretical foundation albeit a bit simple.
Originality: The theoretical analysis of the two sampling bounds is inspiring with practical meaning. 
Significance: This is an important research direction because noise data is common in the real world.
Pros: * Important problem * theoretical foundation * nice results and sufficient experiments
Cons: * The solution is a bit simple and kind of engineering consideration.   
Summary: This is a nice paper that gives a simple solution to the ambient diffusion models with theoretic analysis and while not groundbreaking, certainly merits a publication.

strengths:
The theoretical analysis of the two sampling bounds is inspiring with practical meaning.

weaknesses:
The solution is a bit simple and kind of engineering consideration.

questions:
What's the exact number of samples or dimensions in your settings in line 93, page 2?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703631144

thread_type:
official_review

<thread_object>
creator: reviewer_brqj

<content>
summary:
This paper studies data scaling laws for ambient diffusion. The authors find that training on a small set of clean data and a large set of noisy data significantly outperforms training solely on either one of these sets. The paper provides both theoretical analysis and empirical results to verify their findings.

strengths:
1. The studied problem is interesting and important.
2. The authors provide theoretical results to support their findings.

weaknesses:
1. I'm a bit confused about the relationship between thm 4.2 and alg 1. The thm 4.2 states that there exists a procedure which can return a good estimate of D. Is Algorithm 1 the procedure referenced in this theorem?
2. Alg 1 need to know whether each data is clean or noisy. Is it possible to generalize the proposed algorithm to handle cases where clean and noisy data are mixed?
3. In the experiments, The paper only presents the empirical results of the proposed method and does not compare it with previous work. (Daras et al., 2024; 2023b).

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713964639

thread_type:
official_review

<thread_object>
creator: reviewer_mdgl

<content>
summary:
The paper investigates the impact of noisy data on (ambient) diffusion models. The paper shows that noisy data can be leveraged to improve the performance of diffusion models. In addition they provide the data regime where the noisy data are useful.

strengths:
- the paper is well written
- the tackled problem is very interesting
- experiments are exhaustive and convincing (Figure 1 shows the regime of interest for using noisy data)

weaknesses:
- I am not familiar with proof techniques for the mixture of Gaussian sample complexity, but I am not sure I understood the goal/take-away of Theorems 4.2-4.4. IMO much more discussion is needed here.
- Lack of motivation: the tackled problem is overall very interesting, but I am not sure that the authors provided a concrete application where such new noisy data are available, and known as such.

questions:
- ¨For example, a blurry image might get dismissed from the filtering pipeline¨ Can authors provide a real example of such a filtering, i.e., a real filter used and a real discarded image?
- "For realistic sample sizes" Do you have an order of magnitude of "realistic sample size" in your context?
- Figure 1: in addition to training on noisy data, are other data augmentation techniques used? if not, do you know how new noisy data would compare against already existing standard data augmentation?


- The heteroscedastic model: I do not understand how the proposed model in Definition 4.1 is a heteroscedastic mixture of Gaussian. As currently written in the paper, it feels that all the points of the distribution are convolved with the Gaussian noise (which is coherent with the proposed setting). Following Shah (2023), shouldn't a heteroscedastic mixture of Gaussian write $X_i \sim \\mathcal{N}(\mu_i, \sigma_i)$.
In other words, IMO maths matches the setting, but I am not sure of the ¨heteroscedastic mixture of Gaussian¨ terminology.

- Regarding the cost of the augmentation, are all models trained with the augmentation, do you confirm that all models are trained with the same budget? (i.e., with the same number of images seen, since the datasets are different) I guess yes since EDM code takes ¨million number of images¨ as input


Theorem 4.2:
- what are $c_1$ and $c_2$? Absolute constants as defined in Vershynin (2018)?
- "Discussion" I did not understand the discussion after Theorem 4.2, could you elaborate on the "error in estimating the low-dimensional sub-space" and the "low-dimensional estimation procedures"
- "Then, there is a procedure which when given n independent ...." could you give a word on the procedure?

- Algorithm 1: is the noise level of each sample require in Algorithm 1?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper explores the challenge of training generative models when access to large-scale, high-quality datasets is limited. The authors demonstrate that a small subset of clean data (e.g., 10% of the total dataset) combined with a larger set of highly noisy data is sufficient to achieve performance comparable to models trained solely on similar-sized clean datasets, reaching near state-of-the-art results. To support these findings, the paper provides theoretical evidence through novel sample complexity bounds for learning from Gaussian Mixtures with heterogeneous variances. The theoretical analysis reveals that, for sufficiently large datasets, the effective marginal utility of a noisy sample is exponentially lower than that of a clean sample. This insight is further corroborated by experiments showing that a small clean subset significantly reduces the sample size requirements for noisy data.


Pros:
+ Important Problem Setting: The paper addresses the practical challenge of training diffusion models with noisy data, which is common in real-world applications.
+ Combination of Theory and Empirical Results: The work provides theoretical insights into sample complexity for Gaussian Mixture Models (GMMs) with heterogeneous variances, alongside extensive experiments demonstrating the effectiveness of using a small clean subset with a large noisy dataset.
+ Clear Presentation: The paper is well-written and logically structured, making the arguments and findings easy to follow.
+ Practical Utility: The findings offer significant implications for dataset curation strategies, balancing the use of noisy and clean data to achieve near state-of-the-art results.
+ Thorough Experiments: The study involves training over 80 models across datasets of varying noise levels, providing empirical validation of the claims.

Cons:
- Limited Novelty: The proposed algorithm and proof techniques are largely based on existing works, which raises questions about the paper's novelty.
- Narrow Scope of Analysis: The theoretical analysis is restricted to Gaussian noise and discrete distributions with finite support, limiting the applicability to more general noise models or continuous distributions.
- Gap Between Theory and Practice: The theoretical results focus on the Wasserstein loss, which does not perfectly align with the actual loss used in diffusion models, creating a disconnect between theory and implementation.
- Assumption of Known Noise Levels: The proposed method assumes that the noise level is known, which may not hold in practical scenarios. Addressing unknown noise levels would improve the method’s applicability.
- Simplistic Solution: While effective, the proposed solution is considered by some reviewers as overly simple and primarily engineering-driven.

After the revision as discussed below all reviewers recommend acceptance. I concur.