Is Noise Conditioning Necessary for Denoising Generative Models?
Authors: Qiao Sun
Zhicheng Jiang
Hanhong Zhao
Kaiming He
Venue: ICML 2025 poster (2025)
Keywords: noise conditioning
generative models
diffusion
score-based generative models
flow matching
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: Contrary to common belief, noise conditioning is not essential for denoising generative models, as most models perform well—or even better—without it, supported by theoretical analysis and a noise-unconditional model achieving competitive results.

Abstract: It is widely believed that noise conditioning is indispensable for denoising diffusion models to work successfully. This work challenges this belief. Motivated by research on blind image denoising, we investigate a variety of denoising-based generative models in the absence of noise conditioning. To our surprise, most models exhibit graceful degradation, and in some cases, they even perform better without noise conditioning. We provide a mathematical analysis of the error introduced by removing noise conditioning and demonstrate that our analysis aligns with empirical observations. We further introduce a noise-*unconditional* model that achieves a competitive FID of 2.23 on CIFAR-10, significantly narrowing the gap to leading noise-conditional models. We hope our findings will inspire the community to revisit the foundations and formulations of denoising generative models.

Is Noise Conditioning Necessary for Denoising Generative Models?
Qiao Sun Zhicheng Jiang"
Hanhong Zhao
Kaiming He
Abstract
It is widely believed that noise conditioning is indispensable for denoising diffusion models to work successfully. This work challenges this be- lief. Motivated by research on blind image denois- ing, we investigate a variety of denoising- -based generative models in the absence of noise con- ditioning. To our surprise, most models exhibit graceful degradation, and some cases, they even perform better without noise conditioning. We provide theoretical analysis of the error caused by removing noise conditioning and demonstrate that our analysis aligns with empirical observa- tions. We further introduce a noise-unconditional model that achieves a competitive FID of 2.23 on CIFAR-10, significantly narrowing the gap to leading noise-conditional models. We hope findings will inspire the community to revisit the foundations and formulations of denoising gener- ative models.
NN
NN₂
Target
Target
(a)
(b)
Figure I: (a) A denoising generative model takes a noisy data z and a noise level indexed by t (such as σ₂) as the inputs the neural network NNg. (b) This work investigates the scenario of removing noise conditioning in the network.
estimated from corrupted data, making blind denoising (i.e., without knowing the noise level) feasible task. Notably, noise-level estimation and blind image denoising have been active research topics for decades (Stahl et al., 2000; Salmeri et al., 2001; Rabie, 2005), with neural networks offering effective solutions (Chen et al., 2018; Guo et al., 2019; Zhang et al., 2023). This raises an intriguing question: can related research on image denoising be generalized denoising-based generative models?
1. Introduction
At the core of denoising diffusion models (Sohl-Dickstein al., 2015) lies the idea of corrupting clean data with various levels of noise and learning reverse this process. The remarkable success of these models has been partially underpinned by the concept of "noise conditioning" (Sohl- Dickstein et al., 2015; Song & Ermon, 2019; Ho al., 2020): a single neural network is trained to perform denoising across all noise levels, with the noise level provided as a conditioning input. The concept of noise conditioning has been predominantly incorporated diffusion models and is widely regarded critical component.
Motivated by this, in this work, we systematically com- pare a variety of denoising- generative models with and withou! noise conditioning. Contrary to common belief, we find that many denoising generative models per- form robustly even in the absence of noise conditioning. In this scenario, most methods exhibit only modest degrada- tion in generation performance. More surprisingly, find that some relevant patitood--rrttcullaryy flow-based ones (Lipman et al., 2023; Liu et al., 2023), which originated from different perspectives can even produce improved generation results without noise conditioning. Among all the popular methods we studied, only one variant fails dis- astrously. Overall, our empirical results reveal that noise conditioning may not be necessary for denoising generative models to function properly.
In this work, we examine the necessity of noise conditioning in denoising- based generative models. Our intuition is that, in natural data such as images, the noise level can be reliably
'Equal contribution. Listing order is random. 'MIT. Cor- respondence to: Qiao Sun <sqa24@mii.edu>> Zhicheng Jiang <jzc_2007 @ mit. edu>, Hanhong Zhao <zhh24 mdu. Kaim- ing He <kaiming @ @it.eduu>.
We present a theoretical analysis of the behavior of these models the absence of noise conditioning. Specifically, we investigate the inherent uncertainty in the noise level dis- tribution, the error caused by denoising without noise condi-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Is Noise Conditioning Necessary for Denoising Generative Models?
tioning, and the accumulated error in the iterated sampler. Put together, we formulate an error bound that can be com- puted without involving any training, depending solely on the noise schedules and the dataset. Experiments show that this error bound correlates well with the
2023; Albergo et al., 2023) reformulate and generalize the framework by learning flow fields that map one distribution to another. In all these methods, noise conditioning (also called time conditioning) is the de facto choice.
Beyond diffusion models, Consistency Models (Song et al., 2023) have emerged as a new family of generative models for non-iterative generation. It has been found (Song & Dhariwal, 2024) that noise conditioning and its implementa- tion details are critical for the success of consistency models, highlighting the central role of noise conditioning.
behaviors of the models we studied particulallyy in cases where the model fails catastrophically, its error bound is orders of magnitudes higher.
Because noise- uncoondiiniiii models have been rarely con- sidered, it is worthwhile to design models specifically for this underexplored scenario. To this end, we present a sim- ple alternative derived from the EDM model (Karras et al., 2022). Without noise conditioning, our variant can achieve a strong performance, reaching an FID of 2.23 on the CIFAR- 10 dataset. This result significantly narrows the gap between noise-uncondittioal system and its noise-conditional coun- terpart (e.g.. 1.97 FID of EDM).
Blind Image Denoising. In the field of image processing, blind image denoising has been studied for decades. It refers to the problem of denoising an image without any prior knowledge about the level, type, or other characteristies of the noise. Relevant studies include noise level estimation from noisy images (Stahl et al., 2000; Shin et al., 2005; Liu et al., 2013; Chen et al., 2015), as well as directly learning to perform blind denoising from data (Liu et al., 2007; Chen et al., 2018; Batson & Royer. 2019; Zhang et al., 2023). Modern neural networks, including the U-Net (Ronneberger et al., 2015) commonly used in diffusion models, have been shown highly effective for these tasks.
Looking ahead, we hope that removing noise conditioning will pave the way for new advancements in denoising- based generative modeling. For example, only in the absence of noise conditioning can a score- based model learn a unique score function and enable the classical, physics-grounded Langevin dynamics.' Overall, we hope that our findings will motivate the community to re-examine the fundamental principles of related methods and explore new directions in the area of denoising generative models.
Our research is closely related to classical work on blind denoising. However, the iterative nature of the generative process, where errors can accumulate, introduces new chal- lenges. In addressing these challenges, our work opens up new research opportunities that extend classical approaches.
2. Related Work
Noise Conditioning. The seminal work of diffusion mod- els (Sohl-Dickstein et al., 2015) proposes iteratively perturb- ing clean data and learning a model to reverse this process. In this pioneering work, the authors introduced a "time de- pendent readout function", which is an early form of noise conditioning.
3. Formulation
In this section, we present a reformulation that can summa- rize the training and sampling processes of various denoising generative models. The core motivation of our reformula- tion is to isolate the neural network NNg, allowing us to focus on its behavior with respect to noise conditioning.
The modern implementation of noise conditioning is popu- larized by the introduction of Noise Conditional Score Net- works (NCSN) (Song& & Ermon, 2019). NCSN is originally developed for score matching. This architecture is adopted and improved in Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., 2020), which explicitly formulate gen- eration as an iterative denoising problem. The practice of noise conditioning has been inherited in iDDPM (Nichol & Dhariwal, 2021), ADM (Dhariwal & Nichol, 2021), and nearly all subsequent derivatives.
3.1. Denoising Generative Models
Training Objective. During training, a data point x is sampled from the data distribution p(x), and a noise is sampled from a noise distribution p(E), such normal distribution A/ I). A noisy image is given by:
z = a(t)x + b(t)e.
(I)
Here, a(t) and b(t) are schedule functions that are method- dependent. The time step t, which can be a continuous or discrete scalar, is sampled from p(t). Without loss of generality, we refer to b(t). or simply f, as the noise level.
DDIM (Song et al., 2021a) and EDM (Karras et al., 2022) reformulate the reverse diffusion process into an ODE solver, enabling deterministic sampling from a single initial noise. Flow Matching (FM) models (Lipman et al., 2023; Liu et al.,
In general, a denoising generative model involves minimiz- ing a loss function that can be written as:
I Otherwise, relies on the annealed Langevin dynamics (Song & Ermon, 2019)) that does not correspond to unique underlying probability distribution independent of noise levels.
C(0) Ex.e. [(t)][NNo(zlt) - r(x, r(x,,,t))²] (2)
2

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Table 1: Schedules used by different models in our reformu- lation. Notations and details are in Appendix D.
r(x r6
iDDPM, DDIM
EDM
a(1)
Va(()
Mi
b(t)
VT-a())
r(X2 (2)77)
o(t)
ayy -
d(t)
+
Figure 2: Illustration of the effective target R(z). A given z corresponds to multiple triplets (x, c, G, Here, we take Flow Matching (Lipman et al., 2023) as an example. On the left are the samples of and on the right are samples of x. For a noisy sample (1 t)x + te, it can be produced by different triplets. Each triplet gives different regression target r. The effective target [(z)) is the expectation of all possible r.
Here, NNg a neural network (e.g., U-Net) to be learned, r(x, e, t) is a regression target, and w(t) is weight. The regression target r can be written as:
c. c()x c(t)x+((())) d(t)c,
(3)
where c(t) and d(t) are also wrethod-speciffe schedule func- tions. Common choices of r include 6-prediction (Ho et al., 2020), x-prediction (Salimans & Ho, 2022), or v-prediction (Salimans& & Ho, 2022; Lipman et al., 2023).
(left). This is commonly implemented as t-embedding, such as Fourier (Tancik et al., 2020) positional embedding (Vaswani, 2017). This t-embedding provides time-level in- formation as an additional input to the network. Our study concerns the influence of this noise conditioning, that is, we consider NNo(z) (z) vs. NNe(z)t). See Fig. (right). Note that NNg (z) or (Z involves all learnable parameters in the model, while the schedules (a(t). b(t), etc.) are pre- designed and not learned.
The specifics of the schedule functions of several existing is worth noting that, in our refor-
methods are in Table mulation, we concern the regression target r with respect to the neural network NN2's 's direct output."
Given trained NNg, the sampler performs it- erative denoising. Specifically, with an initial noise xo N/, b(tmmxx) )2)) the sampler iteratively computes:
Sampling.
x K2X; (x, Gre,
(4)
4. Analysis of Noise-Uncondition Models
Here, a discrete set of time steps {ty} pre-specified and indexed by 0 & i N. The schedules, Ki, Mhi and Çi. can be computed from the training-time noise schedules in Table 1 (see their specific forms in Appendix D). In Eq. (4), é; ~ (N I) is sampling- -imelnng--meeeeeee noise that only takes effect in 3DE-based solvers; there is no noise added in ODE-based solvers, i..,, G
Based on the above formulation, we present a theoretical analysis of the influence of removing noise conditioning. Our analysis involves both the training objectives and the sampling process. We first analyze the effective target of regression at the training stage and its error in single denoising step (Sections 4.1 to 4.3), and then give an upper bound on the accumulated error in the iterative sampler (Section 4.4). Overall, our analysis provides an error bound that to be examined by experiments.
Eq. (4) is a general formulation that can encapsulate many first-order samplers, such as (annealed) Langevin sampling and Euler-based ODE solver. Higher-order samplers (e.g., Heun) can be formulated similarly with extra schedules. In this paper, our theoretical analysis is based on Eq. (4), and order cases are evaluated empirically.
4.1. Effective Targets
While the loss function is often written in a form like Eq. (2), the underlying unique regression target for NZ)(z is not r(x, e, (). The function NNo(2)() which w.r.t. and is regressed onto multiple values corresponding different possible triplets (x, t, that produce the same (see Fig. 2). Intuitively, the unique effective target, denoted as R(z(() to emphasize its dependence on z and f, is the expectation of over all possible triplets.
3.2. Noise Conditional Networks
In existing methods, the neural network (Zo() is con- ditioned on the noise level specified by t. See Fig.
For methods like EDM where the network output is wrapped with a precondition, rewrite the schedules to expose the term of NNo (see Appendix D.3). This network NNo is called the "raw network" in EDM (see Eq. (8) (Karras et al., 2022)).
Formally, optimizing the loss in Eq. (2) is equivalent to optimizing the following loss, where each term inside the
3

---

Is Noise Conditioning Necessary for Denoising Generative Models?
expectation [[] has a unique effective target:
((0)- Bx-p(e), [| R(
(5)
600-
Here, p(z) is the marginalized distribution of z:=a(t)x++ + b(t)e in Eq. (1), under the joint distribution e,),,, == p())) 3 It is easy to show that:
500
400-
A(z t) = (xx.:///ntx.t...o//xxxxxx.
(6)
120-
that is, the expectation over all (x, E) subject to the condi- tional distribution. One can show (Appendix C.1) that mini- mizing Eq. (5) is equivalent to minimizing Eq. (2), and sim- ilar analysis has also been done in previous work(Lehtinen et al., 2018).
5:0 0.30
01.50
0.40
0.50
0.72
0.82
1000
Figure 3: The Posterior distribution p((]z) is concen- trated. We picked z = (1 (1- fx)xx++ f4e with &s from 0.1 to 0.9 for illustration. This plot is empirically simulated from 15,000 images in the AFHQ-v2 dataset with a size 64 64 (see Appendix A.2).
Effective Targets without Noise Conditioning. Similarly, if the network (N)(z) does not accept as the condition, its unique effective target R(z) should depend on only. In this case, the loss is:
Intuitively, this statement suggests that higl--iimensional data induces a sharply peaked p(L z). In Appendix C.2, we derive rigorous upper bound on this variance and extend the analysis to the multi-data -oint setting. To corroborate these theoretical findings, we empirically run a simulation on a real dataset and plot p(t|z) (see Fig. 3). The empiri- cal distribution of p(f|z) well concentrated. Moreover, smaller t* leads to more concentrated p(t|z), as also indicated by Eq. (9).
(() Eu-p(e) [INNO()) R(
where the unique effective target is:
R(z)= = Ex-p((ia) R(z]t)].
[R(z)))
(8)
Eq. (8) suggests that if the conditional distribution p(t (z) is close to a Dirac delta function, the effective largel would be the same with and without conditioning on t. If so, assuming the network is capable enough to fit the target, the noise- unconditional variant would produce the same output as the conditional one.
4.3. Error of Effective Regression Targets
With p(t|z), we investigate the error between the effective regression targets R(z) and R(z](). Formally, we consider:
4.2. Concentration of Posterior p(t|z)
E(z) := Ex-p((t))
[[(())) R(2)
(10)
Next, we investigate how similar p(t|z) is to a Dirac delta function. For high-dimennsooal data such as images, it has been long realized that the noise level can be reliably estimated (Stahl et al., 2000; Salmeri et al., 2001; Shin et al., 2005), implying a concentrated p((]z). We note that the concentration of p(t|z) depends on data dimensionality:
We show that this error F(z) is substantially smaller than the norm of R(z):
Statement (Error of effective regression targets). Con- sider the scenario in Statement I and the Flow Matching case. The error defined in Eq. (10) satisfies:
Statement 1 (Concentration of p(t]z)). Consider a single datapoint x E [-1, _] e~N(O, I). t~U[0, 1], and (1 t)x + te (the Flow Matching case). Given noisy image z (1 )x + tyE produced by given tar the variance of under the conditional distribution p(t]z), is:
E(z) ~ 1 [1
(11)
when the data dimension d satisfies 3 & t and 1 < 1 to- Here, aa denotes the per-pixel standard deviation of the dataset. (Derivation in Appendix C.3)
d
12
Varz=pitle
(9)
2d'
Intuitively, Statement 2 suggests that for sufficiently high- dimension d, the error E()) is substantially smaller (x1) than the L2 norm of the target R(z) (sd). In our real-data verification, we find that E(z) is at the order of 1/103 of R(z) (see Appendix A.2). In this case, regressing to R(z(t) can be reliably approximated by regressing to R(z).
when the data dimension d satisfies 1 << t, and & - 4. (Derivation in Appendix C.2)
For simplicity, we consider =1,)=1, which happens be the case for all methods Table when we expose NNo explicitly.

---

Is Noise Conditioning Necessary for Denoising Generative Models?
4.4. Accumulated Error in Sampling
5. A Noise Unconditional Diffusion Model
Thus far, we have been concerned with the error of a sin- gle regression step. In a denoising generative model, the sampler at inference time is iterative. We investigate the accumulated error in the iterative sampler.
In addition to investigating existing models, we also design diffusion model specifically tailored for noise uncondition- ing. Our motivation to find schedule functions that are more robust in the absence of noise conditioning, while still maintaining competitive performance. To this end, we build upon the highly effective EDM framework (Karras et al., 2022) and modify its schedules.
To facilitate our analysis, we assume the network NNg is sufficiently capable of fitting the effective regression target R(z(t) or R(z). Under this assumption, we replace NNo in Eq. (4) with R. This leads to the following statement:
A core component of EDM is "preconditioned" denoiser:
Statement 3 (Bound of accumulated error). Consider a sampling process, Eq. (4), of N steps, starting from the same initial noise Xo = Xb- With noise conditioning, the sampler computes:
Cakip (esip(t)e (1)2 Coul NN) (N t)
Here, := x te is the noisy input before the nor- malization performed by Cm(t),* which we simply set as (l)( रमम The main modification we adopt for the noise unconditioning scenario is set:
X1+1 K2X, R(x((t) + GLE,,
and without noise conditioning, it computes:
Coat (t) 1.
x(+1 = K,X, + R(x)) + SiE..
As a reference, EDM set Clme(1) Vag+ez where da is the data std. As Cou (t) is the coefficient applied to NNø, we expect setting it to a constant will free the network from modeling a t-dependent scale. In experiments (Section 6.2), this simple design exhibits lower error bound (Statement 3) than EDM. We name this model as uEDM, which is short for (noise- Juncondttional EDM. For completeness, the re- sulting schedules of uEDM are provided in Appendix D.5.
oat
Assuming [R(x[tt)) R(x/(t)) [X] -xxl & L, and [ R(x(44) & 8, it can be shown that the error between the sampler outputs xN and x'y is bounded:
x'vl ||xN-xvl Ao A + ... AN-1BN-1 (12)
where:
N-1 A [ (k+ and B, = 1,=ln,
5-1+1
6. Experiments
depend on the schedules and the dataset. (Derivation in Appendix C.4)
Experimental Settings. We empirically evaluate the im- pact of noise conditioning across a variety of models:
Here, the assumption on & can be approximately satisfied as per Statement 2. The assumption on L, models the function R(-It) as Lipschitz-continuous Although it is unrealistic for this assumption to hold exactly in real data, we empirically find that an appropriate choice of L; can ensure the Lipchitz condition holds with high probability (Appendix A.3).
Diffusion: iDDPM (Nichol & Dhariwal, 2021). DDIM (Song et al., 2021a), ADM (Dhariwal & Nichol, 2021),
EDM (Karras et al., 2022), and uEDM (Sec. 5)
Flow-based Models: we adopt the implementation of Rectified Flow (1-RF) (Liu et al., 2023), which is a form of Flow Matching (Lipman al., 2023) (FM).
Statement 3 suggests that the schedules Kę and тı are influ- ential to the estimation of the error bound. With different schedules across methods, their behavior in the absence of noise conditioning can be dramatically different.
Consistency Models: iCT (Song & Dhariwal, 2024) and ECM (Geng al., 2025).
Our main experiments are on class-unconditional gener- ation on CIFAR-10 (Krizhevsky et al., 2009), with extra results on ImageNet x2 (Deng et al., 2009), and FFHQ 64 64 (Karras et al., 2019). We evaluate Fréchet Inception Distance (FID) (Heusel et al., 2017) and report Number of Function Evaluations (NFE). For a fair comparison, all methods are based on our re-implementation as faithful as possible (see Appendix B.3): with and without noise condi- tioning run in the same implementation for each method.
Discussions. Remarkably, the bound estimation can be computed without training the neural networks: it can be evaluated solely based on the schedules and the dataset.
Furthermore, our analysis of the "error" bound implies that the noise-conditional variant is more accurate, with the noise-unconditional variant striving to approximate it. In fact, there is no reason to assume that the former should be a more accurate generative model. Nonetheless, in ex- periments, we find that the noise-umconditionn case can outperform its noise-conditional counterpart in some cases.
"To make notations consistent with our reformulation in Eq. (2), we denote z = Cin ē. See details Appendix D.3.
5

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Table 2: Changes of FID scores in the absence of noise conditioning, for different methods on CIFAR-10. 10. Here 'w/o means without noise conditioning. color of yellow denotes non-disastrous (and often decent) degradation; green denotes improvement; red denotes failure.
10
DDIM
EDM
Per-stop Error Bound (ABB:
FM (1-RF)
10
VEDM
109
model
sampler
NFE
w/o
10
1DDPM
SDE
500
3.13
5,51
iDDPM (x-pred)
SDE
500
5.64
+ 6.33
10
ODE
100
40.90
DDIM
SDE
100
8.07
10.85
10-1
SDE
1000
3.18
5.41
ADM
SDE
250
2.70
5.27
20
40
60
80
100
Step index
Heun
3,36
BDM
Euler
50
2.98
4.55
Model
accum,
Fuler
100
3.01
2.61
DDIM
3e6
3.99
40.90
FM (I-RF)
Hcun
99
2.87
2.63
EDM
lc3
2,34
3.80
RK45
127
2.53
2.63
(I-RF)
le2
3,01
2.61
uEDM (Sec. 5)
le2
2.62
ICT BCM
2
2.59
3.57
2
2.57
3,27
Figure 4: Error bound and the influence of noise con- ditioning. ODE with N = 100 steps is applied for each variant. The plot shows the per-step error bound A, B, in Eq. (12), and the table shows the accumulated error bound.
uEDM (Sec. 5)
Heun
35
2.04
2.23
6.1. Main Observations
The y-axis is log-scale.
Table 2 summarizes the FID changes in different generative models, with and without noise conditioning, denoted "w/ f" and "w/o t". Fig. 5 shows some qualitative results. We draw the following observations:
qualitative behavior: the model is still able to make sense of shapes and structures; is "overshoot" or "undershoot", producing over-saturated noisy results.
(i) Contrary to common belief, noise conditioning is not an enabling factor for most denoising-base models to function properly. Most variants can work gracefully, exhibiting small but decent degradation (yellow).
Summary. Our experimental findings highlight that noise conditioning, though often helpful for improving quality. is not essential for the fundamental functionality of denoising generative models.
(ii) More surprisingly, some flow-based variants achieve improved FID (green) after removing noise conditioning. In general, flow-based methods investigated in this paper are insensitive to whether we use noise conditioning or not. We hypothesize that this is partially because FM's regression target independent on (see Table 1: c = d.
6.2. Analysis
Error Bound. In Fig. 4, we empirically evaluate the error bound in Statement 3 for different methods under a 100-step ODE sampler. The computation of the bound depends only on the schedules for each methods, as well as the dataset (detailed in Appendix A.3).
(iii) The uFDM variant (Sec. 5) achieves competitive FID of 2.23 without noise conditioning, narrowing the gap to the strong baseline of the noise-condin conditionl methods (here, 1.99 of EDM, or 1.97 reported in Karras et al. (2022)).
Fig. 4 shows strong correlation between the theoretical bound and the empirical behavior. Specifically, DDIM's catastrophic failure can be explained by its error bound that is orders magnitudes higher. On the other hand, EDM, FM, and uEDM all have small error bounds throughout. This is consistent with their graceful behavior in the lack of noise conditioning.
(iv) Consistency Models (here, iCT and ECM), which are re- lated diffusion models but present substantially different objective function, can also perform gracefully. While iCT was found highly sensitive the subtleties of t-conditioning (see Song& & Dhariwal (2024)), we find that removing it does not lead to disastrous failure.
These findings suggest that the error bound derived in our analysis serves as reliable predictor of a model's robust- ness to the removal of noise conditioning. Importantly, the bound can be computed solely based on the model's for- mulation and dataset statisties, without training the neural
(v) Among all variants we investigate, only "DDIM w/ ODE sampler" results in a catastrophic failure (red), with FID significantly worsened 40.90. Fig. (a) demonstrates
6

---

Is Noise Conditioning Necessary for Denoising Generative Models?
(a) DDIM (FID: 3.99 10.90)
(b) EDM (FID: 1.99 3.36)
(c) FM (I-RF) (FID: 3.01 - 2.61)
(d) uEDM (FID: 2.04 2.23)
Figure 5: Samples of noise-conditional vs. noise-unconditiinal models. Samples are generated by (a) DDIM, (b) EDM, (c) FM (I-RF), and (d) uEDM, on the CIFAR-10 daassunnconiiional case. For each subfigure, the left panel the noise-conditional case, and the right panel is the noise-unconditional counterpart, with the same random seeds. The change of FID is from "w/ t" to "w/o f", See also Table 2 for more quantitative results.
further investigate the level of stochasticity in Fig.
SDE
ODE
Specifically, with the flexibility of DDIM (Song al., 2021a), one can introduce a parameter À that interpolates be- tween the ODE and SDE samplers by adjusting 1/: and G in Eq. (4) (see Eq. (56) in Appendix D.2). As shown in Fig. 6, increasing À (more stochasticity) consistently improves FID scores. When X 1, DDIM behaves similarly to iDDPM.
30
20
10-
We hypothesize that this phenomenon can be explained by error propagation dynamics. Our theoretical bound in Statement 3 assumes worst-case error accumulation, but in practice, stochastic sampling enables error cancellation. The ODE sampler's consistent noise patterns lead to cor- related errors, while the SDE sampler's independent noise injections at each step promote error cancellation. This error cancellation mechanism can improve performance with in- creasing stochasticity, as further evidenced by iDDPM and ADM's results (Table 2) produced by SDE.
01
0.0
0.2
0.3
0.6
0.8
1.0
A (stochasticity)
Figure 6: Influence of Stochasticity on DDIM, in the lack of noise conditioning. The level of stochasticity is specified by X, with À = 0 denoting the ODE case. Here, the number of sampling steps is fixed as 500.
network. Consequently, it can provide valuable tool for estimating whether given denoising generative model can function effectively without noise conditioning, prior to model training.
Alternative Noise-conditioning Scenarios. Thus far, we have focused on removing noise conditioning from existing models. This is analogous to blind image denoising in the field of image process. Following the research topic on noise level estimation, we can also let the network explicitly or implicitly predict the noise level. Specifically, we consider the following four cases (Fig. 7):
Level of Stochasticity. In Table 2, DDIM only fails with the deterministic ODE sampler (the default sampler in (Song et al., 2021a)); it still performs decently with the SDE sam- pler (i.e,, the DDPM sampler). With this observation, we

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Table 3: Changes of FID scores in the absence of noise conditioning, on class-unconditional ImageNet 32x 32 and FFHQ 64x64, and class-conditional CIFAR-10.
MSE
FID
Model
Sampler
NFF
w/ w/o
NN;
N
ImageNet 32x
FM (I-RF)
Fuller
100
5.15 -+ 4,85
FFHQ 64x64
Target
Target
Target
Target
EDM
Heun
79
2.64 -. 3.59
CIEAR-10 Class-conditional
(a)
(b)
(c)
(d)
EDM
Heun
35
1.76 -+ 3.11
FM (I-RF)
Buler
100
2.72 - 2.55
Model
(a)
(b)
(c)
(d)
1DPMM
2.69
4.91
4.95
EDM
1.99
3.27
3.39
3.36
6.3. Extra Datasets and Tasks.
(M
3.01
2.58
2.65
2.61
Thus far, our experiments have been on the CIFAR-10 class- unconditional task. To show the generalizability of our findings, we further evaluate class-unconditionn generation on ImageNet 32 x 32, FFHQ 64x64, and class-conditional generation on CIFAR- 10. See Table 3.
Figure 7: Alternative Noise-conditional Scenarios. (a)
Noise- Voise-conditional baseline. (b) Noise-conditional but t' predicted by a noise level predictor P. (c) Similar to (b), but the noise level predictor is not supervised and is trained jointly. (d) Noise un-conditional baseline. For iDDPM, EDM, and FM, all of (b), (c), and (d) perform similarly.
The behavior is in general similar to that in our previous experiments. Specifically, removing noise conditioning can also be effective for other datasets the class-cone conditiona generation task. FM can exhibit improvement in the absence of noise conditioning; EDM has decent degradation, but experience no catastrophic failure.
(a) The standard noise-conditioning baseline, which is what we have been comparing with. See Fig. 7(a).
(b) A noise-conditioning variant, in which the noise level is predicted by another network. In this variant, the noise predictor P is a small network pre-trained to regress f. This predictor is then frozen when training NNg. and NNg is conditioned on the predicted t', rather than the true See Fig. 7(b).
6.4. Classifier-Free Guidance
We further examine the impact of omitting noise condition- ing when using classifier-free guidance (CFG) (Dhariwal Nichol, 2021), standard technique for significantly improv- ing sample quality in class-conditional diffusion models.
Our experiments with CFG conducted on the ImageNet 256x 256 dataset on SiT (Ma et al., 2024), which is flow- matching variant of DìT (Peebles & Xie, 2023). For com- parison, we train SiT-B/2 under the original paper's 's configu- ration for both noise-conditional and unconditional model. At inference, we employ an Euler sampler with 250 steps and vary the CFG scale. See more details in Appendix B.
(c) An "unsupervised" moise-conditioning variant. This architecture is exactly the same as the variant (b), ex- cept that the noise predictor P is trained from scratch without any ground-truth I. If we consider P and NNo jointly as a larger network, this also represents design for noise-unconditional modeling. See Fig. 7(c).
(d) The standard noise-unconditional baseline, which is what we have been investigating. See Fig. 7(d).
The results in Fig. 8 indicate that removing the noise condi- tioning incurs almost no degradation at different guidance scales, corroborating our analysis.
Fig. 7 compares all four variants. Notably, consistent be- havior is observed for all models (iDDPM, EDM, and FM) studied here: the results of (b), (c), and (d) are similar. This suggests that (b). (c). and (d) could be subject to the same type of error, that is, the uncertainty of estimation. Note that even in the case of (b) where the noise predictor is pre-trained with the true l given, its prediction cannot be perfect due to the small yet inevitable uncertainty in p(t|z) (see Section 4.2). As a result, the supervised pre- pre-trained noise predictor (b) does not behave much different with the unsupervised counterpart (c).
7. Discussion and Conclusion
We hope that rethinking the role of noise conditioning will open up new opportunities. Modern diffusion models are closely related to Score Matching (Hyvärinen & Dayan, 2005; Song & Ermon, 2019; Song et al., 2021b), which pro- vides an effective solution to Energy-Based Models (EBM) (Hopfield, 1982; Ackley et al., 1985; LeCun et al., 2006; Song & Kingma, 2021). The key idea of EBM is to represent probability distribution p(x) by p(x) e -E(e)) ZZ where
8

---

Is Noise Conditioning Necessary for Denoising Generative Models?
tunities Program (UROP). We thank Cheng Jiang, Yiyang Lu, Mingyang Deng. Xingjian Bai, and Xianbang Wang for their comments and discussions.
45
w/
10
w/o
35
30
Impact Statement
255
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential socictal consequences of our work, none which we feel must be specifically highlighted here.
-5-
1.0
1.5
2.0
2.5
3.0
3.5
4.0
(80 37G)
CFG scale
References
Figure 8: Classifier-free guidance results for SiT-B/2 on ImageNet 256 We an Fuler sampler with 250 steps. The best guidance scale in each setting is marked with a star. All other experimental details follow the original paper. Removing noise conditioning leads to almost no degradation regardless of guidance scale.
Ackley, D. H., Hinton, G. E., and Sejnowski, J.J. A learning algorithm for boltzmann machines. Cognitive science, 9 (1):147-16,, 1985.
Albergo, M. S., Boffi, N. M., and Vanden-Eijnden E. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797 2023.
B(x) is the energy function. With the score function of p(xr) (that is, vx)())), one can sample from the underlying p(z) by Langevin dynamics. This classical formulation models the data distribution p(x) by a single energy function E(x) that is solely dependent on z. Therefore, classical EBM inherently i-unconditional. However, with the presence of t-conditioning, the sampler becomes annealed Langevin dynamics (Song & Ermon, 2019), which implies a sequence of energy functions [Exx, h) indexed by f, with one sam- pling step performed on each energy. Our study suggests that it is possible to pursue a single energy function E(x), aligning with the goal of classical EBM.
Batson, J. and Royer, L. Noise2self: Blind denoising by
self-supervision. In International Conference on Machine Learning, pp. 524-533. PMLR, 2019.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations Python+NumPy programs, 2018.
Chen, G., Zhu, F., and Ann Heng, P. An efficient statistical method for image noise level estimation. Proceed- ings of the IEFE International Conference on Computer Vision, pp. 477-485, 2015.
Our study also reveals that certain families of models, e.g., Flow Matching (Lipman et al., 2023; Liu et al., 2023; Al- bergo et al., 2023), can be more robust to the removal of I-conditioning. Although these models are closely related to diffusion, they can be formulated from substantially differ- a flow field between two distri- butions. While these models can inherit the t-conditioning design of diffusion models, their native formulation does not require the flow field to be dependent on t. Our study sug- gests that there exists single flow field for these methods to work effectively.
Chen, J., Chen, J., Chao, H., and Yang, M. Image blind de- noising with generative adversarial network based noise modeling. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3155-3164, 2018.
ent perspective-estimaaii
Deng, J., Dong, W., Socher, R., Li, I.-.,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, -4:8780-8794, 2021.
In summary, noise conditioning has been predominant in modern denoising-based generative models and related ap- proaches. We encourage the community explore new models that are not constrained by this design.
Geng. Z.. Pokle, A., Luo, W., Lin, J., and Kolter, J. Z. Consistency models made easy. In The Thirteenth Inter- national Conference on Learning Representations, 2025.
Acknowledgements
Guo, S., Yan, Z., Zhang, K., Zuo, W., and Zhang, L. Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1712-1722 2019.
We greatly thank Google TPU Research Cloud (TRC) for granting us access to TPUs. Q. Sun, Z. Jiang, and H. Zhao are supported by the MIT Undergraduate Research Oppor-

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Liu, C., Szeliski, R., Kang, S. B., Zitnick, C. L.,, and Free- man, W. T. Automatic estimation and removal of noise from single image. IEEE transactions pattern analy- sis and machine intelligence, 30(2):299 2007.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-66 -116640-685,,,, 2020.
Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. In International Conference Learning Representations, 2020.
Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554-2558 1982.
Liu, X., Tanaka, M., and Okutomi, M. Single-image noise level estimation for blind denoising. IEEE transactions image processing, 22(12):5226-5233 2013.
Liu, X., Gong, C., and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023.
Hyvärinen, A. and Dayan, P. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4395,4405, 2019. doi: 10.1109/ CVPR.2019.004550
Ma, N., Goldstein, M., Albergo, M. S., Boffi, N. M., Fijdenn,ijnden, E., and Xie, S. Sit: Exploring flow and diffusion- generative models with scalable in- terpolant transformers. arXiv preprint, arXiv:2401.082 2024. doi: 10 10.48550/arXiv.24011.ooo 008740. URL https:
Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., and Aila, T. Alias-free generative adversarial networks. Advances in neural information processing systems, 34:852-866 863, 2021.
/ //arxiv. org abs 2401.08740
Nichol, A. Q. and Dhariwal, P. Improved denoising diffu- sion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of iiffusion-basee generative models. Advances in neural information processing systems, 35: 26565-26577, 2022.
Peebles, W. and Xie, Scalable diffusion models with trans-
formers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4195-4205. 2023.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Confer- ence Track Proceedings, 2015.
Rabie, T. Robust estimation approach for blind denoising. IEEE transactions on image processing, 14(11):1755- 1765, 2005.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted intervention- 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., Huang, F., et al. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Salimans, T. and Ho, J. Progressive distillation for fast sam- pling of diffusion models. In International Conference on Learning Representations, 2022.
Lehtinen, J., Munkberg, J., Hasselgren, J., Laine, S., Karras, T., Aittala, M., and Aila, T. Noise2Noise: Learning image restoration without clean data. arXiv preprint, 1rXiv:1803.04189, 2018. URL https:// arxiv.
Salmeri, M., Mencattini, A., Ricci, E., and Salsano, A. Noise estimation in digital images using fuzzy processing. In Proceedings 2001 International Conference on Image Processing (Cat. No. 01CH37205), volume 1, pp. 517- 520. IEEE, 2001.
wariii.rrg//. wrg. abs/ 18003000188
Lipman, Y., Chen, R. T. Q., Le, M. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023.
H., Nickel, M., and
Shin, D.-.., Park, R.-H., Yang, S., and Jung, J.-H. Block- based noise estimation using adaptive gaussian filtering.
10

---

Is Noise Conditioning Necessary for Denoising Generative Models?
IEEE Transactions on Consumer Electronics, 51(1):218- 226, 2005.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a.
Song, Y. and Dhariwal, P. Improved techniques for train- ing consistency models. In The Tweifth International Conference on Learning Representations, 2024.
Song. Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Song, Y. and Kingma, P. How to train your energy-basee models. arXiv preprint (0t.::20010032888 2021.
Song, Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, Consis- tency models. In International Conference on Machine Learning, pp. 32211 1-32252. PMLR, 2023.
Stahl, V., Fischer, A., and Bippus, R. Quantile based noise estimation for spectral subtraction and wiener filtering. In 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 00CH37100), volume 3, pp. 1875-1878. IEEE, 2000.
Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Kein S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., and Ng, R. Fourier features let networks learn high fre- quency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Zhang, K., Li, Y., Liang, J., Cao, J., Zhang, Y., Tang, H., Fan, D.-P., Timofte, R., and Gool, V. Practical blind image denoising via swin-conv-unet and data synthesis. Machine Intelligence Research, 20(6):822-88) 836, 2023.

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Contents
Introduction
2
Related Work
3 Formulation
3.1 Denoising Generative Models
3.2 Noise Conditional Networks
Analysis of Noise-Unconditions Models
4.1 Effective Targets
4.2 Concentration of Posterior p(t|z)
4.3 Error of Effective Regression Targets
4.4 Accumulated Error in Sampling
5
A Noise Unconditional Diffusion Model
Experiments
6.1 Main Observations
6.2
Analysis
6
6.3 Extra Datasets and Tasks.
8
6.4
Classifier-Free Guidance
7 Discussion and Conclusion
8
Appendices
13
Appendix A Details of Numerical Experiments
13
A.I Computation of Relavent Quantities
13
A.2 Numerical Experiments
15
A.3 Evaluation of the Bound Values
15
Appendix B
Additional Experimental Details
17
B.1
General Experiment Configurations
17
B.2
Special Experiments
18
B.3 Our Reimplementation Faithfulness
19
Appendix C
Supplementary Theoretical Details
19
C.1 Proof of the Effective Target
19
C.2 Approximation the Variance of p(t|z)
12

---

Is Noise Conditioning Necessary for Denoising Generative Models?
C.2.1
Rigorous Upper Bound Variance.
21
C.2.2
Intuitive derivation of the approximate constant
24
C.3 Approximation of E(z)
25
C.4 Bound Accumulated Error
26
Appendix D Derivation of Coefficients for Different Denoising Generative Models
27
D.I iDDPM
27
D.2 DDIM
................................
29
D.3 EDM
30
D.4 Flow Matching
31
D.5 Our uEDM Model the Formulation
31
Appendix E Additional Samples
32
Appendices
A. Details of Numerical Experiments
In this section, we provide additional details on all our real dataset numerical experiments, By first computing the value of some relevant quantities (e.g. the underlying time distribution p(1]z), effective target R(()(), R(z)), we are able to evaluate Ezz), which average error introduced by removing noise conditioning. See Appendix A.I.
As we introduce single data point assumption in our theoretical framework, we verify the accuracy of this assumption by comparing the empirical values of p((|z) and E(z) with the theoretical values derived from estimations. See Appendix A.2.
Finally, in Appendix A.3, we show how we derive the numbers in Fig. 4 by detailing on our estimation of bound values A, and B, in Statement 3.
A.1. Computation of Relavent Quantities
We consider the data distribution Pdata constituted solely from the data points in the dataset: Pxaa(xx) Σ 5(x x:), where x; e Rd are the images the dataset, and 5(.) is the delta distribution. We denote N as the number of data points in the dataset, and d as the dimension the image.
Calculation of p(t|z) (Section 4.2). First, we calculate p(z(t) by marginalizing over all the data points:
p(a(() [(2)11
Σ
p(z)1, x;).
The random variable is given by Eq. (1), which implies
plz|(,)) N (2:a(()x
(13)
where N22; n,, E) denotes the probability density function of the Gaussian distribution with mean μ and covariance Σ.
This leads to
1 p(z)t) N
Σ N b(t))
(14)
13

---

Is Noise Conditioning Necessary for Denoising Generative Models?
and we finally obtain:
N
p(t)
Σ N b(t))
p(() plz)
p(t|z) p(z)t)
(15)
S. p(t) Σ N (z: a(txx, b()))) dt
Note that there is an integral to evaluate in Eq. (15). In practice, the calculation is performed in two-step manner for fixed z. In the first step, we use a uniform grid of 100 / values in [0, 1] (i.e. = 0.00, 0.01....,0... 0.99). We calculate the value of p(t)p(z]t) for each value.
Typically, we observe that within interval [2, r1 (where < < 1), the value of p(t)p(z)t) significantly larger than for other / [0, 1] S. We then approximate the integral as:
N ! p(t) Σ N (z,(txx)x,, dt v p(t) Σ N (2),,))x,
N b(t)) dt.
(16)
In the second step, evaluate the integral by using uniform grid of 100 t values in [t, This two-step procedure effectively reduces computational costs while maintaining low numerical error.
Calculation of R(zlt) and R(z) (Section 4.1). By definition,
R(z(t)= == Ex [)] Ex-p(x)a.1) c(f)x + d(t) 2 a(f)x
b(1)
Notice that p(x|2, f) = p(x)
p(z(() p(z)x, t). and p(z|x, 1) is given in Eq. (13). Consequently, we have
Σ plz|xx, ((())x; + d(t)) + z-)())x; M()
Σ N ((tatt)x;,(tt)1, x,
d(t).
z + ((t) a(t)d(t)
R(z]t)-
(17)
b(t)
b(t)
p(z)xi,t)
N (z;a(t)x, b(t)2
which can be then explicitly calculated by scanning over all the data points Once we obtain R(z[t), using Theorem and p(t/z), R(z) can calculated by
R(z) = 12- [((z)))) -S. p(2))d)
(18)
For the integration, we utilize the selected time steps in [4, r] that were used when computing p(t]z). On another word, we ignore the parts where p(t|z) is negligible.
Calculation of Ezz) (Section 4.3). E(z) can be computed simply utilizing p(t]z), and R(z):
E(z) EE R(z,t) R(2)]1 p(t]))||R(((t R( dt.
(19)
We again use the same time steps for estimating the integral term and reuse the terms of p(())x;,)) t). This ensures computational efficiency while maintaining accuracy.
"Actually, this exactly matches our observation that pff2)) concentrated, since p(tlz)x<p(t)p(z)) fixed
14

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Table 4: The variance and E values on the CIFAR-I0 dataset. The empirical values are calculated by scanning the entire dataset, while the (theoretical) estimated values are derived from Statements and 2. For reference, the values for [R(2)]]? are also included to illustrate that (((z) is significant smaller than [[(2)] ? The results show that our approximation generally accurate, except for the E value in the very noisy case, where the single data point approximation becomes less accurate.
h*
Var,- Vatl)
E(z)
[ |
Empirical (x10
Estimation (x 10
Empirical
Estimation
Empirical
0.1
0.0143 + 0.0002
0.0163
0.558 0.005
0.628
3894 + 87
0.3
0.1280 + 0.0002
0.1465
0.561 + 0.006
0.628
3953 + 102
0.5
0.3695 + 0.0004
0.4069
0.556 + 0.006
0.628
3878 + 108
0.7
0.7008 + 0.0010
0.7975
0.564 + 0.005
0.628
3968 +
0.9
1.3085 + 0.0007
1.3184
1.822 0.245
0.628
3310 +
A.2. Numerical Experiments
Verification of the Single Data Point Assumption. Recall that Statements 1 and 2 assume that the dataset contains single data point. In this section, we conduct numerical experiments on CIFAR-10 dataset to demonstrate that this assumption provides a reasonable approximation of the variance of p(t|z) and the error between the effective targets in practice.
For both p(t|z) and Ez). we calculate their values by scanning the entire dataset as shown in the previous section, and compare them with our estimated theoretical values.
For the CIFAR-10 dataset, have N 50000 and d 3 x 32 3072, from which derive the estimated values of p(tlz) and (z)) in Table 4.
As for empirical calculation, we compute the desired values via Monte Carlo sampling. Specifically, we select 5 time levels 4+ 0.1, 0.3, 0.5, 0.7, 0.9. Here, Ey are independent samples from (0, La), and I, are independent random integers from [1, .V. We then compute Vare=p(t))]d] [+], [[2) and E(-) for each z, as we specified in Appendix A.1. Finally, we average the M values to obtain the empirical values along with their statistical uncertainties. Results are shown in Table 4.
0.9. For each la. we sample M = 25 noisy images by 2j = a((a)x1, + b(,)),,, = 1, 2, M.
Table 4 shows that our estimations closely align with the observed values, except when gets very close to I (i.e. in highly noisy images), where the single data point approximation becomes less precise. However, even in these cases, the estimated values remain within the same order of magnitude, providing acceptable explanations for the concentration of p(t|z) and the small error between the two effective targets.
Visualization of p(t]z). We plot the value of p(t]z) in Fig. for one and t. from 0.1 to 0.9. This is carried out exactly in the same manner the variance calculation for t, but with AFHQ-v2 dataset at 64 64 resolution for better visualization quality. Fig. 3 functions as reliable visual verification of the concentration of p( p(()z).
A.3. Evaluation of the Bound Values
In this section, we provide additional experiment details how compute the bound values and present the plot of the bound terms A, B, in Fig. 4. For reference, we also include separate plots for A; and B; in Figs, 9a and 9b 6,
Recall that in Statement 3, we define A; and B; as
N-1
A
II (Rs +n: B, 1nd 3-++1
(20)
"An interesting fact in Fig. 9b is that, for EDM, there is "phase change" at around 50, which is caused by non-smooth 8 value. We hypothesize that this transition occurs at noise level high enough that the data distribution can longer he approximated point distribution, leading to a noticeable shift in behavior.
15

---

Is Noise Conditioning Necessary for Denoising Generative Models?
10a
DDIM
10
DDIM
EDM
EDM
FM (L-RF)
(1-RF)
10
10"
VEDM
0 10
10
10
10
10a
10-6
0
20
40
60
100
20
40
60
80
100
(a) The values A, different denoising generative models.
(b) The values B, for different denoising generative models.
Figure 9: The bound applied DDIM, EDM, FM and uEDM model with first-order ODE sampling process of N 100 steps. The figures visualize the different terms A, and B, in the bound.
Since K; and 77 are already given by the configurations for each model (see Table 8), we only have evaluate L; and dG.
Statement 3 requires the following condition to hold:
[R(x() - R(x(14)) & d;
[R(x((t,) Rtx](t))l Li
(21)
-x
Now we are going to pick reasonable values of 6; and Li- As mentioned in Section 4.4, it is unrealistic for this assumption to hold exactly in real data due to bad-behaviors of the effective target It when the noisy image is close to pure noise or pure data. However, we aim make the conditions hold with high probability instead of considering worst case. As a result, our choice of 6, Ia corresponds high probability
Estimation of 6,. We estimate & using a maximum among different samples:
max R(zig)t) R(z,,))l
(22)
3
where we sample 10 different z from p(z[t). R(z)() and R(z) values are computed as specified in Appendix A.1. We use maximum value across different samples ensure that the condition holds with high probability.
Estimation of Lg. The condition of L; is similar to "Lipchitz constant" of R(-lts). Inspired by this, we evaluate the value of |R(x f.) R(x]tt)]l for x, and x that are close to each other.
x
x,||
To model this, we sample x, from p(z]t,), and let x = x; + òč. Here, we pick 5 0.01, and ė NN I) represents random direction, which serves as first-order estimation.
Based on this, sample 10 different pairs of and x for each t, and evaluate the max value of the "Lipchitz constant". In another word, we are calculating
L max R[zz, 82, R(zz,,|t.) 8||,,1
(23)
for j = 1, 2,..... 10. Again, here we use the maximum value across different samples to ensure that the condition holds with high probability.
16

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Bound Values of the uEDM Model. It is worth noticing that our uEDM model (see Section 5) has non-constant weighting te(t) * which doesn't tt match our assumption when deriving the effective target R(z]t) and R(z) (as did in Section 4.1). However, we choose not to introduce more mathematical complexities, and instead use the formulas above to calculate the bound value for uEDM. This implies that the bound for uEDM is no longer mathematically strict, but it can still serve as a reasonable intuition for the choice of our uFDM configuration.
Absolute Magnitude of the Bound Values. As shown in Fig. 4, one might observe that the magnitude of the bound (around 102 to 10%) is actually significantly larger compared to the typical magnitude of |xv] (which is around oav ~ 10*). We hypothesize that this is because of the following two reasons:
(1) We are assuming the error accumulating on each step, which might not be the case in practice, as studied our discussion of SDE samplers in Section 6.2.
(2) When the noisy image approaches clean data, some properties of the effective target R become bad (e.g. the Lipchitz constant L, will be very big). This leads to large error estimation, but in real cases, as the neural network will smooth the learned function, the error is typically smaller. If we consider ignoring the last 10 steps in our bound value, the bound will be in a reasonable range (approximately 10 for FM and uEDM, 140 for EDM, and > 10y for DDIM).
B. Additional Experimental Details
B.1. General Experiment Configurations
We implement our main code base using Google TPU and the JAX (Bradbury et al., 2018) platform, and run most of our experiments TPU v2 and v3 cores. As the official codes are mostly provided as GPU code, we re-implemented most of the previous work in JAX. For the faithfulness of our re-implementation, please refer to Appendix B.3.
FID Evaluation. For evaluation of the generative models, we calculate FID (Heusel et al., 2017) between 50,000 generated images and all available real images without any augmentation. We used the pre-trained Inception-v3 model provided by StyleGAN3 (Karras et al., 2021) and converted it into model class compatible with JAX. As we have reproduced most of the results in Appendix B.3, we believe that our FID calculation is reliable.
Noise Conditioning Removal. When refer to "removing noise conditioning". technically set the scalar before passing into the time-embedding to zero. Alternatively, we can also set the embedded time vector to zero. The results turn out to have negligible differences.
iDDPM (x- (x-pred). We design a x-prediction version of iDDPM to show the generalizability of our theoretical framework. During training time, we simply change the target r(x, e, t) or r(x, e) to be x. The sampling algorithm has to be modified accordingly, and we directly translate the x-prediction to 6-prediction by Eq. (1).
ADM. In the original work of ADM, Dhariwal & Nichol (2021) don't provide result on the CIFAR-10 dataset in class- unconditional settings. In our implementation of ADM, we keep the main method of learning e-prediction and the variance Σ simultaneously, but employ it on the class-nnconditional CIFAR-10 10 task. Notice that this ADM formulation is also not included in our theoretical framework, but still gives a reasonable result after removing noise conditioning (see Table 2).
Hyperparameters. A table of selected important hyperparameters can be found in Table 5. For ICM and ECM we use the RAdam (Liu et al., 2020) optimizer, while for all other models we use the Adam (Kingma & Ba, 2015) optimizer. Also, we set the parameter 32 to 0.95 to stabilize the training process.
For all CIFAR-10 experiments, we used the architecture of NCSN++ Song et al. (2021b), with 56M parameters. For the ImageNet 32x32 32 experiment, we used the same architecture but a larger scale, with total of 210M parameters. For experiments on classifier-free guidance on ImageNet 256, we strictly follow the SiT-B/2 model with 400k training steps in Ma et al. (2024).
We highlight that for all experiments, we only tune hyperparameters on the poise-conditionss model, and then directly use exactly the same hyperparameters for the coieeunnconiitioal model and don't 't perform any further hyperparameter tuning. Thus, we expect that tuning these hyperparameters may further improve the performance of the noise-une insinnnacniiii model.
17

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Caas-Conditional Generation on CIFAR-10. For the class-conditional CIFAR-10 experiments, we use exactly the same configurations and hyperparameters of EDM and FM with the unconditional generation case, except that train the network with labels. For the conditioning on labels, we use the architecture in Karras et al. (2022). We do not apply any kind of guidance at inference time.
FFHQ 64x 64 Experiments. For FFHQ 64x 64 experiments, we directly use the code provided by Karras et (2022) and run it on 8 H100 GPUs. We keep all hyperparameters the same as the original code in the experiments. For the removal of noise-conditioning, we simply set the Cnoise variable in the code to zero.
Table 5: Selected important hyperparameters in our main experiments.
Experiment
Duration
Warmup Epochs
Batch Size
Learning Rate
EMA Schedule
EMA Half-life Images
Dropout
iDDPM & ADM
100M
200
2048
4
(DDPMix--eod
200M
200
2048
1.2x10-3
EDM
50M
DDIM
100M
200
512
4x10
EDM
50M
FM
100M
200
2048
8x10
EDM
50M
0.15
FM ImageNet32
256M
64
2048
2 10
EDM
5OM
EDM
200M
200
512
1x10-8
EDM
0.5M
0.13
3EDM (ours)
200M
200
512
4x10-1
EDM
0.5M
0.2
ICM& & BCM
400M
0
1024
1x10 4
Cons((.99993)
256
1x10
Consi(O 9999)
B.2. Special Experiments
This section covers the specific experiment details in Section 6.2.
DDIM-iDDPM Interpolate Sampler. In the analysis of stochasticity, we examine VP diffusion models with cosine and linear a(t) schedule with and without noise conditioning, using customized interpolate sampler featured by A, which is given by Eq. (56). For the cosine schedule, we use 500 sampling steps, while for the linear schedule, we use 100 sampling steps. As discussed in Appendix D.2, when 1, the model with the cosine schedule has the same setting as "iDDPM" in Table when 0, the model with the linear schedule has the same setting as "DDIM ODE 100" Table
Results for the experiment are shown in Table 6, and the result for the cosine schedule model is visualized in Fig. 6. From Table 6 one also find consistent trend that À increase, the degradation of FID becomes smaller for the noise-unconditional model, regardless of the specific schedule of (t).
Alternative Architectures: Noise Level Predictor and Noise-like Condition. our experiment of Noise Level Predictor, we train very lightweight network to predict the noise level given the input z. To be specific, our predictor network only contains two convolutional layers with relu activation, followed by global average pooling layer and linear layer. The network has more than 30K parameters, so it hardly affects the expressiveness of the whole model.
It's worth noticing that directly predicting the input / is usually not desirable, as the value of t may have different ranges for different models. Instead, for each specific model we choose a customized target for the prediction.
Training objectives for different models are shown below (P& represents the predictor network):
FM: L Exx-pan P~ E..1. (Po(1 f)x + tc) - 2
VP models: L =
( tx+ Vie) - 1/²
1 1
EDM models: L
Pale)((((x+te)+t
1+1
Here, for EDM, we apply a transformation y 1+g. mapping the original noise level E (0, +00) EDM to [0, 1].
18

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Table 6: Comparison of inference performance between noise-conditional and noise-unconditional models. The left panel uses a cosine noise schedule, while the right panel uses a linear noise schedule. Both panels compare the performance of noise-conditional and noise-unconditional settings across different values of the ablation sampler coefficient X, ranging from
0.0 1.0.
cosine schedule
lincur schedule
FID
Change
FID
Change
0.0
2.9%
- 34.56
+31.58
3,99 40.90
+36.91
30.34
+27.74
0.4
2.52
21.33
+18.81
4.45
28.08
+23.63
2.59
12.47
9.88
4,95
18.32
+13.37
2.77
7.53
4.76
5.90
10.36
4.46
1.0
3.13 -
5.51
2.38
8.07
10.85
+ 2.78
Experimentally, the MSE loss for all these settings all have magnitude on the order of 10-4, which means that even our very lightweight predictor can predict the noise level with a mean error of about 0.01.
In our Noise-like Conditioning experiment, the same lightweight network as mentioned above is connected to a noise- conditional U-Net, as visualized in Fig. 7c. This joint training architecture is noise unconditional. The main difference between this experiment and the "Noise Level Predictor" experiment that there is supervision the intermediate output, it may not be the noise level itself.
For the experiments in Fig. 7 column (b) and (c), we again use the same set of hyperparameters and configurations as the noise-conditional and noise-unconditional experiments (in columns (a) and (d)), to ensure a fair comparison. However, a subtle detail is that in the implementation of iDDPM in Fig. 7, the results for noise-conditional and noise-unconditional
experiments are not the same as the results in Table 2. This is due to that we use T 500 instead of = 4000 for the training process.
B.3. Our Reimplementation Faithfulness
As we have mentioned, we reimplement most of the models in the platform of JAX and TPU. Table 7 shows the comparison of our reimplementation and the original reported results.
For some models, the reproduction doesn't 't meet expectations. For example, since the official code for iCT is currently not open-sourced, with the originally reported score of 2.46). For EDM on FFHQ-64, we directly run the given official code with the VP configuration on 8 H100 GPUs, but still can't reproduce the result. We suspect that the difference may come from random variance or different package versions used in the experiments.
only follow all configurations mentioned in their work, and get best FID score of 2.59 (compared
Note that iDDPM (Nichol & Dhariwal, 2021) only reports results with 1000 and 4000 training steps. Due to computational constraints, only evaluate the model with 500 sampling steps. Since 500-step result has already been better than the 1000-step result reported in their work, we believe that successfully reproduced the model.
After all, our goal is to compare the performance of noise-conditional and noise- coise-unconditional models, and the absolute performance is not the focus of our work. Thus, even though we haven't 't fully reproduced some of the results, we believe that our comparison is still meaningful.
C. Supplementary Theoretical Details
C.1. Proof of the Effective Target
Theorem 1. The original regression loss function with condition shown in Eq. (2) with w(t)
((0) Ex.e. (1NN -(x, e, t)
is equivalent to the loss function with the effective target shown in Eq. (5)
L' C'(0) (xx-p(x) [INNNoZ)t)) R(z]t))²]
19

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Table 7: Comparison of our reimplementation and the original reported results.
Model
Sampler
NFE
Reproduced
Original
CIFAR-10
500
3.13
iDDPM (Nichol Dhariwal, 2021)
1000
3.29
4000
2,90
ODE
100
3.99
4.16
(Song
ODE
1000
2.85
4.04
EDM (Karras et al., 2022)
Houn
35
1.99
1.97
I-RF (Liu et al., 2023)
RK45
~ 127
2.53
2.58
iCT (Song Dhariwal, 2024)
2.59
2.46
CIFAR-10 conditonni
EDM al.,
ImageNet x
(Lápman 2023) FM (Lápman al., 2023)
RK45
~ -125
4.30
5.02
FFHQ 64
HDM (Karas al., 2022)
Heun
2.64
2.39
only up to a constant term that is independent of 0, where
R(z[t) -xx.e)
Here, p(z) is the marginalized distribution of ===a(t) + b(t)e in Eq. (1), under the joint distribution p(x, e,t) = p(x)p(t)p(t).
Proof. The original regression loss function with i condition can be rewritten as
(())= Ex--p(x) (x.c)) |INNo r(x, E, e>|
Ex-p(x)
[NNo( (xxe)--oxxeln.o)(((xx... e V (x
Ex-p(x),t [[NNo(z|t) R(z|t)]|² const
(xxp(x))(=p(t)) [N]o) - R(Z())) + const ('( const.
(24)
This finishes the proof.
Theorem 2. The original regression loss function without noise conditioning
(()) = Ex.e, |NNo(()N r(x, e, 1)]
is equivalent to the loss function with the effective target shown in Eq. (7)
(') (Ezpp() [[NNO(2) R(z) [?]
only up to a constant term that is independent of 0, where
R(z) Ex-p(t R(z]t)].
Defintions on p(z) and p(t|z) z) are the same as Theorem
20

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Proof. The original regression loss function without noise conditioning be rewritten
[()) B(zp(() (xxe))))(xxxe)))p( [NNNO()) r(x,,,
Ex-p(a) [NNo(z) - Bxxxat))
Pz=p(e), (INNo(Z) B(xe.t))xx O]| const
Bu-p(x)) [INNo (z) B(x..))) -xx.a.)) t) |]] + const
(25)
And notice that
E [xx,,.)) (txc///-ptxcc//////xx.. Be (Tx,,, = Ex-p((e) R(z)t).
(26)
So
())) = Ex (x))(())
[NNo(Z) - Bixcct)-otxcca///xxxx.c + const
( [NNo(z) - R(z) 2 const = (0) + const.
This finishes the proof.
C.2. Approximation of the Variance of p(t|z)
We claim Statement that, for fixed noisy image whose true noise level t4, the posterior variance of p(t|z) scales like / 2d. In this section, we first derive ( /d) upper bound on the variance under minimal technical assumptions. While obtaining the exact constant requires delicate optimizations, then present concise, intuitive argument recover the & 2d scaling, which confirmed by our numerical results in Appendix A.2 serves as an accurate and practical estimate of Varp(t/a)[it..
Big-O presentation keeps the proof accessible. We
C.2.1. RIGOROUS UPPER BOUND ON VARIANCE.
Single Data Point Case, We first consider the case where N 1. For brevity, write x = X1. Recall that our goal is to bound, with high probability over the randomness
p(2), the posterior variance Varrjz[1].
Note that some ill-behaved can lead strange distribution of f. Thus we work under the high-probability regime in which concentrates around its typical behavior.
Because AO, Ia) is rotationally invariant, we may, without loss of generality, assume x to only have 1 nonzero coordinate.
Theorem 3. Assume that
x = (x. 0,,,0,,.... ,0,,, -vd, z = (1- (1- toE, e = 6')
Here, Eo is scalar that is the first coordinate of e, and é' is its other coordinates.
We further assume that
1 1 3
to
||E | -d O(Vdlog d), lcol < log d
(27)
d'
Then, we have (tr,jz(t)= (tt//d).
Note that Eq. (27) oceur with probability O(2), due to the norm-concemtraaion properties of the Gaussian distribution.
Proof. We directly compute the probability density functions follows.
21

---

to Is Noise Conditioning Necessary for Denoising Generative Models?
Firstly, we have
1/1 z (1 - t)x 2)
p(1z) p(z)()(() x (d exp
since the distribution p(z]t) is simply a Gaussian with mean (1 t x and variance t? per dimension. Looking into the exponent, it can be simplified as
z (1 t)x
(t to)x + toc
=
(1-
to
x + ty
2
t
2
to
to
to
x
+
E0
+
|6'12
1
to
2
which is quadratic function on
We write it as a
p
+ C for some constants a, μr, C independent of t, where
= (z- -o) c0)² | = -²- x0 (Vooog + d O(Vdlog d)
+2 (VVlogd)
(28)
and
au 5(x 60) x² 10(s//ogg
(29)
by the assumption Eq. (27). Substituting back, we get
( 1 to - 11 ) 2
1
p(t|z) xc tu oxp
We find its maximum by differentiating its log-density:
a log p (l/z)
d
40 -
to
10
2
+ a
a
ap
ōt t
t2
t
t
Let A1 > 0, A2 < be the two roots of the equation f(X) = ax² aux - (since and are both positive).
d
Notice that 2ιd2
We claim that
log vd
d2 - 1 1+00 (log d' )) (1))
_
A1 1
(30)
This is because
f(1) = OlValog d), f(11 -1)=222 2d + O(VolooVg d)
by Eq. (28) and Eq. (30), from which we derive the desired root bounds.
Using the factorization into roots, have
à log p(tlz)
to
2
ajl to
to
to
Ōt
a
t
d
=
t
A
X2
to
A1 9(d)
by Eq. (28) and Eq. (30).
22

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Denote t* to A
as the unique maximizer of lo)z() As first step, we show that in range (0, 3/ the probability
density of p(t|z) is more concentrated than a Gaussian with variance (t d). Intuitively, it is due to the fact that in this range,
p(tzz)
d2 7 (
1
to ) ((d) (z
to(t t) 2(d) (1* 1)0
(3)
d
((d)
121$
utilizing Eq. (30). This inequality implies that in this range, the gradient of this probability density sharper than N(1,, O(1//d))
To be specific, when 0 < t <t<t & to or 1* <t<ta
<t< 3t we have
logp(1'|z) logp((1z) = llgg((tiz)
- (C'
d (g)) 10/
di = o
(31)
12
where the RHS is exactly the difference in log proopabbbbyi density of N(t", ,((g//)) at t and t'. This fact supports that in this range, p(t|z) is sharper than Nitt, ( or
Ex-r((t- t+] & BI= ++) = O(t
where p', q denotes the distribution of p(t|z) and Ntt Olt)/d) d)) restricted on (0, 3f respectively. Finally, we only need to consider the part when 31* < l & 1, where we are going to prove that
Pr [t 3t*1 (<p(1/2)
is small. In this case, according to Eq. (31), for any 1 e [t. 21*]:
p(t|z) p((lz)
< exp (-s 1g
(4072-
(41#2 - -t3)
=exp(-
which is actually exponentially small. This implies that
=0 001
Pr [t> 3t < exp(- (<p((2)
min p(t'lz) < <exp(- ** eeee* 20
d3
Consequently, the contribution of this tail to the variance is O(1/d²), which is negligible compared to (())/)) Combining the two regimes, we conclude that
Varp(e/a][t] & (t- O(t/d)
Multiple Data Points Case. We now turn to the setting of N data points. Intuitively, one can identify the ground-truie truth clean image that generates the given noisy image, as long as the noise is not too large to swamp all signal. This can be done simply by comparing inner product: the noisy observation z will correlate most strongly with its corresponding clean sample X,, and only weakly with all others.
Lemma 1. Suppose that X1, X2,..., are i.i.d Gaussian samples from (0, Ia), and (1 (o)x; + lefor some i.
Then, we have the following two properties hold with probability -
(x,, z) (Xc, 2) - -(1 -,0)1x191
= =Olvd(log N + d)),
Vj ii
OlVilogg)
23

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Proof. We first deal with the second inequality. We have
X 2 (1 x (x,, (1 to)x; + -1- to (x,,, E) NoO, (x
Using standard Gaussian tail bounds, this probability that (xx, - -0)) ky/ is at most
k2d kodlogg dl
exp(-k² log d) d -((#))
exp
x
Therefore, there exists a constant such that the probability above is at most dN
For the first inequality, notice that
(x,, 2) (1 - (x,, Xj) (x),,)
each term is the dot product two normal Gaussian variables, which still sub-exponential.
In this way, we derive that each term at most O(d(log N + loo d)) with probability gives the desired result.
dN so union bound over all the i
log +
log
Theorem 4. There exists constant C' such that when 1 to CV
we can recover the correct with
d
probability at least d'
Proof. Using Lemma 1, we can compute the dot products of all x, with and take the largest one.
This theorem implies that the concentration property of p(t|z) be extended to multiple data points case, when there are exponentially-many data points. By first recovering the correct clean image, the multi-data-pont toiitt setting reduces immediately to the single-point analysis, and hence the posterior variance bound (1 /d) extends to the case of exponentially many candidates.
C.2.2. INTUITIVE DERIVATION OF THE APPROXIMATE CONSTANT
In this section, we show the reason why we estimate the variance as t 2d.
Firstly, have in Theorem 4, the ground-truth clean image be recovered with high confidence, with prior of exponentially-many data points. As result, we revert to the single data point setting.
For simplicity, we restrict attention to the bulk regime t E d d postponing edge-case analysis to future work.
Now, we can still view the data point x as (x, ..... 0) due to symmetry, where x stands for the norm of x. Because the dominant contribution to the likelihood comes from the d noise dimensions when d is large, further approximate by neglecting the signal component in the first coordinate (i.e., assume x 0).
On this assumption, we have
z' = (1 1)x + te' = tc.
and denote |2] = to d. We have
(-
dig
p(t|2) x ta exp
2t
The variance of this distribution can calculated exactly from integration, as long as we extend the distribution from [0, 1] the whole R:
24

---

Is Noise Conditioning Necessary for Denoising Generative Models?
diu
2
d
-
ld-1
exp
212
dt
2
d
T
2
Bp(t)a)
[t]
40
00
td
exp
2f
dtg
di
d-1
d
1
2
r
d -
2
1
and similarly
dig
Ep(tp)](1
=
d=11
d
d d-3
which yields
dig.
2
T( T( r( r(
[arp(e/a][]]=
(32)
2 d-3
Again, here we extend the distribution of p(1]z) from [0, 1] to IR, as this relaxation will only increase the variance.
Finally, we use the Stirling's expansion for the gamma function to get
(1 1 )()) dt) - d-1
+
3
)
1+
+
(33)
r (
(d21) d-2 (1
12 + +
4(d 1)
Plugging Eq. (33) into Eq. (32), we derive our final estimation:
dt
2
2
(
)
Var [t =
11
+
2
d
- 3
d-1
2(d - 1)
+ 1
)
d
2
d
which aligns with our Statement 1.
C.3. Approximation of E(z)
Statement 1 (Error of effective regression targets). Consider single datapoint E [-1, e NOO, t- u 1]. and z = ()x + te (as in Flow Matching). Define R(z) and R(z]t) with the Flow Matching configuration in Table 8. Given a noisy image z (1 - 1x)x + L, produced by given tu the mean squared error E(z) in Eq. (10) can be approximated by
E(z)
1 /
63) o3)
(34)
under the situation that the data dimension d satisfies 1/2 < to and << 1 - ts. Here, oî denotes the mean of squared pixel values of the dataset.
Derivation. We start by the definition of E(2):
Elz) E, [()]))))) R(z]t) R(z)]
Rxp((() R(z]t) [R
(35)
Next, compute t) using its definition under the Flow Matching configuration:
R(z(t) := ((x.e)--nxx.ea/0 e, t) E(x,)) Exx 2 x
(36)
tt
25

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Using Eq. (36), we obtain
F((z) -| - x|2 Varc=p(t(z)) i
(37)
We now compute the two terms separately. For the first term, we can rewrite as
x|| Ea x e=ll
ta (||x||² + ||E=ll²)
= t (do3 + d) = (a on).
(38)
Here, employ the fact that x Es |x|| Ex|l, and that ||+l| vd with high probability. Also, oj |xx|//d since assume that the dataset contains only single data point.
For the second term, note that the variance of p(42), given in Statement 1, is significantly smaller than the concentrated mean & of p(t(z). Thus, we approximate the variance using a first-order expansion:
(x)
1
Varz=p(tlx)
Varz=p(t(e)
Vart-p(da)[(]] = t2d
(39)
th
t
Combining Eqs. (38) and (39), we get the estimation in Eq. (34).
C.4. Bound of Accumulated Error
Statement 2 (Bound of accumulated error). Starting from the same noise X/.- consider sampling process (Eq. (4)) of N steps, with noise conditioning:
X;+1 = K;X; + +.[(x(())) CiE;
and without noise conditioning:
= + R(x)) G.C.
[[[((())) R(x,(+))\/ xill Li and - & 8, it can be shown that the error between the sampler outputs XN and x is bounded:
IxN xxy-xyl- Bo + A Bi +... A BN-1,
(40)
where
N-1
A; = II 1-i+1
(ns In: B, 1n:
Proof.
Define a; K4 In L and b; = |7, Then, have:
|x41 x ke x;) (R(x() - R(x;(4))
(41)
as we assume that the same noise é; is added in the sampling process with and without noise conditioning.
Using the triangle inequality, this can be bounded as:
|x++ x++1l & x x; + 1xd R(x) R(x((t)) + R((())) R(x|(t))l < axlix x; b..
(42)
We now use induction on to establish the bound:
Ix,,
x,1
x
ak
bj:
(43)
1=0
k=
26

---

Is Noise Conditioning Necessary for Denoising Generative Models?
where [ ag is defined for j N
For the base case 1, need to show:
||x] x11 < bu.
(44)
which follows directly from Eq. (42) with
Now, assume the bound holds for some n, i.e.
n
Xx" - -xxl E
a
bj
I ax
|
xol.
(45)
We prove it holds for n + 1. Applying Eq. (42), we obtain:
| - Xn+ill E an xxxli, xn + ba.
(46)
Substitute the inductive hypothesis for Xx1- -xxll:
I
|xx+1 Xn+1l & an Σ
I k-j+1
)
Ok
by
bn = Σ
π
by-
(47)
j-0
3-0
k-j+1
Thus, the bound holds for + 1. By induction, the bound holds for all n. Taking
N yields the desired result.
D. Derivation of Coefficients for Different Denoising Generative Models
In this section, we build upon our formulation in Section 3.1 express common diffusion models iDDPM (Nichol Dhariwal, 2021), DDIM (Song et al., 2021a), EDM (Karras al., 2022), Flow Matching (FM) (Lipman et al., 2023; Liu al., 2023), and uEDM Model-usioe unified notation. The coefficients corresponding each model are summarized in Table 8 and Table 9, followed by concise derivation of their formulations.
D.1. iDDPM
The loss function of iDDMM (Nichol & Dhariwal, 2021) in DDPM (Ho et al., 2020)'s notation is
[||c - colv + v1 - ā,c, a,, f)
simple = -E,,xo.e
This can be directly translated into our notation:
((0) Ex,e,t (2())) r(x,
where we have the coefficients
a(t) = o =
(48)
and with the training weighting and distribution of being
w(t) 1, and p(t) = ((1,...,
(49)
Notice the presence of the diffusion schedule a(l) inside the coefficients. We adapt modified version of the cosine schedule in Nichol Dhariwal (2021):
a(t) ((1 cos
ml. mt
(50)
T
27

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Table 8: The coefficients of different models. For iDDPM, we assume a cosine diffusion schedule a(t). For both iDDPM and DDIM we follow the original notation of DDPM (Ho et al., 2020). Also note that for EDM, coefficients are calculated according to first-order ODE solver, and in the final step we need multiply the output by da to get the final image. See Appendix D for more details and derivations.
iDDPM
DDIM
EDM
FM
Training
a())
Va(1)
a(L)
b(t)
V1
a(t)
a(t)
V=++oj
c(t)
0
-1
d(t)
1
de
1
((t)
Pt
u(I....,77
(11....; T)
exp.N(11.2,1.23)
u(00 1]
Sampling
6141
(1- 444444444
Ki
19-8
Tjs
vI=āi
0i+1
1:1l (1 (e)
1 +
mall:
l+1 t
C:
(1
1 85
1-a:-1
0
1
Schedule to-N
t
N-1 .T
N
t =
N-t T N
t =
Umix
N
min
thax
t =
Parameters
a(t) 1 (1 + cos TT)
a(t) = 17
[-]
(1 (1 k2 k2r )
σa = 0.5, p =
[-0
a = a(4))
a == a(4))
bmax 80, Amin = 0.002
T 4000
T 1000
h = 10
12 = 2 x 10
where T' 4000 is the total number of diffusion steps during training.
Next, consider the sampling process, which their notations is iteratively given by
1
co(x(:t)
1 Ōt-1 3iz,
X1-1
Xt
V 1 1- āg
Var
VT at.
and z ~ (O, a standard Gaussian random noise. is also straightforward to translate this sampling equation into our notation:
11+11
1
(V
a;
ai+1
1/(111
ā; ai+1
N4
α4
7)
VI-aa a.
(1 Va++1
ā;
.5s
ā 1 - a.
(51)
and
N i T.
(52)
This will give the first column in Table 8.
28

---

Is Noise Conditioning Necessary for Denoising Generative Models?
D.2. DDIM
DDIM (Song al., 2021a) shares the training process with DDPM (Ho et al., 2020). However, we choose use the linear schedule for @tt), to demonstrate the generality of our scheme. This schedule has the form
a(1) - I
kn
k/2 -
(53)
1-0
where b 10 and k2 10-2, and T 1000 is the total number of diffusion steps during training.
The sampling process is given by
X v Vdr-1
It
vI (x,)(
+ v -1 ac-
v 0q
which is obtained by substituting d; 0 in their notation. This is again straightforward to translate into notation:
Oi+1
V 0s+1
04+1 (1 -ā;), C: - -0
(54)
0
a;
and
N i N
4
(55)
These give the second column in Table 8.
Moreover, we can consider the generalized sampler proposed by Song et al. (2021a), which contains an adjustable parameter A E [0, 1]. In their original notation, the sampler can be written as
11- - (Pofe((x,,)) + doice Ft VI (ico(x).t)) + v1 11- āc-
X1-1 = V
Vā,
where
(ā - 01) δ(-1) أا-1 āt)
0,
and C; is an independent Gaussian random noise. In our formulation, it can be equivalently written
01+1
1: 1 - 5:41 12 (āi+1 āz)(1 ox+1) ai+1 āz)
10++1
(1 ā.))
(56)
āi
(ōit1 ā1)( x1+1)
G= \
(1+1(1- 1-)ss) ā,))
These expressions are used in our experiment of the "interpolate sampler" in Section 6.2. One can verify that when À = 1, the coefficients K;, Yi and & will be the same as iDDPM (Eq. (51)); and when DDIM (Eq. (54)).
0, the coefficients will be the same as
'Here, we use the notation Exp.N(,, o²) to denote distribution of exp(u), where u N(p,a²).
29

---

Is Noise Conditioning Necessary for Denoising Generative Models?
D.3. EDM
The original EDM (Karras al., 2022) training objective is given by
(()) Ex,c,t A(t)) Do(x + - - x
(57)
where Dg formed by the raw network NNg wrapped with a precondition:
Do(aDlt) Cakip (t)zp Cou( (fn
where zD x + tc. Here we directly use t instead of Croise (t) in the original notation.
As mentioned in Section 3.1, we will consider the regression target with respect to NNg instead of Dg and absorb the coefficients (t), Gi (t) and Cour(t) into the training process. To achieve that, we define z == Cin Cin(t)(x (x + te). Then, we can get an equivalent training objective for NNo given by
((0) Ex Ex,c, ((()))NN(2))) - r(x, e, 0)]3
where
= cn(()x + (f) w(t) x(t) (1)2
(58)
r(x, E, canlt) (x (skip(1) [E))
1-oan(()
Ieaip (t) Cuee (2)
Now, we can plug the specific expressions
aal Vog + 12
0
od + 12 ogt²
Gm and σa 0.5 to get the coefficients
+ Coul
Caip(t)
of + 22 )(t)(())))))
a(t) Vo++ +2 b(t)=
Vo +e ct)
d(t)
Od V0 12
(59)
a.vo +
and w(t 1. Also note that p(t) is given explicitly by the log-norm schedule <xp.N11122 1.2?). discussion of the training process.
This completes the
The (first-order) sampling process is given by
XDD + (1++1 11)) XD,i (Catip(1:) Xp, + Cour (oun(4)NNo(( (4))x.;
XD
t
Here the suffix D to denote this is the sampling process corresponding to Dø. Since also have to remove the external conditioning in the sampling process, we should let x; = DD; and rewrite the sampling equation using x;
t++
cin(t++1)
11
t++1 - to caxip x; + ti - t++1 t++1
X
Cour (t:) (oau(t:))) (t:+))
t
Cin(t,)
t;
This then gives the general sampling coefficients
t++1
(im(t++1)
(1
ti+1 - to
to - t++1
Cakip (1i) .7h =
Coul (la4))) Si O.
(60)
Cin(1.)
li+1
Then, we can plug in the explicit expressions of Cia(ts), (ts) and Cour Cour(t)) to get the final coefficients
o + E
1+
4((1+11 -44)
oa aa(li+1 4:) G
K;
(61)
+ 12-
+
oz
1
+ +z)) + xz))
30

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Moreover, notice that due our change-of-variable during the removal of external conditioning, XN is defined (n(tx) DxD But the sampling algorithm ensures xD,N match the data distribution, instead of xx- Thus, have to multiply the output by aa get the final image, mentioned in the caption of Table 8.
Finally, the sampling time step is also explicitly given in Karras et al. (2022), so we can directly use it here:
sot (N-iNN----) 1)+0.00277
if < N
4 =
N
(62)
ifi= N
These together give the coefficients for EDM, which are shown in the third column of Table
D.4. Flow Matching
The training process of FM (Lipman et al., 2023) given by
(()) = Ex.e. |Ivo + (1-t)x, 1) - (E - x)]33
This can be directly translated into our notation:
a(t) = - t, -1, 1,
(63)
and with w(() and p(1) = 2([0, IJ). The sampling process is given by solving the ODE
dx dt = vo(x,t) t)
from t = to = O. Since we assume using a first-order method (i.e. Euler method), the sampling equation is
X x; + vo(xi, (11+1 4.).
This will give
R; 0, 4:+1
Lis Ci
(64)
well the sampling time
N t = N
(65)
as in the fourth column of Table 8.
D.5. Our uEDM Model in the Formulation
Introduced in Section 5, the uEDM model designed by us is a modified version of EDM (Karras et al., 2022). The only modification that we change Cia (4) and Cour()) by
1
(nn(t) (t)
112
Gin V12
toa
Cour
Cou((t) (t) =
1/2 - o
and remain all other configurations the same as the original EDM model.
In Appendix D.3, we have already derived the general form of the coefficients of EDM with functions en (t), Comr Comr(t), (t), Cakip (t) and X(t) in Eqs. (58) and (60). Plugging in the new set of these functions, we can then derive the coefficients uEDM, as shown in Table 9.
31

---

Is Noise Conditioning Necessary for Denoising Generative Models?
Table 9; Comparison of coefficients of EDM and our uEDM.
Coefficients
a(t)
b(0)
<(()
d(t)
(+))
aalt k++1)
EDM
v a
v
oay 12
0a
1441
1f + oa
V of) + ag)
to
dd +
f 1:(1:44+11 P4n +1
4--4+1 " P4 +
UEDM
V82 +1
v/2 +1
e + 0.
oat
+ od
E. Additional Samples
Beyond the comparison shown in Fig. 5 for noise-conditional and noise-unconditionl models, we also provide additional samples for other models, on other datasets, or in class-condee conditioonin settings. We use the same configuration as Table Figs. 10 and 11 show the samples of ICM and ECM on CIFAR-10 10 with both and 2 inference steps. Fig. 12 show the samples of FM on ImageNet 32x with both Euler and EDM-Heun sampler. Fig. 13 shows the samples of FM and EDM on CIFAR-10 in class-conditional setting.
32

---

Is Noise Conditioning Necessary for Denoising Generative Models?
(a) ICM 1 step (FID: 3.37 12.03)
(b) ICM 2 step (FID: 2.59 - 3.57)
Figure 10: Samples generated ICM CIFAR-10. From left right: step w/ t, step w/o t, step w/ t, 2 step w/o t. All corresponding samples use the same noise.
(a) ECM step (FID: 3.49 - 12.60)
(b) ECM step (FID: 2.57 - 3:27)
Figure 11: Samples generated by ECM on CIFAR-10. From left to right: step w/ f, 1 step w/o f, 2 step w/ t, 2 step w/o t. All corresponding samples use the same noise.
(a) ImageNet FM, Euler Sampler (FID: 5.15 4.85)
(b) ImageNet FM, Heun Sampler (FID: 4.43 4.58)
Figure 12: Samples generated by FM on ImageNet 32x32 with Fuler and EDM- sampler. From left to right: Fuler w/ Euler w/o Heun w/t t, Heun w/o All corresponding samples use the same noise.
33

---

Is Noise Conditioning Necessary for Denoising Generative Models?
(a) Class-conditional FM (FID: 2.72 - 2.55)
(b) Class-conditional EDM (FID: 1.76 3.11)
Figure 13:
Class-conditional samples generated by FM and EDM on CIFAR-1O. In rasterized order: FM w/ t, FM w/o t, EDM w/ t, EDM w/o t. All corresponding samples use the same noise and the same label.
34


Reviews:
[{"review_id": "68jRHPKSbs", "reviewer": "Reviewer_iMKL", "summary": "This paper analyzes noise conditional diffusion models (DMs) and develops theory supporting the viability of noise unconditional DMs. Empirical evidence supports the author's theoretical claims and demonstrates that noise unconditional DMs are capable of performance similar to noise conditional DMs. This challenges pre-existing notions that noise conditioning is fundamentally necessary for DMs to function.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful review and positive feedback regarding our work’s novelty and impact!\n\nWe agree that incorporating the suggested reference *Noise2Noise: Learning image restoration without clean data* will provide valuable context for the derivation in section 4.1. We will revise the manuscript to include this reference.", "claims_and_evidence": "Yes, all claims in the paper are well supported both by theoretical proofs and empirical studies.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I have reviewed the proofs / theoretical claims in the main paper and they appear sound to me.", "experimental_designs_or_analyses": "Yes, the empirical studies appear valid to me.", "relation_to_broader_scientific_literature": "This paper challenges the existing notion that noise conditioning is fundamental to the high performance of DMs.", "essential_references_not_discussed": "The finding in Section 4.1 (i.e., that the expectation over multiple realizations of z is an effective target) is very reminiscent of Noise2Noise [1]. Including a reference to [1] would provide additional context around using the expectation over multiple noisy realizations as a target.\n\n**References**\n\n1. Lehtinen, Jaakko, et al. \"Noise2Noise: Learning image restoration without clean data.\" arXiv preprint arXiv:1803.04189 (2018).", "other_strengths_and_weaknesses": "**Strengths**\n1. Novelty. I find the author's idea to use multiple realizations of z as an effective target to train an unconditional DM to be novel.\n2. The theory and proofs explore the viability of noise unconditional DMs, as well as their error bounds\n3. Impact. The theory is well supported by empirical evidence and unconditional DMs (somewhat surprisingly) perform similarly to noise conditional DMs. This is an important finding for the community because it challenges pre-existing notions about DMs.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "douRDvj9vU", "reviewer": "Reviewer_gvd2", "summary": "The paper tries to debunk a common belief among diffusion model practitioners if a time-condition of the model is necessary for a diffusion model. The authors take both the theoretical and experimental approach to address this issue. The paper mainly focus on the theoretical reasoning rather than practice, which is only demonstrated by CIFAR-10 generation problem, although various samplers are chosen for demonstration.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the thoughtful feedback and recognition of our theoretical contributions. Below, we address the concerns regarding experimental scope and practical applicability to large-scale diffusion models.\n\n**1. Large-scale experiment**\n\nTo address reviewer gvd2’s concerns on experimental scale, we conducted additional large-scale experiments with DiT + FM (i.e. SiT) [1, 2] on ImageNet 256x256 dataset. All experiments use the same configuration as the original paper, using Euler sampler with 250 steps. The results demonstrate that our findings extend successfully to larger scales:\n\n| Model: DiT-B/2 + FM |  |  |\n| --- | --- | --- |\n| CFG Scale | FID w/ t | FID w/o t |\n| 2.0 | *9.36* | 10.66 |\n| 2.5 | _**8.03**_ | 8.15 |\n| 2.7 | 8.24 | _**7.96**_ |\n| 3.0 | 8.88 | _8.15_ |\n| 3.5 | 10.29 | _9.09_ |\n| 4.0 | 11.81 | _10.28_ |\n\nNotably, removing noise conditioning improves performance at optimal CFG scales. Also, for many different CFG scales, removing noise conditioning improves performance. We can see that in the setting of FM on large-scale dataset, the behavior of removing noise conditioning remains consistent with experiments in our paper. \n\n**2. Performance on Other Types of Samplers**\n\n**Reviewer gvd2:** `“It would be nice to include DPM-Solver, DPM-Solver++-type samplers as well, since these are still well-used samplers in practice, and also these samplers use multiple tabbed filtering. The performance for these type of samplers might be closely related to DDIM's failure as well.”`\n\nThis is a very interesting extension to our current work, generalizing our theory and experiment to a wider range of commonly used sampling methods.\n\nRegarding the DPM solver equation:\n\n$$\nx_{t_i}=\\frac{\\alpha({t_i})}{\\alpha(t_{i-1})}x_{t_{i-1}}-\\sigma({t_i})(e^{h_i}-1)\\epsilon_{\\theta}(x_{t_{i-1}},t_{i-1})\n$$\n\n(Eq. 3.7 in [3]), which, in our notation (Eq. 4 in our paper), becomes\n\n$$\n\\kappa_i = \\frac{\\alpha_{i+1}}{\\alpha_{i}},\\quad \\eta_i = -\\sigma_{i+1}(e^{h_{i+1}}-1),\\quad \\zeta_i=0\n$$\n\nThis $\\kappa_i$ for DPM solver behaves very similar with DDIM (which has $\\kappa_i=\\sqrt{\\frac{\\alpha_{i+1}}{\\alpha_i}}$), which leads to large error bound value (the term $\\prod \\kappa_i$ dominates the error bound, and  $\\prod \\kappa_i$ approaches infinity due to division by 0). Thus, we can **expect** a bad performance similar with DDIM in the noise-unconditional scenario by our theory.\n\nTo further confirm this intuition, we perform numerical calculations of bound values as well as FID evaluations for both DPM-Solver and DPM-Solver++ (which has a similar formulation as DPM-Solver but uses $x$-prediction). The bound value of both of them are on the order of $10^6$ (which is of the same order with DDIM), and using DPM solver fails to generate good images (FID $>50$).\n\nThese results shows that our theoretical error bound (in Section 4.4) can qualitatively predict the performance accurately, demonstrating that our theory generalizes on even these carefully designed samplers.\n\n**3. Other Suggestions**\n\nWe appreciate the feedback regarding figure clarity and will incorporate the suggestions to enhance the presentation of our results.\n\n**References**\n\n1. Peebles, William, and Saining Xie. \"Scalable diffusion models with transformers.\" *Proceedings of the IEEE/CVF international conference on computer vision*. 2023.\n2. Ma, Nanye, et al. \"Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers.\" *European Conference on Computer Vision*. Cham: Springer Nature Switzerland, 2024.\n3. Lu, Cheng, et al. \"Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.\" *Advances in Neural Information Processing Systems* 35 (2022): 5775-5787.\n4. Lu, Cheng, et al. \"Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.\" *arXiv preprint arXiv:2211.01095* (2022).", "claims_and_evidence": "I believe that the paper’s main questioning on the necessity of the time-conditioning is valid and might open a new area of research that will benefit generative model communities. I am content with the theoretical justification of the paper and the choice of various sampling mechanisms. However, I am highly suspicious that CIFAR-10 generation task and commercial large-scale latent diffusion models do not share the same level of difficulty, and extending the “success” in CIFAR-10 domain to the large scale LDM should be done in extreme care. I still believe that the paper’s main question is well addressed. However, unless we have an actual experimental result for large scale commercial diffusion models, the practical value of this claim is not strong enough. I do not oppose to rejecting this so far, and I want to hear about other’s opinions regarding this including the authors’.", "methods_and_evaluation_criteria": "As I have mentioned in the previous section, the claim for diffusion model should be done in extreme care and should be equipped with practical large scale latent diffusion models of any kind (e.g., Stable Diffusion, or other image/audio/text diffusion models). Unless any one of these experiments are conducted I cannot give higher score than 3. I do not think we need every single large scale experiments but I only think that CIFAR-10 is not enough, regarding the small size of domain, the simplicity of the manifold, and the extreme sparsity of the dataset compared to the large scale data such as LAION-5B.", "theoretical_claims": "I have found no error in the theoretical claims. However, I would like to humbly admit that I may have missed some errors, though.", "experimental_designs_or_analyses": "It is a redundant statement, but I am consent with the discussion of various different samplers. However, I am not very well consent about the domain of generation.", "relation_to_broader_scientific_literature": "The problem tackles the necessity of noise-level condition for multi-step denoising frameworks. If the paper’s claim is valid in general denoising problems, this can be extended to deep learning-based solutions for ill-posed problem in general.", "essential_references_not_discussed": "It would be nice to include DPM-Solver, DPM-Solver++-type samplers as well, since these are still well-used samplers in practice, and also these samplers use multiple tabbed filtering. The performance for these type of samplers might be closely related to DDIM's failure as well.", "other_strengths_and_weaknesses": "Overall, the paper gives an important question on the necessity of the time condition dependency of the model. This claim reveals that the architecture of current diffusion models may only be suboptimal.\n\nI am also happy that the paper is equipped with rich theoretical justification and experimental justification with various types of samplers.\n\nRegarding weakness claims, please refer to other sections of this review.", "other_comments_or_suggestions": "- The figures seems not to address the main claim of the paper well enough. For example, in Figure 1, I have looked into multiple times to realize that the core idea is that diffusion models may just work well with t removed. This type of presentation issues exist in every figures in the manuscript and I recommend to update those with better clarity.", "questions_for_authors": "- Is there any practical problem to adopt this method in large scale diffusion models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "flN8Gandry", "reviewer": "Reviewer_7y3v", "summary": "This paper investigates whether diffusion models, which are typically noise-conditioning networks, can be converted into noise-unconditional networks. It finds that many models are not significantly affected by the removal of noise conditioning, and in the case of Rectified Flow (RF) models, performance even improves. The paper provides a theoretical analysis suggesting that removing noise conditioning does not introduce significant errors. Additionally, it proposes a noise-unconditional variant of EDM that maintains FID performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank for the constructive feedback and supportive comments. Here, we address the concerns regarding classifier-free guidance (CFG), experimental random variance, model-wise analysis, and explanation on performance of 1-RF.\n\n**1. Latent Diffusion (DiT) & Classifier-free Guidance**\n\nReviewer 7y3v commented, `One issue is that diffusion sampling often involves classifier-free guidance, which introduces instability through extrapolation in `$x_t$`. Noise-unconditioning may exacerbate these instabilities`.\n\nThis comment on the risk of unstability of generation with classifier-free guidance is very valuable. Also, we believe that incorporating larger-scale experiments such as Latent Diffusion (DiT) will definitely make our work more solid. \n\nTo address the concern, we conducted experiments DiT + FM (i.e. SiT) on the ImageNet 256x256 dataset with CFG. Here, all experiments use the same configuration as the original paper, using Euler sampler with 250 steps. The results demonstrate that our findings extend successfully to larger scales:\n\n| Model: DiT-B/2 + FM |  |  |\n| --- | --- | --- |\n| CFG Scale | FID w/ t | FID w/o t |\n| 2.0 | *9.36* | 10.66 |\n| 2.5 | _**8.03**_ | 8.15 |\n| 2.7 | 8.24 | _**7.96**_ |\n| 3.0 | 8.88 | _8.15_ |\n| 3.5 | 10.29 | _9.09_ |\n| 4.0 | 11.81 | _10.28_ |\n\nNotably, removing noise conditioning improves performance at optimal CFG scales. Also, for many different CFG scales, removing noise conditioning improves performance. We can see that in the setting of FM on large-scale dataset, the behavior of removing noise conditioning remains consistent with experiments in our paper. \n\nThese results show that it’s feasible to extend our conclusions to large-scale diffusion models, and our observations are robust with respect to classifier-free guidance.\n\n**2. Statistical Robustness of the Results**\n\n**Reviewer 7y3v:** `In Figure 7, (a) shows 1-RF achieving an FID of 3.01, yet (b) achieves 2.58 and (d) achieves 2.61. Does this suggest that the baseline result of 3.01 was an outlier due to variance in training or sampling? `\n\nTo address this concern, we report the variance of FID on 5 trials with different random seeds (the top row), as well as using three different training checkpoints (the bottom row). Results are as follows. These numbers correspond to the last row in Figure 7(a).\n\n| Model: FM (1-RF) | (a) | (b) | (c) | (d) |\n| --- | --- | --- | --- | --- |\n| different sample seeds | $3.01\\pm 0.02$ | $2.58\\pm 0.02$ | $2.65\\pm 0.02$ | $2.61\\pm 0.01$ |\n| different checkpoints | $2.98\\pm 0.03$ | $2.62\\pm 0.03$ | $2.66\\pm 0.01$ | $2.61\\pm 0.02$ |\n\nNote that here a, b, c, d are different settings.\n\nThese results confirm that the performance enhancement is not due to a statistical anomaly. The generation quality measured by FID-50k is robust enough for the evaluation.\n\n**3. Over simplification?**\n\n**Reviewer 7y3v:** `In Statement 1 and 2, the focus is on FM , while Section 5 is heavily EDM-oriented… oversimplifies the issue`.\n\nWe clarify that our analysis is valid for general cases. While we used FM formulation as our primary example, our analysis extends to all listed diffusion models. We chose FM for its notational simplicity and clarity, avoiding unnecessary complexity while maintaining theoretical rigor.\n\n**4. Explanations on 1-RF: Curvature of the Path**\n\nWe greatly appreciate your insightful perspective on how 1-RF inherently aligns with the diffusion ODE trajectory. \n\nOur theory mainly focuses on the bounding the error of removing noise conditioning, demonstrating that most models can work with the removal of noise-conditioning. We have to modestly admit that our theory is incapable of explaining why 1-RF becomes better when the noise-conditioning is removed. \n\nNevertheless, we believe that using the curvature of trajectory to build connection between the noise-unconditional model’s performance to specific diffusion formulations is a very promising future direction, and we would like to explore in more depth on this topic in follow-up works.", "claims_and_evidence": "The core idea of the paper is strong, but the analysis lacks exploration across a broader set of models. The experimental setup has low variance, raising concerns that some results might be due to chance. The model-wise analysis is limited, as the models selected are among the simplest available.\n\nOne issue is that diffusion sampling often involves classifier-free guidance, which introduces instability through extrapolation in $x_t$. Noise-unconditioning may exacerbate these instabilities. For example, Pan et al. (2023) shows that increasing classifier-free guidance from 1 to 2 to 3 significantly degrades performance in a diffusion inversion setting. Similarly, models like Latent Diffusion Models (LDM) and DiT might exhibit unpredictable behavior under noise-unconditioning.\n\nAnother key concern is why 1-RF performs better without noise conditioning. It is likely due to 1-RF's nature: it inherently learns the shortest path between the data distribution and Gaussian noise, which aligns well with the diffusion ODE trajectory, making noise conditioning unnecessary. However, this explanation is missing from the main text.\n\nDespite these gaps, Figures 2, 3, and 4 effectively support the logical flow between the paper’s core ideas and conclusions.\n\nRefs:\nPan, Zhihong, et al. \"Effective real image editing with accelerated iterative diffusion inversion.\" ICCV 2023.", "methods_and_evaluation_criteria": "The evaluation is somewhat limited. While FID is a reasonable choice given the research focus, relying solely on FID without additional measures is problematic. Running multiple trials and reporting the standard deviation of FID scores could improve the robustness of the evaluation.", "theoretical_claims": "The theoretical statements are mostly reasonable, but the paper mixes different types of diffusion models without clearly distinguishing them. Table 1 properly differentiates DDIM, EDM, and FM, but\nIn Statement 1 and 2, the focus is on FM, while Section 5 is heavily EDM-oriented. Since Table 2 shows that DDIM, EDM, and FM behave differently under noise-unconditioning, a general claim that \"DMs work without noise conditioning\" oversimplifies the issue. Instead, the results suggest that the effects of noise-unconditioning depend on the underlying diffusion formulation, requiring a more detailed explanation that is currently missing.", "experimental_designs_or_analyses": "Sections 4.2 and 4.3, along with Figure 3, focus primarily on FM models. However, to support similar claims for DDIM and EDM, equivalent experiments should be conducted for those models.\nAs mentioned earlier, the analysis does not sufficiently highlight model-specific differences, making some conclusions appear oversimplified.", "relation_to_broader_scientific_literature": "The ideas could be extended in conditional diffusion models (with varying classifier-free guidance settings) and latent diffusion models.", "essential_references_not_discussed": "The difference between DDIM failing and 1-RF succeeding likely stems from DDIM's trajectory curvature. A more detailed analysis of this could strengthen the paper.\n\nHere's one paper that potentially helps discussion:\n\nLee, Sangyun, Beomsu Kim, and Jong Chul Ye. \"Minimizing trajectory curvature of ODE-based generative models.\" ICML 2023.", "other_strengths_and_weaknesses": "The paper is well-written.", "other_comments_or_suggestions": ".", "questions_for_authors": "In Figure 7, (a) shows 1-RF achieving an FID of 3.01, yet (b) achieves 2.58 and (d) achieves 2.61.\nDoes this suggest that the baseline result of 3.01 was an outlier due to variance in training or sampling?\nWould running more trials and reporting standard deviations clarify whether 3.01 was a statistical anomaly?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u7gN4Q4QCO", "reviewer": "Reviewer_FoCJ", "summary": "This paper investigates the necessity of noise conditioning in diffusion models. It provides a theoretical analysis of the effects of removing noise conditioning and presents error bounds. The analysis shows that, under mild conditions, the errors resulting from the removal of noise conditioning are relatively small. The paper further conducts experiments to validate these findings. The results indicate that while noise conditioning is beneficial for enhancing sample quality, it is not a critical factor for denoising generative models.\n\n## update after rebuttal\n\nI thank the authors for their responses. I have read all reviews and authors' responses. I will maintain my scores.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the insightful feedback and the supportive comments to our work!\n\n**1.Definition & Benefits of removing noise conditioning**\n\n**Reviewer FoCJ:** `Without the noise conditioning, how are the models trained? … is it that we still train models with Eq. 2, but only replace `$NN_{\\theta}(\\mathbf{z}\\mid t)$` with `$NN_{\\theta}(\\mathbf{z})$?\n\nRegarding the question: Correct. Without noise conditioning, the neural network is still trained with Eq.2 but only with $NN_{\\theta}(\\mathbf{z}|t)$ replaced with $NN_{\\theta}(\\mathbf{z})$.\n\n**Reviewer FoCJ:** `… we actually still use the noise conditioning. That is, the t is implicitly used since` $z=a(t)x+b(t)\\epsilon$.\n\nThis is the main message we wanted to convey: the noisy image contains information of the noise level $t$, and this is why *explicit* noise conditioning in the neural net is not needed.\n\n**Reviewer FoCJ:** `I am curious about the objective of removing the noise conditioning`. \n\nOur motivation actually roots from the curiosity on the necessity of noise conditioning, which is the “common wisdom” in most of the previous works on denoising generative models. In our opinion, the value of removing noise conditioning lies in challenging common sense and also lies in its theoretical meaning. Finally, as a direct result, some models (such as DiT+FM, as we share with Reviewer gvd2 and 7y3v) can enjoy improvement in removal of noise conditioning.\n\n**2. Low-Dimensional behavior**\n\nReviewer FoCJ suggests to investigate model behavior in the senario where dimension $d$ is low, since our theory assumes a large enough $d$ value. This suggestion is very insightful. \n\nInspired by that, we explored low-dimensional cases, where the approximation $d\\gg 1$ does not hold and our theoretical analysis is ineffective. Specifically, we did experiments on the toy “two moons” dataset (which has $d=2$) with Flow Matching (i.e. Flowing from standard Gaussian to the two-moons dataset) models, and visualize the generated samples in the noise-conditional and unconditional settings. The results are shown in the figure below:\n\n[reb — Postimages](https://postimg.cc/dZwWS6hF)\n\nBy visualizing the generated samples, one can see that removing noise conditioning leads to a significant performance drop in low-dimensional cases, preventing proper modeling of the distribution.", "claims_and_evidence": "The claims are generally clear and supported by theoretical and empirical analysis. I only have a few questions\n\n1. Without the noise conditioning, how are the models trained? That is, is Eq. 7 the objective for training? If yes, how do we sample $\\mathbf{z}$ and compute the expectation? If no, is it that we still train models with Eq. 2, but only replace $NN(\\mathbf{z}|t)$ with $NN(\\mathbf{z})$?\n2. If we still use Eq.2 for training and only replace $NN(\\mathbf{z}|t)$ with $NN(\\mathbf{z})$, we actually still use the noise conditioning. That is, the t is implicitly used since $\\mathbf{z} = a(t)\\mathbf{x} + b(t)\\epsilon$.", "methods_and_evaluation_criteria": "The proposed methods make sense.", "theoretical_claims": "I checked the proof in Appendix B, and did not find issues.", "experimental_designs_or_analyses": "The experiment results are sound. One question I have is\n\n1. The paper states that the error bounds are related to the feature dimension $d$. In experiments of image sets, the $d$ is much greater than $1/t$, so the statements hold. It would also be helpful to vary $d$ and see how the error bounds are impacted.", "relation_to_broader_scientific_literature": "The paper mainly explores the impact of noise conditioning in denoising diffusion generative models. Its analysis and results can be generalized to many different variants of diffusion models, e.g., flow matching and consistency models.", "essential_references_not_discussed": "Related works are sufficiently discussed.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. I am curious about the objective of removing the noise conditioning. That is, it seems that it will not reduce training/inference time too much, but the model performance may be impacted, i.e., in Table 2, most methods are worse.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}]

Author rebuttal:

Meta-review: This paper argues that noise conditioning is not as vital as previously thought to obtain good performance with diffusion models. The reviewers all praised the theoretical contributions and empirical results on CIFAR-10, but criticized the lack of experiments on higher-dimensional data. The authors provided such experiments in their rebuttal, assuing the reviewer's concerns. I thus recommend acceptance.
Decision: Accept (poster)