Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training
Authors: Yunshu Wu
Yingtao Luo
Xianghao Kong
Evangelos E. Papalexakis
Greg Ver Steeg
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion
Primary area: diffusion_based_models

Abstract: Diffusion models learn to denoise data and the trained denoiser is then used to generate new samples from the data distribution. 
In this paper, we revisit the diffusion sampling process and identify a fundamental cause of sample quality degradation: the denoiser is poorly estimated in regions that are far Outside Of the training Distribution (OOD), and the sampling process inevitably evaluates in these OOD regions.
This can become problematic for all sampling methods, especially when we move to parallel sampling which requires us to initialize and update the entire sample trajectory of dynamics in parallel, leading to many OOD evaluations. 
To address this problem, we introduce a new self-supervised training objective that differentiates the levels of noise added to a sample, leading to improved OOD denoising performance. The approach is based on our observation that diffusion models implicitly define a log-likelihood ratio that distinguishes distributions with different amounts of noise, and this expression depends on denoiser performance outside the standard training distribution.
We show by diverse experiments that the proposed contrastive diffusion training is effective for both sequential and parallel settings, and it improves the performance and speed of parallel samplers significantly. Code for our paper can be found at https://github.com/yunshuwu/ContrastiveDiffusionLoss

Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training
Yunshu Wu', Yingtao Luo?, Xianghao Kong', Evangelos E. Papalexakis', Greg Ver Steeg'
University of California Riverside, CCrmegie Mellon University
(ywu380, xkogg016, , e gregoryv)@ucr. edu, yingtaol@andrev. cmu. edu
Abstract
Diffusion models learn to denoise data and the trained denoiser then used to generate new samples from the data distribution. In this paper, revisit the diffusion sampling process and identify a fundamental cause of sample quality degradation: the denoiser is poorly estimated in regions that are far Outside Of the training Distribution (OOD), and the sampling process inevitably evaluates in these OOD regions. This can become problematic for all sampling methods, especially when we move to parallel sampling which requires us to initialize and update the entire sample trajectory of dynamics in parallel, leading many OOD evaluations. To address this problem, we introduce a new self-supervised training objective that differentiates the levels of noise added sample, leading to improved OOD denoising performance. The approach is based on our observation that diffusion models implicitly define likeliiod ratio that distinguishes distributions with different amounts of noise, and this expression depends on denoiser performance outside the standard training distribution. We show by diverse experiments that the proposed contrastive diffusion training is effective for both sequential and parallel settings, and improves the performance and speed of parallel samplers significantly.
Introduction
Denoising diffusion models [29] achieve state-of-the-ar performance various unsupervised learning tasks and have intriguing theoretical connections to methods like denoising autoencoders 37 VAEs (6). stochastic differential equations 20 34] information theory [3], and score matching [31 32.. Diffusion models are presented with data samples corrupted by a forward dynamical process that progressively adds more Gaussian noise and trained to reverse this dynamics denoise the corrupted samples. Samples are then generated by applying the reverse dynamics on images of pure Gaussian noise produce high-quality samples from the target distribution.
The key to the success diffusion models is the dynamics that gradually bridges the source and target distribution, but it suffers from slow sampling, sequentially simulating these dynamics can take thousands of denoising steps for one sample. Most recent works attempt to expedite the sequential dynamics by taking fewer, larger steps [30, II 1g 22). However, the complexity of these samplers and the need for expensive sampling hyper-parameter grid searches tailored to specific
datasets makes them difficult to generalize.
Shih et al. 2024 suggests a different approach by randomly initializing the entire path of the reverse dynamics and then updating all the steps in the path in parallel. The parallel sampling approach promises to drastically reduce wall-clock time the cost of increased parallel computation. However, it encounters problem that has largely gone unnoticed in the sequential sampling literature: while
Code be found at httpa////aa.//tciooo/oaaaaaaaaaa
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1D two mode Gaussian Score Estimate Error
k=3
Parallei sampling trajectories
15.0
k=2
12.5
10.0
k=1 (initialization)
7.5
5.0
2.5
200
400
600
800
1000
Timesteps
We plot the error in the score estimate for ID two mode Gaussian example where diffusion dynamics bridge between a Gaussian and mixture (see Appendix A.3). Regions near the standard forward training data paths have lower error magnitude (light), whereas other areas have higher error magnitude (dark). While sequential samplers adhere as closely as possible to low-error regions, parallel samplers initialize and update the entire sample trajectory (blue trajectories), leading to evaluations in high-error regions. When the sampling trajectory initialized, most are inevitably in the OOD regions and will update the low-error regions gradually.
Figure 1:
sequential paths sampled during generation are designed carefully to stay as close as possible to forward paths that add noise to data, the parallel sampler often evaluates in regions far from where score estimate (the denoiser) is trained, as illustrated in Fig. T The shading shows that the error of the score estimate is large in these regions, leading to poor performance for parallel samplers. Discretization errors can lead to similar issues, even for standard sequential samplers.
We propose to improve the training of diffusion models so that the error of the denoiser is reduced in OOD regions, and we hypothesize that this should significantly improve the performance for parallel samplers as they require more OOD evaluations. Our approach starts with an unexpected connection: the optimal MSE denoiser that defines the diffusion dynamics also defines an optimal noise classifier that distinguishes between samples with different amounts of noise. This provides a useful additional signal for training, because optimizing evaluating the denoiser for one noise level on samples from distributions at different noise levels, while standard MSE optimization only evaluates the denoiser on samples from the matching noisy distribution. Accurate denoiser evaluation in regions that are OOD for standard diffusion training is important for robust sampling dynamics.
the noise classification task involves
Contributions:
We the inoomatiion-therreee formulation of diffusion to draw connections between diffusion, log- ligeelhoo ratio estimation, and classification. This reveals that optimal diffusion denoisers are also implicitly optimal classifiers for predicting the amount of noise added to an image.
We leverage the noise classifier (via density ratio estimation interpretation introduce a novel self-supervised loss function for regularizing diffusion model training, which we call the Contrastive Diffusion Loss (CDL). CDL provides training signal in regions that OOD for the standard MSE diffusion loss.
We show that CDL improves the trade-off between generation speed and sample quality, and that this advantage is consistent across different models, super-parameters. and sampling schemes. The improvement is especially substantial for parallel diffusion samplers 28 which rely heavily on OOD denoiser evaluations.
2 Diffusion Model Background: Optimal Denoisers are Density Estimators
The defining feature of diffusion models is sequence of distributions that progressively add noise to the data, from which we then learn to recover the original data. The ("variance preserving" (34)) channel that mixes the signal æ with Gaussian noise defined as æ,, = / + <-(-are with ~ (O, ~ p(z), where represents the log of the Signal-to- Noise Ratio (SNR), p(æ) is the

---

unknown data distribution for a E IR', and of-) is the sigmoid function. We define the sequence of intermediate distributions drawn according to this channel with subscript as p. (x). By definition, we express -i Po (x) p(x) in this paper. Note that we use a different scaling convention for noise from [ and [6] where the former one takes æ + de as the forward noising channel and the latter one takes va, + v Ot5 the forward noising channel. For further detailed relationships among these scaling conventions, please check App. B.3
The minimum mean square error (MMSE) estimator é for recovering from the noisy channel that mixes æ and e can be derived via variational calculus and written as follows.
É(I,.. a) Be-p(e/a..) arg min Bp(c)p(a)) |l|e - É( a) a)1
(1)
<(->)
Sampling from the true posterior is typically intractable, but by using neural network to approximate the solution to the regression optimization problem, we see that log-likeliine lilelikhehoodd can be written exactly in terms of an expression that depends only on the MMSE solution to the Gaussian denoising problem, i.c.
get approximation for é. From [3],
- -logp(x) = c 1/2 L_ Epte)llee é(æa, a)] da.
(2)
The constant, d/2 log((2r) d Jo dā (ā) does not depend data and will play role in our approach, as it cancels out our derivations in Sec. 3
3 What Your Diffusion Model is Hiding: Noise Classifiers
We now introduce our first main result, which shows that diffusion models implicitly define optimal noise classifiers. Eq. (2) expresses the probability density of the data directly in terms of the denoising function. If we apply Eq. (2) the noisy distributions that bridge the data and a Gaussian, Pç (æ), we can see that all mixture densities can be written in terms of the same optimal denoising function, é(.,)) The complete derivation is presented in App. A.2
/ da E b. E(Fa. B)1
log Pc (x) c 1/2
(3)
La = Vo(a)= + Va(-aje 8=o (o(()(a)) b (=(a)\))
(4)
Intuitively, if we find the optimal denoising function for the data distribution, it may be hypothesized that can denoise an already noisy version of the data distribution. Using Eq. [] this directly translates into an expression for density of mixture distributions. Differences in log likelihoods lead to cancellation of constants, and these Log Likelihood Ratios (L.LR) are related the optimal classifiers 4 as we show below.
To connect LLRs with classification, consider the following generative model. We generate random binary label g(y = + = 1/2. Then, conditioned on y, we sample from some distribution g( vy. Given samples (ær 3) q(xx v) (<vly))(y), the Bayes optimal classifier is:
q(x) y)q(z)
q(<lv)q(1)
<(y|x)
((x))
q 1)4(y = qapl 1)a(y 11)
1/(1 + qxx - y) ) 1/(1 cxp(y(loo = -1) - blog 9 ly = 1))))
q( y)
log (()(z)) = - loo((1 + exply log qx>ly = - softplusss log
g(øly = =11) g(øly =
(6)
q(Øly= =
In the second line, because Vy, q1)) = 1/2, these constants cancel out. Then we can just expand definitions and re-arrange write in terms of log probabilities.
Contrastive Diffusion Loss (CDL) Our next contribution is to use the new connection between diffusion denoisers and noise classifiers to define new training objective. We set the distributions q(ally = 1) and (<x by = 1) to be two distributions at different noise levels that we can write terms of the optimal diffusion denoiser from Eq. 3 So we have q( xxyy- 1) = p(æ), the data distribution,
3

---

and qلااا -1) = Ps (z). for some noise level, Ç. Then given a sample (x. ~ g(2, v) the
per-sample cross-entropy loss for the noise classifier, Eq. @ is as follows.
LCDL = By(a..)) (softplus(u((oggp (a)) log p(a)))
We can estimate both densities directly from our denoising model using Eq. 3 with the constants canceling out in the process. This loss differs significantly from the standard diffusion loss. Intuitively, to distinguish between a sample from the data distribution, p(z), versus a noisy version of the data distribution, Pz(x), we need to evaluate denoisers on points from both distributions. In standard diffusion training. denoisers at noise level Ç are only trained on samples from Pc(x). Limitations: We highlight that CDL is more expensive to compute than the standard diffusion loss, significantly increasing the total cost of diffusion model training. Implementation details appear in App. B.4 and training cost details appear in App. B.5
Choice noise to contrast Next, let's break the Log-Likelihood Ratio (LLR) term in Eq. down to see how to choose C to maximize the benefit of CDL. Combining Eq. 2 and Eq. we have Eq. (8. where the constant cancels out.
LLR log) log p()) [ da Eele)llee é(z, a)] El be(z, 8)1
(8)
with: = Vo(a)x + Va(-a)e
Note that the input a to the LLR term may come from two different distributions, which breaks the standard synchronous denoising pair (Ia, ()) into asynchronous. When it's from data distribution
x ~ p(x), == = Zai and when it's from some noisy data distribution æ ~ Pc (x:) : Z3.
From Eq. 8 we see that é(-,-) is trained on four pairs: (Za, at,, (=3, 9), (Z3, a) and (za. 3). where = o ( (()) < min(( a, () (Eq. 5) During standard training, only the first two pairs are trained (Eq. D. This means that CDL objective trains the denoiser perform correctly even for samples from distributions that are noisier or cleaner than the specified noise level (a pair like (z3, o) or (za, 3)). This can be useful for both sequential and parallel sampling settings. During sequential sampling, extra error noise added due to discretization errors can be corrected by the denoiser trained with CDL. As for parallel sampling, CDL helps with evaluations on asymmetric pairs (æg, a) or (Zo: 3) which we refer to OOD regions for standard diffusion loss.
In practice, diffusion training pipelines are highly tuned popular datasets like CIFAR1O and ImageNet, so the amplitude of discretization errors during sampling is small, meaning that errors won't 't nudge points too far away from the true trajectory. Therefore, when evaluating CDL objective, we sample some large-valued (s. which corresponds to classifying only small differences in noise levels. Empirically find that Ç~ Uniform(6, 15) or Ç (~ logistic(6,, 15) performed equally good.
Denoising, sampling dynamies, and the score connection We have focused so far on denoising and density estimation, but we now want to connect this discussion to the primary use case for diffusion models and the focus of Sec. 4 sampling. There are many choices in how to implement sampling dynamics [Α]] but all of them rely on the score function, V. log Pa (x)). The score function points toward regions of space with high likelihood, and by slowly transitioning (or annealing), from the score function of noisy distribution to one closer to the data distribution, sampling dynamics. To connect denoisers with sampling we must show that a denoising function, é, that is optimal according to Eq. alsoo specifies the score function.
build reliable
č(x, a)
Vx log (x)
Va(-o)
The derivation is straightforward and is given in Appendix
A.I
4 Sequential and Parallel Sampling with Diffusion Models
Sampling dynamics are typically presented in terms of a stochastic process {æ. ) with timestep,
t, rather than in terms of log SNR, a. We will denote 2; Z.((t), Pe Patci to connect to our previous notation, with (<()) representing a monotonic relationship described in App. B.3 Note that decreasing log-SNR a corresponds to increase timestep I, since smaller log-SNR means there more noise added to the data.

---

The general form of sampling dynamics is process of slowly transitioning from samples of simple and tractable distribution to the target distribution. Specifically, start with an isotropic Gaussian ~ (0, I), the sampler steps through series of intermediate distributions with noise levels 1.... } following the score estimates. Many works [4 11 interpret diffusion models as stochastic differential equations (SDEs). The forward process is in the form of
dx;
( t) +
g(t) dwi:
Io p(x)
(10)
diffusion
where wy is the standard Wiener process/Brownian motion, J and are drift coefficient and diffusion coefficient of x; separately. The reverse process of Eq. [0 then used to generate samples
dx, (f,.,t) g" (t) log Pa (x)) dt + g(t) dw.,
2T ~ x())
(11)
dritt
diffusion
Depending on choices for f.g. we can get either stochastic or ordinary (deterministic) differential equation. Either way, numerical differential equation solvers are used to approximate the true dynamics. The solver introduces discretization errors each step, causing the trajectory to deviate into the OOD region where the score (or denoiser) is poorly estimated, further compounding the errors. More discretization steps reduce accumulated error and leads to better sample quality, the expense of more sequential computation. As a result, a significant limitation of diffusion models is that they require many iterations to produce high quality samples.
Sequential Sampling The influential diffusion sampler DDPM 66 iterates over thousands of
discretization steps simulating the dynamics. Recently, many sequential sampling methods have been developed to take fewer and larger steps while introducing less error [30 [2 IT]. Specifically, Karras et 2022 studies the curvature shape of SDE/ODE trajectory and suggests discretization technique where the resulting tangent of the solution trajectory always points towards the denoiser output. However, speeding up the sequential sampling sacrifices generation quality. Furthermore, the SOTA sequential samplers [18] [1] 30 require hyperparameter tuning and grid search on specific datasets, which poses challenges the generalization of these samplers to other datasets.
A
ar
2
J++1
I I++
F+1
1
Figure 2: The computation graph of Picard iteration for parallel sampling Parallel Sampling Shih et al. 202400xxlooeeeeo a parallel sampling scheme, where the entire reverse process path is randomly initialized and then all steps in the path are updated in parallel. Parallel sampling is based on the method of Picard iteration, an old technique for solving ODEs through fixed-point iteration. An ODE is defined by drift function s(x, I) and initial value zo. In the integral form, the value at time / can be written as
28
1' s(x u) du
D = Do
In other words, the value at time t must be initial value plus the integral of the derivative along the path of the solution. This formula suggests natural way of solving the ODE by starting with guess k+1 of the solution { 0 < < 1) at initial iteration & = O. and iteratively refining by updating the value at every time t until convergence
2/ L' s(x u) du
(Picard Iteration)
k+1
(12)
To perform Picard iterations numerically, which is shown in Fig. we can write the discretized form of Eq. 12 with step size 1/T, for 1. [0, T):
k+1 2 x
+
1 T
Σ
i/T)
(13)
For detailed convergence proof, we refer Shih al. 2024 section 3.
5

---

We see that the expensive computations {s(z) i/T)) E (0,T)} can be performed in parallel. After some number of Picard iterations, the error difference between two iterates || x I drops below some convergence threshold. This converged trajectory, zi. should be close to the sequential sampler trajectory. Looking at the example in Fig. we show the trajectories of three iterations ==1, 2,2,3. The trajectories before convergence are consistently appearing in the regions with high score error.
5 Experiments
We sample from models fine-tuned on Contrastive Diffusion Loss (CDL) via both parallel and sequential diffusion samplers across variety of generation tasks, including complex low-dimensional manifold 2D synthetic data and real-world image generation. Our results demonstrate that employing CDL as regularizer in models trained with standard diffusion loss enhances density estimation and sample quality while also accelerating convergence in parallel sampling. All sampling tests are done on A6000 GPUs. We visualize CDL image generation examples in App.[C
Training configuration Our method is echitecturr-agnosttic In synthetic experiments, we adopt a simple MLP architecture with positional encoding for timesteps | it is one of the most versatile models in the literature. In real-world experiments, we consider the standard diffusion loss with two training configurations: DDPM by HH et al. 2020 and EDM by [Karras al. 2022
For more details
on model training, data split, and hyper-parameters, please refer to B.2
B.2
Generation quality metrics For real-world data, our intrinsic metric is Frechet Inception Distance (FID) score (5). The number of images we generated for FID computation follows their baseline models' FID settings, and the FID scores are computed between 5, 0000 generated images and all available real images.
For synthetic data, to measure how well the generated samples resemble samples from the ground truth distribution, we use the (unbiased) kernel estimator of the squared Maximum Mean Discrepancy (MMD), with Gaussian kernel with bandwidth set empirically as described in App. B.T
Sampling speed metrics We adopt the following three metries: (1) Neural function evaluations (NFE) for all settings, i.e. how many times the denoiser is evaluated to produce a sample; (2) For the parallel setting, we report the number of parallel Picard Iterations: (3) Furthermore for the parallel setting, the wall-clock time is reported. While parallel sampling can use fewer total iterations and less wall-clock time than sequential sampler, this may come at the cost of increase in the total number of function evaluations. This gap is called the algorithm inefficiency. In the subsequent section, we use contrastive diffusion loss as training regularizer for standard diffusion losses and refer to the corresponding models as CDL-regularized models.
5.1 Parallel Sampling
In the parallel setting, we use Parallel DDPM sampler [28] with 1000-step diffusion sampling. And for synthetic experiment, to reflect sampling speed by only number of Picard iteration and wall-clock time, we don't 't use the sliding window technique, and the 2D data is small enough to fit the whole sampler trajectory GPU memory. While for real-world experiment, sliding window is still applied.
Synthetic Dataset We consider the 2D Dino dataset [19]. characterized by its highly nonlinear density concentrated on diw-dimnsiional manifold. For baselines, we employ the standard DDPM loss (6), as all standard diffusion losses similarly minimize a sum of MSE losses between the actual and estimated denoisers. Both CDL-rreularize and DDDMMoohjeciive-rrainn with MLP where timestep is encoded by positional encoding. We train it for 2000 epochs to ensure convergence and check the training and validation loss curve to avoid overfitting.
models are trained
In Fig. 4 it is clearly demonstrated that the parallel generated samples from the CDL-regularized model is much better than the model trained only with the standard DDPM loss, especially around the chin, eyes and paws, where the manifolds are close to each other and difficult to distinguish and learn. From the MMD plot, we see that comparing to the baseline curve trained only with standard DDPM loss, the CDL-regularized curve converges faster with smaller number of Picard iterations and better
AArchitcctrre adopted from: http::/giitbbbcomm///nnnnsssDDss

---

Contrastive loss
DDPM loss
0.007
0.006
(a) Ground-tnuth dino.
(h) Using CDT. loss
(c) Using DDPM loss.
0.005
Figure DDPM sampler generated
Comparing to Dino
6.004
sampled from DDPM loss, CDL-loss sampled Dino has better sample quality and density estimate around hard areus.
6.003
Model Loss
MMD
Vlter
NFE
Time (ms)
DDPM
0,0031
36
14,397
1,870
0.0012
13,
1,368
32
Number of Picard Iterations
Table I: Parallel DDPM sampling speed results. We generate 000 samples, Figure 5: Parallel sampling. MMD We that the Here we set MMD threshold= 0.002, and Hiter refers to number picard regulaauldr model and DDPM model converge them- iterations till MMD threshold. Both NEE and Time counted till parallel selves 35 and Picard iterations, separately. convergence.
sample quality (lower MMD scores). Table. that CDL-regularized model converges faster with lower final MMD and better sample quality.
D shows the sampling speed results, from where
Real-world Datasets We select "DDPM++ cont. (VP)" and "NCSN+ cont. (VE)" models by trained on CIFAR-10 at 32 32, unconditional FFHQ, and unconditional AFHQv2 [5] TO 3 as
baselines. comparing to the corresponding DD-regularized models. We adopt the pre-trained models from Ho et al. 2020 and Karras et al. 2022 More experimental results can be found in App. B As shown in Tab. CDL rregularized models always outperformed basclines with respect to FID
scores.
CIFAR-10 10 at 32x32
AFHQv2 64x64
FFHQ 64x64
Models
unconditional
conditional
unconditional
unconditional
VP
3.24 4 0.02
2.93 0.02
2.95 1 0.03
3.67 + 0.04
CDL-VP
2.51 + 0.01
2.41 +: 0.01
2.91 + 0.02
3.33 + 0.03
VE
3.00 + 0.01
2.76 + 0.01
2.98 + 0.03
3.65 + 0.02
CDL-VE
2.38 + 0.01
2.25 + 0.02
2.93 + 0.01
3.29 + 0.02
Table 2: Evaluating FID score (lower is better) of parallel DDPM sampler real-world datasets
using 5, 0000 samples. For reported FID scores, we run three sets of random seeds and reported the average with uncertainty.
5.2 Sequential Sampling
While CDL clearly improves parallel sampling quality and convergence speed, also show that it improves the trade-off between generation speed and sample quality sequential diffusion sampling choices, consider the DDPM sampler from Ho et al. 2020), and both the deterministic and stochastic samplers from Karras et al. 2022 To ensure fair comparisons, we adopt the original sampling aaper-parameter settings for all baselines.
sequential setting. As for
Deterministic samplers For FIID test, we follow the exact sampling settings outlined Karras et al. 2022 for each dataset. FID scores are reported in Tab. 3 for sequential deterministic EDM samplers, CDL objective ensures that the generation quality is consistently similar or better.
In principle, increasing NFE has the potential to decrease the overall discretization errors, conse- quently leading improved sample quality. However, in practice we observed an unusual behavio with the Karras deterministic sampler- as NFE increases, the FID score deteriorates (Fig. 6). In contrast to EDM models, CDL-regularized models exhibit more stable FID score. This partially resolves the deterministic sampler sensitivity while improving the quality.
Stochastic samplers In practice, stochastic samplers often yield superior performance compared to deterministic ones. However this is not true in Karras et al. 2022 stochastic samplers outperform
bttspa///githil wttptthab.com//laaa//eeeeeeee
The same issue is also reported httpt::::.. httptt://gttthttt http:////gghhbbb..ooeeeeeeeeeeee cot.///iabs//chhhsssseee

---

CIFAR-I0 32x32
AFHQQ2 64x64
FFHQ 64x64
Models
unconditional
conditional
unconditional
unconditional
VP
2.00 1 0.02
1.84 1 0.02
2.04 1 0.00
2.38 1 0.01
CDL-VP
1.99 + 0.04
1.82 + 0.03
2.00 + 0.00
2.29 + 0.02
VE CDL-VE
2.01 + 0.01
1.81 + 0.01
2.17 = 0.00
2,56 + 0.03
2.01 + 0.01
1.81 + 0.01
2.11 + 0.01
2.47 + 0.02
NFE (EDM/CDL)
35
35
79
79
Table 3: Evaluating sequential deterministic EDM samplers generation quality. For reported FID scores, we run three sets of random seeds and reported the average with uncertainty.
CIFAR-10,
[b] Uncond. CIFAR-10, VP
[c) Uncond.
Uncond. AFHQ-V2.
2:15
2.625
COL
1.00
1.88
2,000
2.6
1.975
2.05
1.86
1,950
2,00
N1E
32
128
NFE=26
32
64
128
NFE-16
32
64
128
MPC-16
32
64
120
Cond. CIFAR-10,
(1) Uncond. CIFAR-10, VE
(g) Uncond. FFHQ, VE
(h) Un AFHQ-v2. VE
2.90
2,30
2.05
2.25
2 2.00
2.8
2 2.20
1.84
1.95
2.6
2.15
2:10
NFE
MFE-16
NFE-16
128
16
32
Figure 6: The FID comparison between our CDL and the baseline EDM in the deterministic sampler experiment.
deterministic ones only on challenging datasets, for simpler datasets, the introduction of stochasticity not only fails to enhance performance but exhibits image degradation issues, characterized by loss of detail. They attribute this phenomenon to 1.2-trained denoisers excessively removing noise at each step (always remove more than it should), and propose to slightly increase the standard deviation (Snoise) F of newly added noise to 1.007. We argue that this approach may not totally resolve the issue and instead complicates the hyperparameter grid search process by introducing additional parameter, Spoiee- Also, this Smoise logically serves the same function as another hyperparameter "i. where both of them control the amount of noise to add reach a higher noise level.
In this experiment, we conducted two stochastic sampling configurations for our baseline EDM- trained models. The first configuration, referred to EDM-opt, operated the EDM optimal setting with Sposse = 1.007. The second, named as EDM-sub-opt, used setting with Snoise = 1.00, effectively disabling Suoise As for CDL configuration, we exclusively examined the scenario with Smoise 1.00 to determine whether CDL could address the problem of excessive noise removal.
The results, visualized in Figure 7 indicate that CDL outperforms EDM in both Spoise configura- tions. Notably, CDL not only improves upon the EDM-sub-opt configuration (dark blue line) but also surpasses the performance of the EDM-opt configuration (light blue line), even at its optimal setting. This not only demonstrates that CDL robustly provides a better sample quality, but also suggests that CDL can climinate the need for the hyperparameter Spoise. This reduction enables more efficient grid search for the optimal EDM sampling settings, potentially enhancing the practicality of using such sampler for other applications.
Related Work
The generative modeling trilemma 39
seeks generative models that produce samples (i) quickly, (ii) with high quality, and (iii) good mode coverage. Diffusion models excel at the latter two but large amount of research has attempted to address the problem of slow sampling speed. From the inception of diffusion models [29]. dynamics has been at the forefront, most work has focused on
We refer Karras et al. ||| for details
8

---

Stochasticity on CIFAR-10
EDM w. Shose= 1.000
EDM w. =.000=
4.0
CDL W. Stase =1 #.000
3.5
3.0
2.5
2.0
40
Sihamn
Figure 7: The FID comparison between our CDL and the baselines EDM in the stochastic sampler experiment on CIFAR-10. CDL's performance is strictly better for all Sehum outperforming the optimal setting of EDM which inflates the standard deviation Sehurn of the newly added noise.
the interpretation of the dynamics as a differential equation that can be sped up through accelerated numerical solvers [30 34 42 9 [7]. Our approach is compatible with any of these approaches as we are sampler agnostic, seeking only to improve the input the sampler, which is the denoiser or score function estimator, through regularization during the diffusion model training. A separate line of work instead attempts distill diffusion model into a faster model that achieves similar sampling quality 22 35
21 38 39. In principle, these methods could also benefit from distilling based on more robust base diffusion model trained with CDL.
Diffusion models admit a surprisingly diverse array of mathematical perspectives, like variational perspectives 6 7 122, differential equations 34 20 and nonequilibrium thermodynamies [29] Our
approach is mostly inspired by connections between the information- -hheorrtccc perspective [3 44
and the score matching perspective [3] 33 32
37 In particular, we point out that score function
estimates in traditional diffusion training are sub-optimal, and the anformation-theoreet perspective leads to a new objective (CDL) that can improve the score estimate.
While previous diffusion models focus on log- log-likeldoodd estimation, we consider a different approach based density ratio estimation and noise contrastive estimation 4 23 which inspired several notable developments in machine learning [0]. A few works have considered contrastive learning inspired modifications to diffusion either enforce multimodal data relationships [16] 43) for style transfer [41), or for guidance during generation (2), but none use the diffusion model as a noise approach methods that use Density Ratio Estimation (DRE) to estimate a ratio between the data density and some simple noise distribution. The density ratio can be estimated by learning to contrast between data samples and samples from the noisy distribution 4.. 36 Recent work generalized the idea to consider classifying between samples along sequence of distributions between source and target (26 2 Our contribution is to relate this approach to diffusion models by noting that standard diffusion models implicitly already implement the required classifiers for distinguishing distributions on the path from the data distribution to a Gaussian distribution. Concurrent work makes similar connection but while we focus on improving diffusion models by interpreting them as noise classifiers, 400 focused on the converse perspective, improving density ratio estimation by interpreting DREs as denoisers.
classifier improve diffusion training we do. Most similar
7 Conclusion
In this paper, we introduced novel connection between diffusion models and optimal noise classifiers. While this relationship has variety of potential applications that could be explored in future work, we used the connection to propose a new self-supervised loss regularizer for diffusion models, the Contrastive Diffusion Loss (CDL). CDL reduces the error of the learned denoiser regions that are OOD for the standard loss. We showed that CDL improves the robustness of diffusion models across all types of sampling dynamics, and leads significant speed-ups for promising new generation of parallel samplers.

---

Acknowledgements
This research was partially supported by the National Science Foundation under grant no. IIS 1901379.
Broader Impacts: This paper aims innovate on the methodology of diffusion models. We anticipate no direct potential societal consequences of our work, as the main focus of this work is in theory and algorithm design. However, it is important to acknowledge the potential risks associated with diffusion models, as misuse can contribute to the spread of disinformation and deepfakes.
References
[1] Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR. 2020.
[2] Kristy Choi, Chenlin Meng, Yang Song, and Stefano Ermon. Density ratio estimation via infinitesimal classification. arXiv preprint arXiv:2111 2021.
[3] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: multiple domains. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8188 8197, 2020.
Diverse image synthesis for
[4] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statisticx, pages 297-304, JMLR Workshop and Conference Proceedings, 2010.
[5] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances neural information processing systems, 30, 2017.
[6] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[7] Chin-Wer Huang, Jae Hyun Lim, and Aaron C Courville. A variational perspective diffusion-based generative models and score matching. Advances Neural Information Processing Systems, 34:22863- 22876, 2021.
[8] Aapo Hyvärinen and Peter Dayan. Estimation of non- normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Alexia Malicoeur-Martinennn Ke Li, Rémi Piché-Taillefer, Tal Kachman, and loannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105. 1400000000. 2021.
[10] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410 2019.
[11] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models, arXiv preprint arXiv:2206.0 003644 2022.
[12] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2107.006. 2021.
[13] Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. ialformatioon-theorrtii-iiidi diffusion. In International Conference on Learning Representations, 2023. BRRLLittss//aariiv.orr//aaa//
03792
[14] Xianghao Kong, Ollie Liu, Han Li, Dani Yogatama, and Greg Ver Steeg. Interpretable ditfusion via information decomposition. In The Twelfth International Conference on Learning Representations, 2024.
[15] Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009. URL httpa://vww toronto..oo..o....... wwteeedii////////eeee www.eaannn-eeauureeeeeeeeeee 20.--2000090--00 TR.
[16] Chaejeong Lee, Jayoung Kim, and Noseong Park. Codi: Co-evolving contrastive diffusion models for mixed-type tabular synthesis. arXiv preprint arXiv:2304.1 12654. 2023.
[17] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arxiv preprint X2v:2202.009778. 2022.
10

---

[18] Cheng Lu, Yuhao Zhou, Fan Bao. Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of difffusion probabilistic models, arXiv preprint arXiv:2211.01095 2022.
[19] Justin Matejka and George Fitzmaurice. Same stats, different graphs: generating datasets with varied appearance and identical statisties through simulated annealing. In Proceedings of the 2017 CHI conference on human factors in computing systems, pages 1290-1294, 2017.
[20] David McAllester. On the mathematics of diffusion models. arXiv preprint arXiv:2301.11100, 2023.
[21] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma. Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297-14306, 2023.
[22] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162 8171. PMLR, 2021.
[23] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03:: 00704348.00008...00 2018.
[24] Yidong Ouyang, Liyan Xie, and Guang Cheng. Improving adversarial robustness by contrastive guided diffusion process. arXiv preprint arXiv:2210.06443 2022.
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabricl Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[26] Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Advances in neural information processing systems, 33:4905-4916, 2020.
[27] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2021. 2021.
[28] Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[29] Jascha Sohl-Diia Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. arXiv preprint arXiv:1503.00 035555 2015.
[30] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.0255 022.00 2020.
[31] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[32] Yang Song and Stefano Ermon. Improved techniques training score-based generative models, Advances in neural information processing systems, 13:12488-12248, 2020.
[33] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: scalable approach to density and score estimation. arXiv preprint arXiv: 1905.0008078 2019.
[34] Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- generative modeling through stochastic differential equations. arXiv preprint 1.Xiv:201I...44 2020.
[35] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469 2023.
[36] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Bibliography, page 309-326. Cambridge University Press, 2012.
[37] Pascal Vincent. A connection between score matching and denoising autoencoders. Newral computation, 23(7):1661-1674, 2011.
[38] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality.
[39] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In International Conference Learning Representations, 2021.
11

---

[40] Shahar Yadin, Noam Elata, and Tomer Michaeli. Classification diffusion models: Revitalizing density ratio estimation. 2024.
[41] Serin Yang, Hyunmin Hwang, and Jong Chul Ye. Zemo-soo contrastive loss for text-guided diffusion image style transfer. arXiv preprint arXiv:2303.08622 2023.
[42] Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. Advances in Neural Information Processing Systems, 34:16280-16291, 2021.
[43] Ye 7hu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan. Discrete contrastive diffusion for cross-modal and conditional generation. arXiv preprint arXiv:2206.0777,, 2022.
12

---

A Derivations and Proofs
A.1 Score Connection
We derive the following relation.
E(xx a)
Vx log (a))
(14)
(a(-a)
To keep track of intermediate random variables and their associated distributions, we will use the cumbersome but more precise information theory notation where capitals represent random variable and lowercase represents values. Define the channel that mixes the signal, X, with Gaussian noise as Za = Vo(aax + vo(-a)E with & ~ _V( I) and data distribution p(X). represents the log of the Signal-to-Noise Ratio (SNR), and (-) is the sigmoid function. this notation, the probability that mixture distribution takes value, would be written pl Za =))
We start by re-writing the left-hand side new notation, and expand the definition based on the noisy channel model, using 6(-) for the Dirac delta.
VR logp(22 = x) = 1/pl Za = V
Ve)V,((2 ZZ = x)
1/p(Zaa = / dadde 6(x - Vola)=-V(-oo)e)((X E)p(E =)
1/p(Za - =x)/ / dade (V00 V =)p =
1/p(Za - x) / doode
- x)p(E - =))
Va(-a) / dāde 8(Φ 8(0 Volaa Vaai--Va--aee va(-a)))p(X x)V = =)
1
1/p(2. = x)
-1/p(Za x) Vo(-a) / dddde 6(x - =x)p(( E)
-11/((Za x) Vol-a) S de p(Z, =x, e)e
Va(-a) /
de PlE = = = x)
22 (6)
(o(-u)
E(xx a)
Va(-a)
In the second line expand, and in the third we just move the gradient inside the integral. In the fourth line we use the chain rule to relate the gradient over æ to the gradient over (introducing sign flip). In the fifth line we use integration by parts to move the gradient (second sign flip). Taking the gradient of the Gaussian in the sixth line gives our third sign flip, and the factor of e. We can conclude by writing the expression in terms of a conditional distribution, and relating that to the optimal denoiser in Eq. [
A.2 Mixture Distribution Density
In this section, we derive the expression that shows that the density of a continuum of Gaussian mixture distributions can be written in terms of the optimal denoiser, é, for the data distribution.
dā Epellle- o é( ((Vo(a) + vo(-aje, 3)13
- (x) c 1/2
(-B)
B = 0 -
(15)
13

---

As in the previous section, we will adopt information theory notation. If we define the optimal denoiser for the input distribution, Pr (xx, with subseript as cc(,,,,, we can write the density analogously to Eq. 2
- -logp; - c 1/2 { dā (acllle-e((Vo(())))
-
(16)
Note that we now have to keep track of two log SNR values. One indicates how much noise is added to the new "data" distribution, the other how much noise we add and then try to remove with our denoiser. The goal is to relate Eg to É. We can formally write down the optimal solution using the relation in Eq.
(< a) = Re-ppt [c]
Now, however, the noise channel defined differently. The channel mixes the signal, E ~ Pc(X), with Gaussian noise, E ~ (O, as Z = Vo + Vo(-ājE. And the noisy variable, X = Vo(()x+ + Va(- where we must be careful to distinguish the two independent sources of Gaussian noise.
We start by expanding definitions.
1
/ de P(E =6,= = é
(c( a+)
p(Z = xzx)
1
/ dédæ ö(xc - vo(a)a (ola)=-Vo(-ol vo(-aje p(X 3) plE =E) é
pl2 = xxs)
1
/ dédēdxde 6(0z Vol-)=- 8(
p(Z xxç)
pp(X= x) PlE <) =(E==EE
=EE
1
/ dedēdæ ō(0ç Vola\\oo(() vo(-()e) - -aiee--Vo((-o vo(-ā)e
plZ xx.)
p( = x)) P(E c) p(p Et)
Now do a change of variables, a 2-d rotation with:
é' = aE + be, = bē + ae,
a=
This change of variables leads to the following.
plZ = xx)) / dé' dé' dæ 6(æ; - (a -i-oo)
Ea (x)ç.
-p(X= = x) P(E' = =c') = (bé' +
I dé'dé' dæ 8(0; - Vo((S)o(a) -1-
b plz = xx))
.P(X x) plE' E') p(E' =
/ de' dé'dé' dæ 6(Øa Vo(B)x vI-o(BBe) p(X = x) p(-'= =E) p(E' E') è'
plz = xx))
(x a) = b)(
B B= (a=) (o0
Note in the second line that the expectation of e' is zero, and we move the constant for the other term, b, outside the integral. In the third line, we define 3 which represents the log SNR of the two consecutive noisy channels with Ç, a. Then we recognize the resulting integral as Eq. [ the optimal denoiser for recovering samples from from the original (non-noisy) data distribution in Gaussian noise.
A.3 Main Plot 1D Two-mode Gaussian's Analytical Solution
In this section, we calculate the analytical solution to the ID two-mode Gaussian in Fig.
14

---

is plotting the norm of difference between the ground-truth denoiser Ege(:,) and the estimated denoiser -(:
aanoiser_err(a:. xx) ||ē( a) Ege a))
To get this error plot, we need to analytically calculate ground-truth denoiser Ege(x, a). From score connection Eq. [ for any intermediate noisy density log Pa (x)). denoiser function é(x, a)) can be derived from score function V log Pa. (x). Therefore, ultimately what we need to calculate here is the score function of any noisy distribution (x)).
log
The data we used is mixture of two Gaussians, N(#= -5, 1) and N(u = 5, I), and the noise distribution is consists of data plus noise, then the noisy distribution should also be mixture of Guassians. We just need to relate the parameters of the noisy mixture of Gaussians to the parameters of the mixture of Gaussians.
Start with one mode of the Gaussian mixture the data. We could represent it in terms of the standard normal random variable, e.
Ed + de
Now call &, the random variable after applying noisy channel with log-SNR a, here we present sigmoid function as o(-).
T Vo(a)a + va(-a)e
Note that we use different e' here. Now expand this, and then re-arrange.
aa vala)(u Vo(-a)) vo(-a)e
= =o[aa)u+ vo(a)oe vo(-aje
We want represent this in canonical way what the variance and mean of this Gaussian is. Note that for two standard normal random variables, ae + be', we can represent them as a single random variable with the same variance, Va² + Pe" (reparameterization trick).
vo()ju volajo +o(-a)e"
Now we see that the noisy Gaussian (one component of a mixture) is just a modified version of the original. We have to change the mean (moving it towards zero when adding noise) and the variance. In our example, we set 1, so simplifies further.
Ca = Vo(a))++
So the variance doesn't 't change, we just slowly shift the two mixtures together to the center. Therefore, for one mode x ~ [(, I), the intermediate noisy log-density is.
log (x) 1/2 log(2mo²) (x
((()))²
2
Take the gradient of the log-density via torch built-in function aechauuurrd..grdllogg samples), we have the ground-trute score function Égt
B
Implementation Details
B.1 Synthetic Experiment -Maximum Mean Discrepancy Bandwidth Choice
The Maximum Mean Discrepancy (MMD) statistical test used to determine if two distributions are different. It works by comparing the mean embeddings of samples drawn from two distributions high dimensional feature space. Specifically, if the distributions are the same, means should be close; if they are different, the means should be far apart. The embeddings are typically constructed using a feature map associated with a kernel function, and here we select the Gaussian kernel:
(x. v) exp(
202
15

---

The bandwidth parameter of the Gaussian kernel plays critical role in the sensitivity and perfor- mance of MMD. The better bandwidth choice, the more effective MMD computation is. Intuitively. the bandwidth controls the scale at which differences between distributions are detected. A small a makes the kernel sensitive to differences at small scales (fine details), while large highlights differences at larger scales.
The choice of bandwidth is often related to the variance of the data, and the bandwidth should be on the order of the variance of the data. Through a small experiment where we calculate MMD score between our data and the standard Gaussian under various bandwidths øs, we pick the one that maximizes the MMD score.
In the synthetic 2D Dino experiment, plot the relationship between MMD scores and bandwidths (Fig. 8 setting a = 3e 02.
MMD between Data and Standard Gaussian
0.020
mmd values
0.030
bandwidth
Figure 8: MMD between Dino data and the standard Gaussian. We plots the relationship between the MMD values and the bandwidth parameter used the kernel function, and pick the bandwidth value with peak MMD score.
B.2 Details on Model Training
Model Checkpoints We adopt two models baselines to fine-tune with CDL: DDPM model provided by Ho et al. 2020 and EDM model provided by Karras et al. 2019
For DDPM model, the checkpoint we used is a ema one pre-trained on unconditional CIFAR-10. The reason we are not using the most frequently used checkpoint (https: lattps://hugingffet is that, this is not EMA checkpoint and our calculation of the FID score of this model on 50, 000 generated image via sequential DDPM sampler gives 12.43. This FID score is much higher than what reported on the original paper 3.17. Here we provide the FID scores of this non-EMA pre-trained model, results shown in Tab.
co/
google/ /dpm- aafffar0-3333
Parallel DDPM Sampler
Sequential DDPM Sampler
DDPM CDL
10.69
12.43
7.83
10.06
Table 4: Evaluating FID score for both parallel and sequential DDPM samplers. FID scores are calculated using 5, 0000 samples.
16

---

Dataset
Fine-tuning Configurations
uncond/cond CIFAR-10
-duration=0.5 -batch=128 -1r=2e-4
uncond AFHQ-64
-uration=O.. -batch=32 -1r=5e-5 -cres=1,2,,,2 dropout=0..2 25 -augment=0.15
uncond FFHQ-64
batch=32 -1r=5e-5 -cres=1,,,2 2,2,, -dropou=-0.0m augment=0.15
Table 5: Fine-tuning configurations for different datasets
For EDM model, in total eight checkpoints we used are "DDPM++ cont. (VP)" and *NCSN+ cont. (VE)" models pre-trained on three datasets (CIFAR-10 10, uncond-FFHO. and uncond-AFHQv2
15
10
3
with two training settings (unconditional and conditional) 10 14
train all models with the same training setting in their original papers. For DDPM model, we train each model for 10 epochs and keep 'learning rate batch size' ratio to be '10""/64', and this training is exact our fine-tuning configurations, which is still of the same 'learning rate/batch size" ratio. This training using eight V100 GPUs.
As for fine-tuning,
A6000 GPUs. For EDM model, the following table list the
More Experimental results with Parallel DDPM Sampler The previous parallel diffusion sampling paper (28) calculates FID scores by using 5, 000 generated images and another 5, 000 randomly selected real images, and to follow the same experimental setting for comparison, we further provide the FID results in T
CIFAR- 10 32x32
AFHQQ2 64x64
FFHQ 64x64
Models
unconditional
conditional
unconditional
unconditional
DDPM
9.43
NA
NA
NA
CDL-DDPM
9.06
NA
NA
NA
VP
7.93 + 0.07
7.67 + 0.07
4.58 + 0.07
6.26 + 0.07
CDL-VP
7.47 + 0.07
7.27
0.07
4.51 - 0.04
5.89 + 0.07
VE
7.81 + 0.07
7.59 + 0.07
4.65 = 0.10
6.33 + 0.07
CDL-VE
7.35 + 0.07
7.19 0.07
4.54
0.07
5.94 + 0.07
Table 6: Evaluating FID score (lower is better) of parallel DDPM sampler on real-world datasets using 5, 000 samples. "NA" stands for "Not Applicable". For reported FID scores, we run three sets of random seeds and reported the average with uncertainty.
More Details EDM Fine-tuning As the design choices of EDM model is very compre- hensive and complicate, here we list the training noise distribution, loss weighting, network and preconditioning choices make during CDL fine-tuning in Tab.
B.3 Relationship Among Log-SNR, Timesteps, and Noise variance Sigma
To use the pre-trained models the literature with our CDL loss, we need to translate "t" parameter
representing time in Markov chain that progressively adds noise to data in Ho et al. 2020 and [Song
al. 2020 or "a", the variance scale of the Gaussian noise Karras et al. 2022, to log-SNR "a"
ve. pkl
pkl
pk1

---

Network and preconditioning
Architecture of denoising function
(any)
Skip scaling Cakip (7)
o datal dataalo² + to data
Output scaling (o)
-atta V0 data 0²
Input scaling Gin (a)
1/ 0² + o data 1 In(o)
Noise Cond. Cnoise (0)
Training
Noise distribtion
In(o) (N Pa std)
Loss weighting X(o)
(a + 2
data ) )/(a dataa)
Table CDL finetune on EDM experiment -ine-tuning design choices.
Translation between Timesteps and Log-SNR For time-step DDPM and stochastic diffusion notation, we recommend readers check Kong al. 2023 Appendix B.2 about the mapping between and t.
Translation between Noise Variance Sigma and Log-SNR For variance scale of the Gaussian noise in EDM, referring to (7) and (8) in Karras et al. 2022 it's easy to translate the pre- conditioning:
& = Cn (xr + OE)
(17)
xa(a) o(a)
Ta Vo(a)x Vo(-o)e vo (ar
(18)
From Eq.
18 and Eq.
= 17 we see that
therefore, the relationship between and [a))
should be:
= exp(-a/2). a (2)((r)
B.4
Contrastive Loss Implementation
To implement contrastive loss, we follow the definition in Sec. First, we generate a random binary label y. Next, conditioned y, sample from either data distribution p(x) the noisy data distribution (2)) We calculate the point-wise log-likelihood, then the contrastive loss in Eq.
Algorithm 1 Contrastive Diffusion Loss - Training
1:
repeat
2:
Tp p(20)
3:
(~ Uniform(6..... 15)
4:
uniformly sample from p(x) or (<())
5:
if rand-prob < 0.5 then
6:
=
7:
=1 else
8:
9:
generate moctttteeituuet(eo_e {)
y y=
11:
end if
calculate negative log- likellhoo of p(x)).<(())
13:
log_px -n)((x)
14:
log_px_zeta -nll(a, 5
15:
cdl_loss = softmax(y- (log_px_zee) log_px))
16:
until converged
18

---

B.5
Training cost
As we mentioned, CDL training more expensive to compute than the standard diffusion loss, and here we analysis and give the reason.
According to Kong et al. 2023 and therefore the standard diftusion loss is as Eq. 20
write the pointwise standard diffusion loss function Eq. 19
nll(x) logp(z)) = 1/2
Exte)lle -((xn,, a) da.
(19)
00
nll = pp(p) log p(x) = c 1/2
Ep(c) ( É(Tx. a)l da.
(20)
To train the standard diffusion loss, we simply need to optimize Eq. 20 by all the training data. However, to train the contrastive diffusion loss, we need to estimate nll and nll( () term in Algo. [ and there we estimate Eq. by duplicating a single data point x for N = 100 times and
calculate Eq. 20 This pointwise NLL estimation nll( demands N 100 times computational resources compared to the standard diffusion loss.
In principle, the contrastive loss Algo. should be executed for the entire training dataset. However, due to the high computational cost, we optimize only one data point per batch instead of utilizing all the training data.
C Samples Visualization
We provide visualization of the images generated from pre-trained models fine-tuned via CDL loss.
Figure 9: The CDL-loss fine-tuned EDM checkpoint generated examples from Conditional CIFAR-10, via parallel DDPM sampler.
19

---

Figure 10: The CDL-losss fine-tuned EDM checkpoint generated examples from Unconditional CIFAR-10, 10, via parallel DDPM sampler.
Figure 11: The CDL.-loss fine-tuned EDM checkpoint generated examples from Unconditional AFHQ, via parallel DDPM sampler.
20

---

Figure 12: The CDL- fine-tuned EDM checkpoint generated examples from Unconditional FFHQ via parallel DDPM sampler.
Figure 13: The CDL-loss fine-tuned EDM checkpoint generated examples from Conditional CIFAR- 10, via sequential EDM sampler.
21

---

Figure 14: The CDL-loss fine-tuned EDM checkpoint generated examples from Unconditional CIFAR-10, 10, via sequential EDM sampler.
Figure 15: The CDL--loss fine-tuned EDM checkpoint generated examples from Unconditional AFHQ, via sequential EDM sampler.
22

---

Figure 16: The CDL-loss fine-tuned EDM checkpoint generated examples from Unconditional FFHQ. via sequential EDM sampler.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We explain the proposed CDL speeds up parallel sampling convergence and improves generation quality. For sequential sampling, it improves the trade-off between pling quality and gererattion speed. We provide the claims in both abstract and introduction, and justify them in the Results section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The main limitation to our approach is the increased training cost. We explicitly highlight limitations in the methods development in Sec.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
24

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Every theoretical result is supported by complete proof and assumptions as well, if applicable.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide anonymous code link and full description of training configuration.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
25

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide anonymous code link and full description of training configuration.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips. fttaps.cccpublice wwtppbblccgguueeeeeoooooopppppee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all detailds about training and testing details in both experimental section and also in appendix. Since we test on benchmark, our data split are rigously the same all other baseline settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report error bars wherever possible. One set of error bars for Table missing due to lack of computing resources, but will be added in the final version.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computation resources, c.g. GPU information, to reporduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:///n..i... wwwerrssscccpppp//eeeeee cc/publlc/Ethiccuuueeell
Answer: [Yes]
We do not use human subjects and only used standard datasets for experiments. We acknowledge the potential harms of diffusion models in the broader impacts statement, and do not anticipate that our methodological contributions will alter known issues. We also strive for transparency wherever possible, including full, anonymized code with our submission.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
27

---

While we do not expect a direct societal impact from our methodological improvements, we added a broader impacts statement that acknowledges the potential impact of diffusion models.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We used existing standard datasets and public, pretrained models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We give proper attribution for the open license code, data, and models used throughout the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
28

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are provided.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or human subjects research is included.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
29

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "CE7CMBOVrr", "reviewer": "Reviewer_Y79y", "summary": "Building on the denoising score matching loss, the paper introduces a novel contrastive learning loss. This loss estimates the log-likelihood ratio between mixture densities with varying noise levels. The contrastive learning loss is then employed to fine-tune the diffusion model, enhancing its ability to estimate out-of-distribution (OOD) data. To validate this approach, the paper demonstrates experimentally that the fine-tuned diffusion model exhibits improved performance in both sequential and parallel sampling, which typically degrade with poor OOD estimates.", "questions": "1. I think there are issues with the chosen baseline in the paper. In the original parallel sampling method [27], parallel sampling achieves similar performance to sequential sampling in significantly less time. However, the results reported in Table 2 show that unconditional parallel sampling performs much worse than sequential sampling (around 3.0 for the original DDPM, VP, and VE). Could the author provide more clarification on this discrepancy?\n\n2. According to [1], the denoising score matching loss serves as an upper bound for the negative log-likelihood. So minimizing the denoising score matching loss is equivalent to maximizing the negative log-likelihood.  Are there any insights or proofs provided on why minimizing the CDL would also lead to meaningful generation?\n\n\n[1] Song, Yang, Conor Durkan, Iain Murray, and Stefano Ermon. \"Maximum likelihood training of score-based diffusion models.\" Advances in neural information processing systems 34 (2021): 1415-1428.", "limitations": "The authors have covered the limitations of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The derivation of the contrastive diffusion loss is solid and novel. The CDL could potentially inspire more contrastive loss and noise-level data augmentation methods for diffusion model training. Additionally, CDL introduced the asynchronous data pair $(z_\\alpha, \\beta)$ for training, improving the robustness of the out-of-distribution region, especially during sampling. Overall, the idea is very sound and novel.", "weaknesses": "The experimental results are not particularly impressive:\n\n1. Although CDL fine-tuning consistently enhances parallel sampling performance on both synthetic and real-world datasets, the improvements are marginal. Additionally, the baseline chosen for comparison is not state-of-the-art. For instance, in the unconditional generation in Table 2, even with CDL fine-tuning, an FID of around 7.0 on CIFAR-10 falls short of the state-of-the-art FID, which is less than 2.0. The improvement in sequential generation is even less substantial. \n\n2. I believe the proposed CDL has significant potential. For instance, when addressing inverse problems, the diffusion model also encounters OOD issues. The paper could benefit from including more promising experiments to demonstrate how CDL enhances the diffusion model's OOD estimation capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for taking the time to read and review our paper. Please see our top-level comments for clarification on FID scores, where we show that we consistently produce SOTA results across a wide variety of settings on this metric. \n\nWe address the two weaknesses pointed out by the reviewer below. \n\n1. FID Score Discrepancy \n\nThe original Table 2 reported FID scores using 5k samples, as per the settings of our baseline paper [4]. FID scores depend on the number of samples, so that our previous results were not directly comparable to other results reported using 50k samples. In response to your feedback, we have updated Table 2 to include FID scores calculated with 50k samples, results are in the top-level rebuttal response. \n\nThe updated results show that our results are actually very good and in line with SOTAs. With the same parallel sampler, CDL-finetuned models consistently outperforms the models trained by the original diffusion loss (DDPM, VP, and VE losses). For example, compared to all other baseline losses under FID calculated with 50k samples, cdl achieves an FID improvement of ~0.7 on unconditional CIFAR10, ~0.5 on conditional CIFAR10.\n\n\n2. Minimizing CDL and Meaningful Generation\n\n> Are there any insights or proofs provided on why minimizing the CDL would also lead to meaningful generation?\n\nOur method is based on Density Ratio Estimation (DRE) rather than maximum log-likelihood methods. DRE techniques transform the task of learning data distributions into learning to classify between data samples and samples from a reference distribution [1,2,3]. Our approach bridges diffusion-based generative methods with density ratio estimation. Therefore, minimizing CDL effectively learns the underlying data distribution, which leads to meaningful generation. \n\nFor more detailed information about DRE, we refer to [2] section 2.1 Density Estimation by Comparison. \n\n[1] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012\n\n[2] Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of machine learning research, 13(2), 2012\n\n[3] Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Advances in neural information processing systems, 33:4905–4916, 2020.\t\n\n[4] Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eAXwnGD0tR", "reviewer": "Reviewer_M92h", "summary": "The paper establishes a connection between the diffusion model denoiser and noise classifier through an examination of log-likelihood ratio estimation. The authors introduce a novel loss function, termed Contrastive Diffusion Loss (CDL), designed to encourage diffusion models to explore OOD regions in noise levels. Empirical results indicate that the proposed loss function enhances performance in parallel sampling and demonstrates robustness to different hyperparameters in sequential sampling.", "questions": "Why did the authors choose fine-tuning with the CDL over pre-training and using CDL as a regularization?", "limitations": "The authors addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper draws an interesting connection between diffusion model denoiser and noise classifier and introduces a novel loss function based upon. \n* The paper is well-organized and most results are clearly presented.", "weaknesses": "* The clarity of the paper could be further enhanced. For instance, in line 86, the authors introduce the notion $\\zeta$ to indicate the noise level, while in Appendix A.2, they use $\\alpha$ during the derivation. It seems like the authors want to differentiate between noise level and SNR, but the message is not entirely clear here.\n\n* The empirical section of the paper is not as strong as the analytical section. In the parallel sampling example, the method shows good performance, but even with the performance improvement, the results are still far from optimal compared to EDM with the default settings. The experiment on deterministic samplers aims to demonstrate that CDL-regularized models maintain a more stable FID score with changes in NFE, but the authors do not explain why CDL helps in this context. The same issue applies to the experiment on stochastic samplers.\n\nOverall, I am on the positive side for this paper. I would appreciate it if the authors could provide more intuitive explanations to better connect the analytical and empirical sections of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for taking the time to read and review our paper. We’re glad that you are positive about our paper. Please see our top-level comments for clarification on FID scores, where we show that we consistently produce SOTA results across a wide variety of settings on this metric. \n\nWe address the two weaknesses and two questions pointed out by the reviewer below. \n1. Clarity on Notation\n\n> The clarity of the paper could be further enhanced\n\nTo clarify the distinction between the log-SNR used in the integral ($\\alpha$ in Eq. 3) and the density of the noisy data distribution, which is a mixture of data and noise, we used $\\zeta$ to represent the amount of noise in the noisy distribution. We acknowledge that $\\alpha$ was used in Appendix A.2 due to a typographical error. This will be corrected in the revised manuscript.\n\n2. Empirical Results FID scores\n\nThe difference in FID scores compared to EDM's default settings is because the original Table 2 reports FID scores using 5k samples, as per the baseline paper, whereas SOTA methods typically use 50k samples. \n\n> In the parallel sampling example, the method shows good performance, but even with the performance improvement, the results are still far from optimal compared to EDM with the default settings.\n\nTable 2 reports FID scores for samples generated using parallel samplers from EDM-pretrained checkpoints, while EDM's default FID scores are for samples from sequential EDM samplers. This difference in sampling methods explains the difference in FID scores.\n\nWe have updated Table 2 to include FID scores calculated with 50k samples. For example, compared to all other baseline losses, cdl achieves an FID of 2.38 (0.62 better than the best baselines with an FID of 3.00) for unconditional CIFAR10. These updates are included in the top-level comments. \n\n3. Why CDL Helps with Stable FID Scores Across Hyperparameters?\n\nAccording to EDM [1], samplers introduce local discretization errors at each step, which accumulate as global errors. Although sequential samplers are designed to stay close to the forward / training paths $(z_\\alpha, \\alpha)$, local errors can deviate $z_\\alpha$ from these paths. CDL, by training on asynchronous data pairs $(z_\\alpha, \\beta)$, allows the model to see deviations (e.g., $z_\\alpha'$) during training and thus corrects for these local errors. This \"correction\" capability contributes to stability across hyperparameters in both deterministic and stochastic sampling settings. \nMore generally, our approach can be viewed as noise contrastive density ratio estimation, an alternative approach to maximum likelihood for density estimation (Gutmann & Hyvarinen, 2012) [2]. By combining maximum likelihood and density ratio based estimates, we improve over either one individually. \n\n4. Why Fine-Tuning Instead of Pretraining?\n\nFine-tuning on pretrained models is preferred because training diffusion models from scratch is computationally expensive. It is common practice to fine-tune pretrained models to balance efficiency and performance. Results on toy datasets were trained from scratch, with excellent results. \n\n[1] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364, 2022. \n\n[2] Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of machine learning research, 13(2), 2012", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "juJrfqmR1w", "reviewer": "Reviewer_XUix", "summary": "The paper presents a method to improve the parallel sampling of the diffusion model by improving the denoising network for out-of-distribution evaluation. The paper proposes to finetune a trained model using the log-likelihood ratio of a sample at two different noise scales. The log-likelihood ratio is obtained by integrating the denoising error at all noise scales. The proposed method performs better for parallel sampling and performs comparable for sequential sampling on the CIFAR-10, AFHQ, and FFHQ datasets.", "questions": "- It is not clear why only 5k samples are used for calculating FID for parallel sampling. The standard practice is to use 50k samples. As the FID depends on the number of samples, it is hard to make sense of the significance of improvement. \n- Can you experimentally show/quantify the improvement of the denoiser for out-of-distribution evaluation when trained using contrastive diffusion loss (CDL)?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is overall well-written and easy to follow. \n- As the contrastive diffusion loss is expensive to compute, the authors show that finetuning on pretrained model is sufficient for few epochs instead of training using contrastive diffusion loss (CDL) (or linear combination of CDL and denoising error) through out the training.", "weaknesses": "- The contrastive diffusion loss (CDL) contains the integration of denoising error over all noise scales. In practice, I assume that the integration will be replaced by a numerical integration therefore, evaluating the proposed CDL loss is expensive. Considering the training/finetuning using CDL loss is expensive, the benefits from the training don’t seem to be significant in the empirical performance. Additionally, very few/no details are provided on how the integration is approximated in experiments/practice.  \n- Some discussions about the related works need to be included. For example, another work on improving parallel [1] is not mentioned/discussed/compared against. Additionally, the derivation of the CDL loss is somewhat similar to some diffusion classifier papers (for example, [2, 3, 4]) therefore they should be cited/discussed. \n\n[1] Accelerating Parallel Sampling of Diffusion Models\n\n[2] Your Diffusion Model is Secretly a Zero-Shot Classifier\n\n[3] Your Diffusion Model is Secretly a Certifiably Robust Classifier\n\n[4] Robust Classification via a Single Diffusion Model", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please see our top-level comments for clarification on FID scores, where we show that we consistently produce SOTA results across a wide variety of settings on this metric. \n\nWe address the two weaknesses pointed out by the reviewer below. \n\n\n1. Training cost \nThe numerical integration in our loss is done using importance sampling. Details of this standard approach are discussed in many papers, e.g., in Kingma et al, Variational Diffusion Models, NeurIPS 2021. As pointed out by Kingma et al, the standard diffusion loss can also be interpreted as an evaluation of a weighted integral evaluated with importance sampling, so this is not unique to our method. \n\nHowever, it is true that the form of our loss does incur additional training cost, and we pointed this out explicitly as a limitation. \n\n- Trading off extra computation at training time for faster / better quality sampling at inference time (as we do) is considered advantageous for most applications. For instance, the entire area of score distillation methods train two models, rather than one, but are nonetheless popular in practice.  \n- The reviewer guidelines state “authors should be rewarded rather than punished for being up front about the limitations of their work”. We have tried to honor this guideline in our own reviews, and ask the reviewer to consider taking this guideline into consideration when deciding on a final score. \n\n2. Related work \n - The references [2,3,4] from XUix’s review also have “diffusion” and “classification” in the titles. However, these methods all consider classifying objects in images (e.g. “cat”/”dog”). They rely on the longstanding observation that conditional generative models can be used as classifiers via Bayes rule. In contrast, we introduce an interpretation of diffusion models as *noise classifiers* - a classifier that can distinguish the amount of noise added to an image. The derivation of these methods is nontrivial and totally different as it goes through I-MMSE results rather than Bayes rule. While the task of classifying noise may be unfamiliar to some, it comes from a classic line of work started by Gutmann & Hyvarinen on noise contrastive estimation (JMLR 2012) as an alternative foundation for machine learning (see response to Y79y for more detail). We will add a discussion of these papers and the distinction between noise classification and traditional classification with diffusion to related work. \n - Reference [1], like most attempts to accelerate diffusion, relies on changes to the sampler. Our approach, on the other hand, changes the training procedure and can be combined with any sampler. We tested our approach with a representative sample of deterministic and stochastic samplers, and sequential and parallel samplers. We will add this reference to the list of other recent sampling approaches that our method could be combined with. \n\n> Can you experimentally show/quantify the improvement of the denoiser for out-of-distribution evaluation when trained using contrastive diffusion loss (CDL)?\n\nYes, a direct metric is to look at the score matching loss (L2 distance between true and estimated score) for OOD points, we will add this to the supplementary material. This requires access to a ground truth denoiser, so we can only show it for simple cases like the one shown in Fig. 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful for the time reviewers have invested in reviewing our paper and for the insightful feedback provided. In this top-level comment, we will explain concerns about FID results and baseline comparisons, as this was the subject of discussion among the reviewers. In particular, our choice of baseline metric *obscured* the fact that our method produces competitive results with SOTA results. We will respond to other specific points in comments replying to each reviewer. We are posting comments now to give time for discussion, with a revised version coming in a few days.

We have carefully considered your comments and made the following updates:

The FID scores reported in original Table 2 were calculated using 5k samples, following the settings of our baseline paper [1]. This choice was made to maintain consistency with the baseline. 
Additionally, previously we didn't have an appropriate amount of computational power to run a higher number of samples for FID scores.  FID scores depend on the number of samples, so that our results were not directly comparable to other results reported using 50k samples.

We have updated Table 2 with FID scores calculated using 50k samples. Our best results of FID on all datasets are comparable with SOTA results of FID using 50k samples reported in [DDPM, VP, VE, EDM]. 

|                            |      Cond CIFAR10                        ||                      Uncond CIFAR10      ||
|:------------------:|:----------------:|:----------------:|:----------------------:|:----------------------:|
|                    |           VP           |           VE           |           VP           |           VE           |
|   **EDM loss [baselines]**    |          2.93          |          2.76          |          3.24          |          3.00          |
| **CDL loss[ours]**|          **2.41**          |          **2.25**          |          **2.51**          |          **2.38**          |

Table 2: Parallel sampler results from EDM pretrained checkpoints, FID score evaluated by 50k samples. 

Below are a summary of changes in the revised version of the PDF and supplementary material.

 - Update Table 2 to show FID calculated with 50k samples, where CDL continuously outperforms baselines and produces near SOTA FID scores. All the results will be replaced to FIDs calculated with 50k samples, and FIDs with 5k will be reported in appendix for direct comparison to that other paper. 
E.g. unconditional CIFAR10 dataset, with the same sampler, cdl-finetuned model achieves 2.51 FID, whereas edm-loss trained model achieves 3.23 FID, in total cdl improves ~0.7 FID score.
 - Will have wording changes and clarifications throughout to address reviewer comments
 - Will add details about implementation of numerical integration in our loss 
 - Will add a discussion of classification diffusion papers, and the distinction between noise classification and traditional classification diffusion 
 - Will add a direct metric – L2 loss between true and estimated denoiser (score matching loss), which quantify the score / denoiser improvements on OOD points

We have taken great care to address all the concerns raised and hope that our revisions will meet your approval.

Meta-review: Building on the denoising score matching loss, the paper presents a new contrastive learning loss. This loss estimates the log-likelihood ratio between mixture densities with different noise levels. The contrastive learning loss is subsequently used to fine-tune the diffusion model, improving its capability to estimate out-of-distribution (OOD) data. Experimental results show that this approach leads to enhanced performance of the fine-tuned diffusion model in both sequential and parallel sampling, areas that often suffer from poor OOD estimates.

Most reviewers find the paper well-presented and well-written, and the paper draws an interesting connection between diffusion model denoiser and noise classifier and introduces a novel contrastive loss, and the derivation is solid and insightful. 
In the camera-ready version, we suggest the authors take the reviewers' feedback and revise accordingly, such as a more comprehensive discussion with related works and a more convincing evaluation of the proposed methods.
Decision: Accept (poster)