Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training

Authors: yunshu  wu
yingtao  luo
xianghao  kong
evangelos e papalexakis
greg ver steeg
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:56:28

Abstract: Diffusion models learn to denoise data and the trained denoiser is then used to generate new samples from the data distribution. 
In this paper, we revisit the diffusion sampling process and identify a fundamental cause of sample quality degradation: the denoiser is poorly estimated in regions that are far Outside Of the training Distribution (OOD), and the sampling process inevitably evaluates in these OOD regions.
This can become problematic for all sampling methods, especially when we move to parallel sampling which requires us to initialize and update the entire sample trajectory of dynamics in parallel, leading to many OOD evaluations. 
To address this problem, we introduce a new self-supervised training objective that differentiates the levels of noise added to a sample, leading to improved OOD denoising performance. The approach is based on our observation that diffusion models implicitly define a log-likelihood ratio that distinguishes distributions with different amounts of noise, and this expression depends on denoiser performance outside the standard training distribution.
We show by diverse experiments that the proposed contrastive diffusion training is effective for both sequential and parallel settings, and it improves the performance and speed of parallel samplers significantly. Code for our paper can be found at https://github.com/yunshuwu/ContrastiveDiffusionLoss

Main paper: 
 introduction: 
 Denoising diffusion models [29] achieve state-of-the-art performance on various unsupervised learning tasks and have intriguing theoretical connections to methods like denoising autoencoders [37] , VAEs [6] , stochastic differential equations [20, 34] , information theory [13] , and score matching [31, 32] . Diffusion models are presented with data samples corrupted by a forward dynamical process that progressively adds more Gaussian noise and trained to reverse this dynamics or denoise the corrupted samples. Samples are then generated by applying the reverse dynamics on images of pure Gaussian noise to produce high-quality samples from the target distribution.The key to the success of diffusion models is the dynamics that gradually bridges the source and a target distribution, but it suffers from slow sampling, as sequentially simulating these dynamics can take thousands of denoising steps for one sample. Most recent works attempt to expedite the sequential dynamics by taking fewer, larger steps [30, 11, 18, 22] . However, the complexity of these samplers and the need for expensive sampling hyper-parameter grid searches tailored to specific datasets makes them difficult to generalize. Shih et al. 2024 suggests a different approach by randomly initializing the entire path of the reverse dynamics and then updating all the steps in the path in parallel. The parallel sampling approach promises to drastically reduce wall-clock time at the cost of increased parallel computation. However, it encounters a problem that has largely gone unnoticed in the sequential sampling literature: while Figure 1 : We plot the error in the score estimate for an 1D two mode Gaussian example where diffusion dynamics bridge between a Gaussian and a mixture (see Appendix A.3). Regions near the standard forward training data paths have lower error magnitude (light), whereas other areas have higher error magnitude (dark). While sequential samplers adhere as closely as possible to low-error regions, parallel samplers initialize and update the entire sample trajectory (blue trajectories), leading to evaluations in high-error regions. When the sampling trajectory is initialized, most are inevitably in the OOD regions and will update to the low-error regions gradually. sequential paths sampled during generation are designed carefully to stay as close as possible to the forward paths that add noise to data, the parallel sampler often evaluates in regions far from where the score estimate (the denoiser) is trained, as illustrated in Fig. 1 . The shading shows that the error of the score estimate is large in these regions, leading to poor performance for parallel samplers. Discretization errors can lead to similar issues, even for standard sequential samplers.We propose to improve the training of diffusion models so that the error of the denoiser is reduced in OOD regions, and we hypothesize that this should significantly improve the performance for parallel samplers as they require more OOD evaluations. Our approach starts with an unexpected connection: the optimal MSE denoiser that defines the diffusion dynamics also defines an optimal noise classifier that distinguishes between samples with different amounts of noise. This provides a useful additional signal for training, because optimizing for the noise classification task involves evaluating the denoiser for one noise level on samples from distributions at different noise levels, while standard MSE optimization only evaluates the denoiser on samples from the matching noisy distribution. Accurate denoiser evaluation in regions that are OOD for standard diffusion training is important for robust sampling dynamics.• We use the information-theoretic formulation of diffusion to draw connections between diffusion, log-likelihood ratio estimation, and classification. This reveals that optimal diffusion denoisers are also implicitly optimal classifiers for predicting the amount of noise added to an image. • We leverage the noise classifier (via density ratio estimation [4] ) interpretation to introduce a novel self-supervised loss function for regularizing diffusion model training, which we call the Contrastive Diffusion Loss (CDL). CDL provides training signal in regions that are OOD for the standard MSE diffusion loss. • We show that CDL improves the trade-off between generation speed and sample quality, and that this advantage is consistent across different models, hyper-parameters, and sampling schemes. The improvement is especially substantial for parallel diffusion samplers [28] which rely heavily on OOD denoiser evaluations.unknown data distribution for x ∈ R d , and σ(•) is the sigmoid function. We define the sequence of intermediate distributions drawn according to this channel with a subscript as p α (x). By definition, we express lim α→∞ p α (x) = p(x) in this paper. Note that we use a different scaling convention for noise from [11] and [6] , where the former one takes x + σϵ as the forward noising channel and the latter one takes √ α t x + √ 1α t ϵ as the forward noising channel. For further detailed relationships among these scaling conventions, please check App. B.3.The minimum mean square error (MMSE) estimator ε for recovering ϵ from the noisy channel that mixes x and ϵ can be derived via variational calculus and written as follows.EQUATIONSampling from the true posterior is typically intractable, but by using a neural network to approximate the solution to the regression optimization problem, we can get an approximation for ε. From [13] , we see that log-likelihood can be written exactly in terms of an expression that depends only on the MMSE solution to the Gaussian denoising problem, i.e.log p(xEQUATIONThe constant,c = d/2 log(2πe) -d 2 ∞0 dᾱ σ(ᾱ) does not depend on data and will play no role in our approach, as it cancels out in our derivations in Sec. 3. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The main limitation to our approach is the increased training cost. We explicitly highlight limitations in the methods development in Sec. 3.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
what your diffusion model is hiding: noise classifiers: 
 We now introduce our first main result, which shows that diffusion models implicitly define optimal noise classifiers. Eq. ( 2) expresses the probability density of the data directly in terms of the denoising function. If we apply Eq. ( 2) to the noisy distributions that bridge the data and a Gaussian, p ζ (x), we can see that all mixture densities can be written in terms of the same optimal denoising function, ε(•, •). The complete derivation is presented in App. A.2.EQUATIONx α ≡ σ(α)x + σ(-α)ϵ (4)EQUATIONIntuitively, if we find the optimal denoising function for the data distribution, it may be hypothesized that it can denoise an already noisy version of the data distribution. Using Eq. 2, this directly translates into an expression for density of mixture distributions. Differences in log likelihoods lead to cancellation of constants, and these Log Likelihood Ratios (LLR) are related to the optimal classifiers [4] as we show below.To connect LLRs with classification, consider the following generative model. We generate a random binary label q(y = ±1) = 1/2. Then, conditioned on y, we sample from some distribution q(x|y).Given samples (x, y) ∼ q(x, y) = q(x|y)q(y), the Bayes optimal classifier is: q(y|x) = q(x|y)q(y) q(x) = q(x|y)q(y) q(x|y = 1)q(y = 1) + q(x|y = -1)q(y = -1)EQUATIONIn the second line, because ∀y, q(y) = 1/2, these constants cancel out. Then we can just expand definitions and re-arrange to write in terms of log probabilities.Contrastive Diffusion Loss (CDL) Our next contribution is to use the new connection between diffusion denoisers and noise classifiers to define a new training objective. We set the distributions q(x|y = 1) and q(x|y = -1) to be two distributions at different noise levels that we can write in terms of the optimal diffusion denoiser from Eq. 3. So we have q(x|y = 1) ≡ p(x), the data distribution, and q(x|y = -1) ≡ p ζ (x), for some noise level, ζ. Then given a sample (x, y) ∼ q(x, y) the per-sample cross-entropy loss for the noise classifier, Eq. ( 6), is as follows.EQUATIONWe can estimate both densities directly from our denoising model using Eq. ( 3), with the constants canceling out in the process. This loss differs significantly from the standard diffusion loss. Intuitively, to distinguish between a sample from the data distribution, p( Choice of noise to contrast Next, let's break the Log-Likelihood Ratio (LLR) term in Eq. ( 7) down to see how to choose ζ to maximize the benefit of CDL. Combining Eq. ( 2) and Eq. ( 3) we have Eq. ( 8), where the constant cancels out.LLR = log p ζ (x) -log p(x) = ∞ -∞ dα E p(ϵ) [∥ϵ -ε(z, α)∥ 2 2 ] -E p(ϵ) [∥ϵ -bε(z, β)∥ 2 2 ] (8) with: z ≡ σ(α)x + σ(-α)ϵNote that the input x to the LLR term may come from two different distributions, which breaks the standard synchronous denoising pair (x α , α) into asynchronous. When it's from data distribution x ∼ p(x), z = z α ; and when it's from some noisy data distribution x ∼ p ζ (x), z = z β . From Eq. ( 8) we see that ε(•, •) is trained on four pairs: (z α , α), (z β , β), (z β , α) and (z α , β), where 5) . During standard training, only the first two pairs are trained (Eq. 1). This means that our CDL objective trains the denoiser to perform correctly even for samples from distributions that are noisier or cleaner than the specified noise level (a pair like (z β , α) or (z α , β)). This can be useful for both sequential and parallel sampling settings. During sequential sampling, extra error noise added due to discretization errors can be corrected by the denoiser trained with CDL. As for parallel sampling, CDL helps with evaluations on asymmetric pairs (z β , α) or (z α , β) which we refer to OOD regions for standard diffusion loss.β ≡ σ -1 (σ(α)σ(ζ)) < min(α, ζ) (Eq.In practice, diffusion training pipelines are highly tuned on popular datasets like CIFAR10 and ImageNet, so the amplitude of discretization errors during sampling is small, meaning that errors won't nudge points too far away from the true trajectory. Therefore, when evaluating CDL objective, we sample some large-valued ζs, which corresponds to classifying only small differences in noise levels. Empirically we find that ζ ∼ Uniform [6, 15] or ζ ∼ logistic [6, 15] performed equally good.Denoising, sampling dynamics, and the score connection We have focused so far on denoising and density estimation, but we now want to connect this discussion to the primary use case for diffusion models and the focus of Sec. 4, sampling. There are many choices in how to implement sampling dynamics [11] , but all of them rely on the score function, ∇ x log p α (x). The score function points toward regions of space with high likelihood, and by slowly transitioning (or annealing), from the score function of a noisy distribution to one closer to the data distribution, we can build reliable sampling dynamics. To connect denoisers with sampling we must show that a denoising function, ε, that is optimal according to Eq. 1 also specifies the score function.EQUATIONThe derivation is straightforward and is given in Appendix A.1.Figure 2 : The computation graph of Picard iteration for parallel sampling [28] Parallel Sampling Shih et al. 2024 explores a parallel sampling scheme, where the entire reverse process path is randomly initialized and then all steps in the path are updated in parallel. Parallel sampling is based on the method of Picard iteration, an old technique for solving ODEs through fixed-point iteration. An ODE is defined by a drift function s(x, t) and initial value x 0 . In the integral form, the value at time t can be written asx t = x 0 + t 0 s(x u , u) duIn other words, the value at time t must be initial value plus the integral of the derivative along the path of the solution. This formula suggests a natural way of solving the ODE by starting with a guess of the solution {x k+1 t : 0 ≤ t ≤ 1} at initial iteration k = 0, and iteratively refining by updating the value at every time t until convergence2 (Picard Iteration)EQUATIONTo perform Picard iterations numerically, which is shown in Fig. 2 , we can write the discretized form of Eq. 12 with step size 1/T , for t ∈ [0, T ]:EQUATIONWe see that the expensive computations {s(x k i , i/T ) : i ∈ [0, T )} can be performed in parallel. After some number of Picard iterations, the error difference between two iterates ∥x k+1 t x k t ∥ 2 2 drops below some convergence threshold. This converged trajectory, x * t , should be close to the sequential sampler trajectory. Looking at the example in Fig. 1 , we show the trajectories of three iterations k = 1, 2, 3. The trajectories before convergence are consistently appearing in the regions with high score error.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide sufficient information on the computation resources, e.g. GPU information, to reporduce the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
sequential and parallel sampling with diffusion models: 
 Sampling dynamics are typically presented in terms of a stochastic process {x t } T t=1 with timestep, t, rather than in terms of log SNR, α. We will denote x t ≡ x α(t) , p t (x) ≡ p α(t) (x), to connect to our previous notation, with α(t) representing a monotonic relationship described in App. B.3. Note that decreasing log-SNR α corresponds to increase timestep t, since smaller log-SNR means there is more noise added to the data.The general form of sampling dynamics is a process of slowly transitioning from samples of a simple and tractable distribution to the target distribution. Specifically, start with an isotropic Gaussian x T ∼ N (0, I), the sampler steps through a series of intermediate distributions with noise levels {T, T -1, . . . , 1} following the score estimates. Many works [34, 11] interpret diffusion models as stochastic differential equations (SDEs). The forward process is in the form ofEQUATIONwhere w t is the standard Wiener process/Brownian motion, f and g are drift coefficient and diffusion coefficient of x t separately. The reverse process of Eq. 10 is then used to generate samplesEQUATIONDepending on choices for f, g, we can get either a stochastic or ordinary (deterministic) differential equation. Either way, numerical differential equation solvers are used to approximate the true dynamics. The solver introduces discretization errors at each step, causing the trajectory to deviate into the OOD region where the score (or denoiser) is poorly estimated, further compounding the errors. More discretization steps reduce accumulated error and leads to better sample quality, at the expense of more sequential computation. As a result, a significant limitation of diffusion models is that they require many iterations to produce high quality samples.The influential diffusion sampler DDPM [6] iterates over thousands of discretization steps in simulating the dynamics. Recently, many sequential sampling methods have been developed to take fewer and larger steps while introducing less error [30, 22, 11] . Specifically, Karras et al. 2022 studies the curvature shape of SDE/ODE trajectory and suggests a discretization technique where the resulting tangent of the solution trajectory always points towards the denoiser output. However, speeding up the sequential sampling sacrifices generation quality. Furthermore, the SOTA sequential samplers [18, 11, 30] require hyperparameter tuning and grid search on specific datasets, which poses challenges to the generalization of these samplers to other datasets.x 0 x 1 x 2 x T . . . Examining the iterative update rule in Eq. ( 5), we see that an update at time t depends on all previous timesteps instead of just the previous timestep t 1. This amounts to introducing skip dependencies in the computation graph (Fig. 2 ), which enables information to propagate quickly down the chain and accelerate sampling.x k 0 x k 1 x k 2 x k T x k+1 0 x k+1 1 x k+1 2 x k+1 T . . . . . .The key property of interest is that each Picard iteration can be parallelized by performing the expensive computations {s(x k i , i T ) : i 2 [0, T )} in parallel and then, with negligible cost, collecting their outputs into prefix sums. Given enough parallel processing power, the sampling time then scales with the number of iterations K until convergence, instead of the number of denoising steps T .The number of iterations until convergence depends on the drift function s. More concretely, sequential evaluation can be written as a nested evaluation of functions x ? t+1 = h t (. . . h 2 (h 1 (x 0 ))) on the initial value x 0 where h i (x) = x + s(x, i/T )/T . If, for all timesteps, the drift at the true solution can be accurately obtained using the drift at the current guess, then the parallel evaluation will converge in one step.Proposition 1. (Proof in Appendix A) s(x k i , i/T ) = s(h i 1 (. . . h 2 (h 1 (x 0 ))), i/T ) 8i  t =) x k+1 t+1 = x ? t+1It is also easy to see that even in the worst case, exact convergence happens in K  T iterations since the first k points x 0:k must equal the sequential solution x ? 0:k after k iterations. In practice, the number of iterations until (approximate) convergence is typically much smaller than T , leading to a large empirical speedup.The idea of Picard iterations is powerful because it enables the parallelization of denoising steps. Remarkably, Picard iterations are also fully compatible with prior methods for reducing the number of denoising steps. Recall that the drift term s(x t , t/T )/T can be written as h t (x t ) x t and approximated using Euler discretization as p ✓ (x t+1 | x t ) x t , but it can also be readily approximated using higher-order methods on p ✓ . In our experiments, we demonstrate the combination of the two axis of speedups to both reduce the number of denoising steps and compute the steps in parallel. 
experiments: 
 We sample from models fine-tuned on Contrastive Diffusion Loss (CDL) via both parallel and sequential diffusion samplers across a variety of generation tasks, including complex low-dimensional manifold 2D synthetic data and real-world image generation. Our results demonstrate that employing CDL as a regularizer in models trained with standard diffusion loss enhances density estimation and sample quality while also accelerating convergence in parallel sampling. All sampling tests are done on A6000 GPUs. We visualize CDL image generation examples in App. C.Training configuration Our method is architecture-agnostic. In synthetic experiments, we adopt a simple MLP architecture with positional encoding for timesteps 3 Generation quality metrics For real-world data, our intrinsic metric is Frećhet Inception Distance (FID) score [5] . The number of images we generated for FID computation follows their baseline models' FID settings, and the FID scores are computed between 5, 0000 generated images and all available real images.For synthetic data, to measure how well the generated samples resemble samples from the ground truth distribution, we use the (unbiased) kernel estimator of the squared Maximum Mean Discrepancy (MMD), with Gaussian kernel with bandwidth set empirically as described in App. B.1.We adopt the following three metrics: (1) Neural function evaluations (NFE) for all settings, i.e. how many times the denoiser is evaluated to produce a sample; (2) For the parallel setting, we report the number of parallel Picard Iterations; (3) Furthermore for the parallel setting, the wall-clock time is reported. While parallel sampling can use fewer total iterations and less wall-clock time than a sequential sampler, this may come at the cost of an increase in the total number of function evaluations. This gap is called the algorithm inefficiency. In the subsequent section, we use contrastive diffusion loss as a training regularizer for standard diffusion losses and refer to the corresponding models as CDL-regularized models.In the parallel setting, we use Parallel DDPM sampler [28] with 1000-step diffusion sampling. And for synthetic experiment, to reflect sampling speed by only number of Picard iteration and wall-clock time, we don't use the sliding window technique, and the 2D data is small enough to fit the whole sampler trajectory in GPU memory. While for real-world experiment, sliding window is still applied.We consider the 2D Dino dataset [19] , characterized by its highly nonlinear density concentrated on a low-dimensional manifold. For baselines, we employ the standard DDPM loss [6] , as all standard diffusion losses similarly minimize a sum of MSE losses between the actual and estimated denoisers. Both CDL-regularized and DDPM-objective-trained models are trained with a MLP where timestep is encoded by positional encoding. We train it for 2000 epochs to ensure convergence and check the training and validation loss curve to avoid overfitting.In Fig. 4 , it is clearly demonstrated that the parallel generated samples from the CDL-regularized model is much better than the model trained only with the standard DDPM loss, especially around the chin, eyes and paws, where the manifolds are close to each other and difficult to distinguish and learn. From the MMD plot, we see that comparing to the baseline curve trained only with standard DDPM loss, the CDL-regularized curve converges faster with smaller number of Picard iterations and better Table 2 : Evaluating FID score (lower is better) of parallel DDPM sampler on real-world datasets using 5, 0000 samples. For reported FID scores, we run three sets of random seeds and reported the average with uncertainty.While CDL clearly improves parallel sampling quality and convergence speed, we also show that it improves the trade-off between generation speed and sample quality in the sequential setting. As for sequential diffusion sampling choices, we consider the DDPM sampler from In principle, increasing NFE has the potential to decrease the overall discretization errors, consequently leading to improved sample quality. However, in practice we observed an unusual behavior 6with the Karras deterministic sampler -as NFE increases, the FID score deteriorates (Fig. 6 ). In contrast to EDM models, CDL-regularized models exhibit a more stable FID score. This partially resolves the deterministic sampler sensitivity while improving the quality. deterministic ones only on challenging datasets, for simpler datasets, the introduction of stochasticity not only fails to enhance performance but exhibits image degradation issues, characterized by a loss of detail. They attribute this phenomenon to L2-trained denoisers excessively removing noise at each step (always remove more than it should), and propose to slightly increase the standard deviation (S noise ) 7 of newly added noise to 1.007. We argue that this approach may not totally resolve the issue and instead complicates the hyperparameter grid search process by introducing an additional parameter, S noise . Also, this S noise logically serves the same function as another hyperparameter γ i , where both of them control the amount of noise to add to reach a higher noise level.In this experiment, we conducted two stochastic sampling configurations for our baseline EDMtrained models. The first configuration, referred to as EDM-opt, operated at the EDM optimal setting with S noise = 1.007. The second, named as EDM-sub-opt, used a setting with S noise = 1.00, effectively disabling S noise . As for CDL configuration, we exclusively examined the scenario with S noise = 1.00 to determine whether CDL could address the problem of excessive noise removal. The results, as visualized in Figure 7 , indicate that CDL outperforms EDM in both S noise configurations. Notably, CDL not only improves upon the EDM-sub-opt configuration (dark blue line) but also surpasses the performance of the EDM-opt configuration (light blue line), even at its optimal setting. This not only demonstrates that CDL robustly provides a better sample quality, but also suggests that CDL can eliminate the need for the hyperparameter S noise . This reduction enables a more efficient grid search for the optimal EDM sampling settings, potentially enhancing the practicality of using such a sampler for other applications. 
related work: 
 The generative modeling trilemma [39] seeks generative models that produce samples (i) quickly, (ii) with high quality, and (iii) good mode coverage. Diffusion models excel at the latter two but a large amount of research has attempted to address the problem of slow sampling speed. From the inception of diffusion models [29] , dynamics has been at the forefront, so most work has focused on The FID comparison between our CDL and the baselines EDM in the stochastic sampler experiment on CIFAR-10. CDL's performance is strictly better for all S churn , outperforming the optimal setting of EDM which inflates the standard deviation S churn of the newly added noise.the interpretation of the dynamics as a differential equation that can be sped up through accelerated numerical solvers [30, 34, 42, 9, 17] . Our approach is compatible with any of these approaches as we are sampler agnostic, seeking only to improve the input to the sampler, which is the denoiser or score function estimator, through regularization during the diffusion model training. A separate line of work instead attempts to distill a diffusion model into a faster model that achieves similar sampling quality [27, 35, 21, 38, 39] . In principle, these methods could also benefit from distilling based on a more robust base diffusion model trained with CDL.Diffusion models admit a surprisingly diverse array of mathematical perspectives, like variational perspectives [6, 7, 12] , differential equations [34, 20] , and nonequilibrium thermodynamics [29] . Our approach is mostly inspired by connections between the information-theoretic perspective [13, 14] and the score matching perspective [31, 33, 32, 8, 37] . In particular, we point out that score function estimates in traditional diffusion training are sub-optimal, and the information-theoretic perspective leads to a new objective (CDL) that can improve the score estimate.While previous diffusion models focus on log-likelihood estimation, we consider a different approach based on density ratio estimation and noise contrastive estimation [4, 23] , which inspired several notable developments in machine learning [1, 25] . A few works have considered contrastive learning inspired modifications to diffusion either to enforce multimodal data relationships [16, 43] , for style transfer [41] , or for guidance during generation [24] , but none use the diffusion model as a noise classifier to improve diffusion training as we do. Most similar to our approach are methods that use Density Ratio Estimation (DRE) to estimate a ratio between the data density and some simple noise distribution. The density ratio can be estimated by learning to contrast between data samples and samples from the noisy distribution [4, 36] . Recent work generalized the idea to consider classifying between samples along a sequence of distributions between source and target [26, 2] . Our contribution is to relate this approach to diffusion models by noting that standard diffusion models implicitly already implement the required classifiers for distinguishing distributions on the path from the data distribution to a Gaussian distribution. Concurrent work makes a similar connection but while we focus on improving diffusion models by interpreting them as noise classifiers, [40] focused on the converse perspective, improving density ratio estimation by interpreting DREs as denoisers. 
conclusion: 
 In this paper, we introduced a novel connection between diffusion models and optimal noise classifiers.While this relationship has a variety of potential applications that could be explored in future work, we used the connection to propose a new self-supervised loss regularizer for diffusion models, the Contrastive Diffusion Loss (CDL). CDL reduces the error of the learned denoiser in regions that are OOD for the standard loss. We showed that CDL improves the robustness of diffusion models across all types of sampling dynamics, and leads to significant speed-ups for a promising new generation of parallel samplers. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: While we do not expect a direct societal impact from our methodological improvements, we added a broader impacts statement that acknowledges the potential impact of diffusion models. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We used existing standard datasets and public, pretrained models. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We give proper attribution for the open license code, data, and models used throughout the paper. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Score ConnectionWe derive the following relation.EQUATIONTo keep track of intermediate random variables and their associated distributions, we will use the cumbersome but more precise information theory notation where capitals represent a random variable and lowercase represents values. Define the channel that mixes the signal, X, with Gaussian noise as Z α ≡ σ(α)X + σ(-α)E with E ∼ N (0, I) and data distribution p(X), α represents the log of the Signal-to-Noise Ratio (SNR), and σ(•) is the sigmoid function. In this notation, the probability that a mixture distribution takes a value, x, would be written p(Z α = x).We start by re-writing the left-hand side in new notation, and expand the definition based on the noisy channel model, using δ(•) for the Dirac delta.∇ x log p(Z α = x) = 1/p(Z α = x)∇ x p(Z α = x) = 1/p(Z α = x)∇ x d xdϵ δ(x -σ(α) x -σ(-α)ϵ)p(X = x)p(E = ϵ) = 1/p(Z α = x) d xdϵ (∇ x δ(x -σ(α) x -σ(-α)ϵ))p(X = x)p(E = ϵ) = 1/p(Z α = x) d xdϵ ((-1/ σ(-α))∇ ϵ δ(x -σ(α) x -σ(-α)ϵ))p(X = x)p(E = ϵ) = 1/p(Z α = x) 1 σ(-α) d xdϵ δ(x -σ(α) x -σ(-α)ϵ)p(X = x)∇ ϵ p(E = ϵ) = -1/p(Z α = x) 1 σ(-α) d xdϵ δ(x -σ(α) x -σ(-α)ϵ)p(X = x)ϵp(E = ϵ) = -1/p(Z α = x) 1 σ(-α) dϵ p(Z α = x, E = ϵ)ϵ = - 1 σ(-α) dϵ p(E = ϵ|Z α = x)ϵ = - E ϵ∼p(E|Zα=x) [ϵ] σ(-α) = - ε(x, α) σ(-α)In the second line we expand, and in the third we just move the gradient inside the integral. In the fourth line we use the chain rule to relate the gradient over x to the gradient over ϵ (introducing a sign flip). In the fifth line we use integration by parts to move the gradient (second sign flip). Taking the gradient of the Gaussian in the sixth line gives our third sign flip, and the factor of ϵ. We can conclude by writing the expression in terms of a conditional distribution, and relating that to the optimal denoiser in Eq. 1.In this section, we derive the expression that shows that the density of a continuum of Gaussian mixture distributions can be written in terms of the optimal denoiser, ε, for the data distribution.-EQUATIONAs in the previous section, we will adopt information theory notation. If we define the optimal denoiser for the input distribution, p ζ (x), with a subscript as εζ (•, •), we can write the density analogously to Eq. 2.logEQUATIONNote that we now have to keep track of two log SNR values. One indicates how much noise is added to the new "data" distribution, the other is how much noise we add and then try to remove with our denoiser. The goal is to relate εζ to ε. We can formally write down the optimal solution using the relation in Eq. 1.εζ (x, ᾱ) = E ϵ∼p(E|Z=x) [ϵ]Now, however, the noise channel is defined differently. The channel mixes the signal, x ∼ p ζ ( X), with Gaussian noise, ε ∼ N (0, I), as Z ≡ σ( ᾱ) X + σ(-ᾱ) Ē. And the noisy variable, X = σ(ζ)X + σ(-ζ)E, where we must be careful to distinguish the two independent sources of Gaussian noise.We start by expanding definitions.εζ (x ζ , ᾱ) = 1 p(Z = x ζ ) dε p(E = ε, Z = x ζ ) ε = 1 p(Z = x ζ ) dεd x δ(x ζ -σ( ᾱ) x -σ(-ᾱ)ε) p( X = x) p( Ē = ε) ε = 1 p(Z = x ζ ) dεd xdxdϵ δ(x ζ -σ(ᾱ) x -σ(-ᾱ)ε) δ( x -σ(ζ)x -σ(-ζ)ϵ) • p(X = x) p(E = ϵ) p( Ē = ε) ε = 1 p(Z = x ζ ) dϵdεdx δ(x ζ -σ(ᾱ)( σ(ζ)x + σ(-ζ)ϵ) -σ(-ᾱ)ε) • p(X = x) p(E = ϵ) p( Ē = ε) εNow do a change of variables, a 2-d rotation with:ε′ = aε + bϵ, ϵ ′ = -bε + aϵ, a = σ(ᾱ)σ(-ζ)/(1 -σ(ζ)σ(ᾱ)), b = σ(-ᾱ)/(1 -σ(ζ)σ(ᾱ)). This change of variables leads to the following.εα (x ζ , ᾱ) = 1 p(Z = x ζ ) dϵ ′ dε ′ dx δ(x ζ -σ(ζ)σ(ᾱ)x -1 -σ(ζ)σ(ᾱ)ε ′ ) • p(X = x) p(E ′ = ϵ ′ ) p( Ē′ = ε′ ) (bε ′ + aϵ ′ ) = b 1 p(Z = x ζ ) dϵ ′ dε ′ dx δ(x ζ -σ(ζ)σ(ᾱ)x -1 -σ(ζ)σ(ᾱ)ε ′ ) • p(X = x) p(E ′ = ϵ ′ ) p( Ē′ = ε′ ) ε′ = b 1 p(Z = x ζ ) dϵ ′ dε ′ dx δ(x α -σ(β)x -1 -σ(β)ε ′ ) p(X = x) p(E ′ = ϵ ′ ) p( Ē′ = ε′ ) ε′ εζ (x ζ , ᾱ) = bε(x ζ , β), β ≡ σ -1 (σ(ᾱ)σ(ζ)), b = σ(-ᾱ)/(1 -σ(ζ)σ(ᾱ))Note in the second line that the expectation of ϵ ′ is zero, and we move the constant for the other term, b, outside the integral. In the third line, we define β which represents the log SNR of the two consecutive noisy channels with ζ, ᾱ. Then we recognize the resulting integral as Eq. 1, the optimal denoiser for recovering samples from from the original (non-noisy) data distribution in Gaussian noise.In this section, we calculate the analytical solution to the 1D two-mode Gaussian in Fig. 1 .Fig. 1 is plotting the norm of difference between the ground-truth denoiser εgt (•, •) and the estimated denoiser ε(•, •):denoiser_err(x, α) = ∥ε(x, α) -εgt (x, α)∥ 2 2To get this error plot, we need to analytically calculate ground-truth denoiser εgt (x, α). From score connection Eq. 9, for any intermediate noisy density log p α (x), denoiser function ε(x, α) can be derived from score function ∇ x log p α (x). Therefore, ultimately what we need to calculate here is the score function of any noisy distribution p α (x).The data we used is a mixture of two Gaussians, N (µ = -5, I) and N (µ = 5, I), and the noise distribution is consists of data plus noise, then the noisy distribution should also be a mixture of Guassians. We just need to relate the parameters of the noisy mixture of Gaussians to the parameters of the mixture of Gaussians.Start with one mode of the Gaussian mixture for the data. We could represent it in terms of the standard normal random variable, ϵ.x d = µ + σϵ Now call x α the random variable after applying a noisy channel with log-SNR α, here we present sigmoid function as σ(•) .x α = σ(α)x d + σ(-α)ϵ ′Note that we use a different ϵ ′ here. Now expand this, and then re-arrange.x α = σ(α)(µ + σϵ) + σ(-α)ϵ ′ = σ(α)µ + σ(α)σϵ + σ(-α)ϵ ′We want to represent this in a more canonical way to see what the variance and mean of this Gaussian is. Note that for two standard normal random variables, aϵ + bϵ ′ , we can represent them as a single random variable with the same variance, √ a 2 + b 2 ϵ ′′ (reparameterization trick).x α = σ(α)µ + σ(α)σ + σ(-α)ϵ ′′Now we see that the noisy Gaussian (one component of a mixture) is just a modified version of the original. We have to change the mean (moving it towards zero when adding noise) and the variance.In our example, we set σ = 1, so it simplifies further.x α = σ(α)µ + ϵ ′′ So the variance doesn't change, we just slowly shift the two mixtures together to the center.Therefore, for one mode x ∼ N (µ, I), the intermediate noisy log-density is.log p α (x) = -1 /2 log(2πσ 2 ) - (x -σ(α)µ) 2 2σ 2Take the gradient of the log-density via torch built-in function torch.autograd.grad(log-density, samples), we have the ground-truth score function εgt (x, α).The Maximum Mean Discrepancy (MMD) is a statistical test used to determine if two distributions are different. It works by comparing the mean embeddings of samples drawn from two distributions in high dimensional feature space. Specifically, if the distributions are the same, the means should be close; if they are different, the means should be far apart. The embeddings are typically constructed using a feature map associated with a kernel function, and here we select the Gaussian kernel:K(x, y) = exp(- ∥x -y∥ 2 2σ 2 )The bandwidth parameter σ of the Gaussian kernel plays a critical role in the sensitivity and performance of MMD. The better bandwidth choice, the more effective MMD computation is. Intuitively, the bandwidth σ controls the scale at which differences between distributions are detected. A small σ makes the kernel sensitive to differences at small scales (fine details), while a large σ highlights differences at larger scales.The choice of bandwidth is often related to the variance of the data, and the bandwidth should be on the order of the variance of the data. Through a small experiment where we calculate MMD score between our data and the standard Gaussian under various bandwidths σs, we pick the one that maximizes the MMD score.In the synthetic 2D Dino experiment, we plot the relationship between MMD scores and bandwidths (Fig. 8 ), setting σ = 3e -02. For DDPM model, the checkpoint 8 we used is a ema one pre-trained on unconditional CIFAR-10. The reason we are not using the most frequently used checkpoint (https://huggingface.co/ google/ddpm-cifar10-32) is that, this is not EMA checkpoint and our calculation of the FID score of this model on 50, 000 generated image via sequential DDPM sampler gives 12.43. This FID score is much higher than what reported on the original paper 3.17. Here we provide the FID scores of this non-EMA pre-trained model, results shown in Tab. uncond FFHQ-64 -batch=32 -lr=5e-5 -cres=1,2,2,2 -dropout=0.05 -augment=0.15 Table 5 : Fine-tuning configurations for different datasets For EDM model, in total eight checkpoints we used are "DDPM++ cont. (VP)" and "NCSN++ cont. (VE)" models pre-trained on three datasets (CIFAR-10, uncond-FFHQ, and uncond-AFHQv2 [15, 10, 3] ) with two training settings (unconditional and conditional) 9 10 11 12 13 14 15 16 .As for fine-tuning, we train all models with the same training setting in their original papers. For DDPM model, we train each model for 10 epochs and keep 'learning rate / batch size' ratio to be '10 -4 /64', and this training is on two A6000 GPUs. For EDM model, the following table list the exact our fine-tuning configurations, which is still of the same 'learning rate/batch size' ratio. This training is using eight V100 GPUs.The previous parallel diffusion sampling paper [28] calculates FID scores by using 5, 000 generated images and another 5, 000 randomly selected real images, and to follow the same experimental setting for comparison, we further provide the FID results in Table 6 Table 6 : Evaluating FID score (lower is better) of parallel DDPM sampler on real-world datasets using 5, 000 samples. "NA" stands for "Not Applicable". For reported FID scores, we run three sets of random seeds and reported the average with uncertainty.As the design choices of EDM model is very comprehensive and complicate, here we list the training noise distribution, loss weighting, network and preconditioning choices we make during CDL fine-tuning in Tab. 7.To use the pre-trained models in the literature with our CDL loss, we need to translate "t", a parameter representing time in a Markov chain that progressively adds noise to data in Ho et al. 2020 Translation between Noise Variance Sigma and Log-SNR For variance scale of the Gaussian noise σ in EDM, referring to Eq.( 7) and ( 8) in Karras et al. 2022, it's easy to translate the preconditioning:EQUATIONEQUATIONFrom Eq. 18 and Eq. 17, we see that σ ≡ σ(-α) σ(α) , therefore, the relationship between α and σ should be: σ ≡ exp(-α/2), α ≡ -2 ln(σ)To implement contrastive loss, we follow the definition in Sec. 3. First, we generate a random binary label y. Next, conditioned on y, we sample from either data distribution p(x) or the noisy data distribution p ζ (x). We calculate the point-wise log-likelihood, then the contrastive loss in Eq. 7. As we mentioned, CDL training is more expensive to compute than the standard diffusion loss, and here we analysis and give the reason.According to Kong et al. 2023 , we can write the pointwise standard diffusion loss function as Eq. 19, and therefore the standard diffusion loss is as Eq. 20.EQUATIONEQUATIONTo train the standard diffusion loss, we simply need to optimize Eq. 20 by all the training data. However, to train the contrastive diffusion loss, we need to estimate nll(x) and nll(x + ζ) term in Algo. 1, and there we estimate Eq. 19 by duplicating a single data point x for N = 100 times and calculate Eq. 20. This pointwise NLL estimation nll(x) demands N = 100 times more computational resources compared to the standard diffusion loss.In principle, the contrastive loss Algo. 1 should be executed for the entire training dataset. However, due to the high computational cost, we optimize only one data point per batch instead of utilizing all the training data.We provide visualization of the images generated from pre-trained models fine-tuned via CDL loss. Justification: We explain the proposed CDL speeds up parallel sampling convergence and improves generation quality. For sequential sampling, it improves the trade-off between sampling quality and gerneration speed. We provide the claims in both abstract and introduction, and justify them in the Results section.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We do not use human subjects and only used standard datasets for experiments. We acknowledge the potential harms of diffusion models in the broader impacts statement, and do not anticipate that our methodological contributions will alter known issues. We also strive for transparency wherever possible, including full, anonymized code with our submission.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Code can be found at https://github.com/yunshuwu/ContrastiveDiffusionLoss.git 38th Conference on Neural Information Processing Systems (NeurIPS2024).For detailed convergence proof, we refer to Shih et al. 2024 sectionArchitecture adopted from: https://github.com/Jmkernes/Diffusionhttps://github.com/pesser/pytorch_diffusionhttps://github.com/NVlabs/edmThe same issue is also reported in https://github.com/NVlabs/edm/issues/4We refer to Karras et al.[11] for detailshttps://github.com/pesser/pytorch_diffusion 


Decision: Building on the denoising score matching loss, the paper presents a new contrastive learning loss. This loss estimates the log-likelihood ratio between mixture densities with different noise levels. The contrastive learning loss is subsequently used to fine-tune the diffusion model, improving its capability to estimate out-of-distribution (OOD) data. Experimental results show that this approach leads to enhanced performance of the fine-tuned diffusion model in both sequential and parallel sampling, areas that often suffer from poor OOD estimates.

Most reviewers find the paper well-presented and well-written, and the paper draws an interesting connection between diffusion model denoiser and noise classifier and introduces a novel contrastive loss, and the derivation is solid and insightful. 
In the camera-ready version, we suggest the authors take the reviewers' feedback and revise accordingly, such as a more comprehensive discussion with related works and a more convincing evaluation of the proposed methods.

Reviews: review: paper_summary: paper_summary: Building on the denoising score matching loss, the paper introduces a novel contrastive learning loss. This loss estimates the log-likelihood ratio between mixture densities with varying noise levels. The contrastive learning loss is then employed to fine-tune the diffusion model, enhancing its ability to estimate out-of-distribution (OOD) data. To validate this approach, the paper demonstrates experimentally that the fine-tuned diffusion model exhibits improved performance in both sequential and parallel sampling, which typically degrade with poor OOD estimates.
strength_weakness: strength_weakness: The experimental results are not particularly impressive:

1. Although CDL fine-tuning consistently enhances parallel sampling performance on both synthetic and real-world datasets, the improvements are marginal. Additionally, the baseline chosen for comparison is not state-of-the-art. For instance, in the unconditional generation in Table 2, even with CDL fine-tuning, an FID of around 7.0 on CIFAR-10 falls short of the state-of-the-art FID, which is less than 2.0. The improvement in sequential generation is even less substantial. 

2. I believe the proposed CDL has significant potential. For instance, when addressing inverse problems, the diffusion model also encounters OOD issues. The paper could benefit from including more promising experiments to demonstrate how CDL enhances the diffusion model's OOD estimation capabilities.
questions: questions: 1. I think there are issues with the chosen baseline in the paper. In the original parallel sampling method [27], parallel sampling achieves similar performance to sequential sampling in significantly less time. However, the results reported in Table 2 show that unconditional parallel sampling performs much worse than sequential sampling (around 3.0 for the original DDPM, VP, and VE). Could the author provide more clarification on this discrepancy?

2. According to [1], the denoising score matching loss serves as an upper bound for the negative log-likelihood. So minimizing the denoising score matching loss is equivalent to maximizing the negative log-likelihood.  Are there any insights or proofs provided on why minimizing the CDL would also lead to meaningful generation?


[1] Song, Yang, Conor Durkan, Iain Murray, and Stefano Ermon. "Maximum likelihood training of score-based diffusion models." Advances in neural information processing systems 34 (2021): 1415-1428.
limitations: limitations: The authors have covered the limitations of their work.
paper_summary: paper_summary: The paper establishes a connection between the diffusion model denoiser and noise classifier through an examination of log-likelihood ratio estimation. The authors introduce a novel loss function, termed Contrastive Diffusion Loss (CDL), designed to encourage diffusion models to explore OOD regions in noise levels. Empirical results indicate that the proposed loss function enhances performance in parallel sampling and demonstrates robustness to different hyperparameters in sequential sampling.
strength_weakness: strength_weakness: * The clarity of the paper could be further enhanced. For instance, in line 86, the authors introduce the notion $\zeta$ to indicate the noise level, while in Appendix A.2, they use $\alpha$ during the derivation. It seems like the authors want to differentiate between noise level and SNR, but the message is not entirely clear here.

* The empirical section of the paper is not as strong as the analytical section. In the parallel sampling example, the method shows good performance, but even with the performance improvement, the results are still far from optimal compared to EDM with the default settings. The experiment on deterministic samplers aims to demonstrate that CDL-regularized models maintain a more stable FID score with changes in NFE, but the authors do not explain why CDL helps in this context. The same issue applies to the experiment on stochastic samplers.

Overall, I am on the positive side for this paper. I would appreciate it if the authors could provide more intuitive explanations to better connect the analytical and empirical sections of the paper.
questions: questions: Why did the authors choose fine-tuning with the CDL over pre-training and using CDL as a regularization?
limitations: limitations: The authors addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: The paper presents a method to improve the parallel sampling of the diffusion model by improving the denoising network for out-of-distribution evaluation. The paper proposes to finetune a trained model using the log-likelihood ratio of a sample at two different noise scales. The log-likelihood ratio is obtained by integrating the denoising error at all noise scales. The proposed method performs better for parallel sampling and performs comparable for sequential sampling on the CIFAR-10, AFHQ, and FFHQ datasets.
strength_weakness: strength_weakness: - The contrastive diffusion loss (CDL) contains the integration of denoising error over all noise scales. In practice, I assume that the integration will be replaced by a numerical integration therefore, evaluating the proposed CDL loss is expensive. Considering the training/finetuning using CDL loss is expensive, the benefits from the training don’t seem to be significant in the empirical performance. Additionally, very few/no details are provided on how the integration is approximated in experiments/practice.  
- Some discussions about the related works need to be included. For example, another work on improving parallel [1] is not mentioned/discussed/compared against. Additionally, the derivation of the CDL loss is somewhat similar to some diffusion classifier papers (for example, [2, 3, 4]) therefore they should be cited/discussed. 

[1] Accelerating Parallel Sampling of Diffusion Models

[2] Your Diffusion Model is Secretly a Zero-Shot Classifier

[3] Your Diffusion Model is Secretly a Certifiably Robust Classifier

[4] Robust Classification via a Single Diffusion Model
questions: questions: - It is not clear why only 5k samples are used for calculating FID for parallel sampling. The standard practice is to use 50k samples. As the FID depends on the number of samples, it is hard to make sense of the significance of improvement. 
- Can you experimentally show/quantify the improvement of the denoiser for out-of-distribution evaluation when trained using contrastive diffusion loss (CDL)?
limitations: limitations: yes

Comments: title: Response to rebuttal
comment: Thanks to the authors for the additional experiments and clarifications. I hope that the authors would include these additional results and discussions when they update the paper.

My concerns are addressed, thus raising 6 -> 7.
Thank you very much for enhancing the score. We appreciate your careful review once again!
Dear Reviewer M92h,

We greatly appreciate the time you took to review our paper. Due to the short duration of the author-reviewer discussion phase, we would appreciate your feedback on whether your main concerns have been adequately addressed. We are ready and willing to provide further explanations and clarifications if necessary.

Thank you very much!
Dear Reviewer XUix,

We greatly appreciate the time you took to review our paper. Due to the short duration of the author-reviewer discussion phase, we would appreciate your feedback on whether your main concerns have been adequately addressed. We are ready and willing to provide further explanations and clarifications if necessary.

Thank you very much!
Dear Reviewers,

I hope this message finds you well. I would like to take this opportunity to express my gratitude for the valuable feedback and insights you have provided during the review process. Your expertise and dedication are essential to maintaining the high standards of our community.

The authors have devoted considerable time and effort to addressing the concerns and suggestions you raised in your reviews. They have carefully crafted a detailed rebuttal, aiming to clarify any misunderstandings and incorporate your feedback into their manuscript.

I kindly ask that you take a thorough and thoughtful look at the authors' responses. Your assessment of their rebuttal is crucial in determining whether they have satisfactorily addressed the issues and concerns highlighted in your initial review.

Thank you once again for your hard work and commitment to advancing the quality of our scholarly community. Your contributions are greatly appreciated.

Best regards, 

AC
Thanks for the author's response, I have increased my score to 5.