Title: Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion
Abstract: Diffusion models learn to denoise data and the trained denoiser is then used to generate new samples from the data distribution. 
In this paper, we revisit the diffusion sampling process and identify a fundamental cause of sample quality degradation: the denoiser is poorly estimated in regions that are far Outside Of the training Distribution (OOD), and the sampling process inevitably evaluates in these OOD regions.
This can become problematic for all sampling methods, especially when we move to parallel sampling which requires us to initialize and update the entire sample trajectory of dynamics in parallel, leading to many OOD evaluations. 
To address this problem, we introduce a new self-supervised training objective that differentiates the levels of noise added to a sample, leading to improved OOD denoising performance. The approach is based on our observation that diffusion models implicitly define a log-likelihood ratio that distinguishes distributions with different amounts of noise, and this expression depends on denoiser performance outside the standard training distribution.
We show by diverse experiments that the proposed contrastive diffusion training is effective for both sequential and parallel settings, and it improves the performance and speed of parallel samplers significantly. Code for our paper can be found at https://github.com/yunshuwu/ContrastiveDiffusionLoss

Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training

Abstract

Diffusion models learn to denoise data and the trained denoiser is then used to generate new samples from the data distribution. In this paper, we revisit the diffusion sampling process and identify a fundamental cause of sample quality degradation: the denoiser is poorly estimated in regions that are far Outside Of the training Distribution (OOD), and the sampling process inevitably evaluates in these OOD regions. This can become problematic for all sampling methods, especially when we move to parallel sampling which requires us to initialize and update the entire sample trajectory of dynamics in parallel, leading to many OOD evaluations. To address this problem, we introduce a new self-supervised training objective that differentiates the levels of noise added to a sample, leading to improved OOD denoising performance. The approach is based on our observation that diffusion models implicitly define a log-likelihood ratio that distinguishes distributions with different amounts of noise, and this expression depends on denoiser performance outside the standard training distribution. We show by diverse experiments that the proposed contrastive diffusion training is effective for both sequential and parallel settings, and it improves the performance and speed of parallel samplers significantly. 1

1 Introduction

Denoising diffusion models [29] achieve state-of-the-art performance on various unsupervised learning tasks and have intriguing theoretical connections to methods like denoising autoencoders [37], VAEs [6], stochastic differential equations [20, 34], information theory [13], and score matching [31, 32]. Diffusion models are presented with data samples corrupted by a forward dynamical process that progressively adds more Gaussian noise and trained to reverse this dynamics or denoise the corrupted samples. Samples are then generated by applying the reverse dynamics on images of pure Gaussian noise to produce high-quality samples from the target distribution.

The key to the success of diffusion models is the dynamics that gradually bridges the source and a target distribution, but it suffers from slow sampling, as sequentially simulating these dynamics can take thousands of denoising steps for one sample. Most recent works attempt to expedite the sequential dynamics by taking fewer, larger steps [30, 11, 18, 22]. However, the complexity of these samplers and the need for expensive sampling hyper-parameter grid searches tailored to specific datasets makes them difficult to generalize.

Shih et al. 2024 suggests a different approach by randomly initializing the entire path of the reverse dynamics and then updating all the steps in the path in parallel. The parallel sampling approach promises to drastically reduce wall-clock time at the cost of increased parallel computation. However, it encounters a problem that has largely gone unnoticed in the sequential sampling literature: while

Parallel sampling trajectories

k=2

k=1 (initialization)

200

400

600

800

1000

17

15

12

10

7.

5.

2.5

sequential paths sampled during generation are designed carefully to stay as close as possible to the forward paths that add noise to data, the parallel sampler often evaluates in regions far from where the score estimate (the denoiser) is trained, as illustrated in Fig. 1. The shading shows that the error of the score estimate is large in these regions, leading to poor performance for parallel samplers. Discretization errors can lead to similar issues, even for standard sequential samplers.

We propose to improve the training of diffusion models so that the error of the denoiser is reduced in OOD regions, and we hypothesize that this should significantly improve the performance for parallel samplers as they require more OOD evaluations. Our approach starts with an unexpected connection: the optimal MSE denoiser that defines the diffusion dynamics also defines an optimal noise classifier that distinguishes between samples with different amounts of noise. This provides a useful additional signal for training, because optimizing for the noise classification task involves evaluating the denoiser for one noise level on samples from distributions at different noise levels, while standard MSE optimization only evaluates the denoiser on samples from the matching noisy distribution. Accurate denoiser evaluation in regions that are OOD for standard diffusion training is important for robust sampling dynamics.

Contributions:

We use the information-theoretic formulation of diffusion to draw connections between diffusion, log-likelihood ratio estimation, and classification. This reveals that optimal diffusion denoisers are also implicitly optimal classifiers for predicting the amount of noise added to an image.

We show that CDL improves the trade-off between generation speed and sample quality, and that this advantage is consistent across different models, hyper-parameters, and sampling schemes. The improvement is especially substantial for parallel diffusion samplers [28] which rely heavily on OOD denoiser evaluations.

We leverage the noise classifier (via density ratio estimation [4]) interpretation to introduce a novel self-supervised loss function for regularizing diffusion model training, which we call the Contrastive Diffusion Loss (CDL). CDL provides training signal in regions that are OOD for the standard MSE diffusion loss.

2 Diffusion Model Background: Optimal Denoisers are Density Estimators

The defining feature of diffusion models is a sequence of distributions that progressively add noise to the data, from which we then learn to recover the original data. The ('variance preserving' [34]) channel that mixes the signal x with Gaussian noise is defined as x α ≡ √ σ ( α ) x + √ σ ( -α ) ϵ with ϵ ∼ N (0 , I ) , x ∼ p ( x ) , where α represents the log of the Signal-to-Noise Ratio (SNR), p ( x ) is the

unknown data distribution for x ∈ R d , and σ ( · ) is the sigmoid function. We define the sequence of intermediate distributions drawn according to this channel with a subscript as p α ( x ) . By definition, we express lim α →∞ p α ( x ) = p ( x ) in this paper. Note that we use a different scaling convention for noise from [11] and [6], where the former one takes x + σ ϵ as the forward noising channel and the latter one takes √ α t x + √ 1 -α t ϵ as the forward noising channel. For further detailed relationships among these scaling conventions, please check App. B.3.

The minimum mean square error (MMSE) estimator ˆ ϵ for recovering ϵ from the noisy channel that mixes x and ϵ can be derived via variational calculus and written as follows.

Sampling from the true posterior is typically intractable, but by using a neural network to approximate the solution to the regression optimization problem, we can get an approximation for ˆ ϵ . From [13], we see that log-likelihood can be written exactly in terms of an expression that depends only on the MMSE solution to the Gaussian denoising problem, i.e.

The constant, c = d/ 2 log(2 πe ) -d 2 ∫ ∞ 0 d ¯ α σ (¯ α ) does not depend on data and will play no role in our approach, as it cancels out in our derivations in Sec. 3.

3 What Your Diffusion Model is Hiding: Noise Classifiers

We now introduce our first main result, which shows that diffusion models implicitly define optimal noise classifiers. Eq. (2) expresses the probability density of the data directly in terms of the denoising function. If we apply Eq. (2) to the noisy distributions that bridge the data and a Gaussian, p ζ ( x ) , we can see that all mixture densities can be written in terms of the same optimal denoising function, ˆ ϵ ( · , · ) . The complete derivation is presented in App. A.2.

Intuitively, if we find the optimal denoising function for the data distribution, it may be hypothesized that it can denoise an already noisy version of the data distribution. Using Eq. 2, this directly translates into an expression for density of mixture distributions. Differences in log likelihoods lead to cancellation of constants, and these Log Likelihood Ratios (LLR) are related to the optimal classifiers [4] as we show below.

To connect LLRs with classification, consider the following generative model. We generate a random binary label q ( y = ± 1) = 1 / 2 . Then, conditioned on y , we sample from some distribution q ( x | y ) . Given samples ( x , y ) ∼ q ( x , y ) = q ( x | y ) q ( y ) , the Bayes optimal classifier is:

In the second line, because ∀ y, q ( y ) = 1 / 2 , these constants cancel out. Then we can just expand definitions and re-arrange to write in terms of log probabilities.

Contrastive Diffusion Loss (CDL) Our next contribution is to use the new connection between diffusion denoisers and noise classifiers to define a new training objective. We set the distributions q ( x | y = 1) and q ( x | y = -1) to be two distributions at different noise levels that we can write in terms of the optimal diffusion denoiser from Eq. 3. So we have q ( x | y = 1) ≡ p ( x ) , the data distribution,

and q ( x | y = -1) ≡ p ζ ( x ) , for some noise level, ζ . Then given a sample ( x , y ) ∼ q ( x , y ) the per-sample cross-entropy loss for the noise classifier, Eq. (6), is as follows.

Limitations: We highlight that CDL is more expensive to compute than the standard diffusion loss, significantly increasing the total cost of diffusion model training. Implementation details appear in App. B.4 and training cost details appear in App. B.5.

We can estimate both densities directly from our denoising model using Eq. (3), with the constants canceling out in the process. This loss differs significantly from the standard diffusion loss. Intuitively, to distinguish between a sample from the data distribution, p ( x ) , versus a noisy version of the data distribution, p ζ ( x ) , we need to evaluate denoisers on points from both distributions. In standard diffusion training, denoisers at noise level ζ are only trained on samples from p ζ ( x ) .

Choice of noise to contrast Next, let's break the Log-Likelihood Ratio (LLR) term in Eq. (7) down to see how to choose ζ to maximize the benefit of CDL. Combining Eq. (2) and Eq. (3) we have Eq. (8), where the constant cancels out.

From Eq. (8) we see that ˆ ϵ ( · , · ) is trained on four pairs: ( z α , α ) , ( z β , β ) , ( z β , α ) and ( z α , β ) , where β ≡ σ -1 ( σ ( α ) σ ( ζ )) < min( α, ζ ) (Eq. 5). During standard training, only the first two pairs are trained (Eq. 1). This means that our CDL objective trains the denoiser to perform correctly even for samples from distributions that are noisier or cleaner than the specified noise level (a pair like ( z β , α ) or ( z α , β ) ). This can be useful for both sequential and parallel sampling settings. During sequential sampling, extra error noise added due to discretization errors can be corrected by the denoiser trained with CDL. As for parallel sampling, CDL helps with evaluations on asymmetric pairs ( z β , α ) or ( z α , β ) which we refer to OOD regions for standard diffusion loss.

Note that the input x to the LLR term may come from two different distributions, which breaks the standard synchronous denoising pair ( x α , α ) into asynchronous. When it's from data distribution x ∼ p ( x ) , z = z α ; and when it's from some noisy data distribution x ∼ p ζ ( x ) , z = z β .

In practice, diffusion training pipelines are highly tuned on popular datasets like CIFAR10 and ImageNet, so the amplitude of discretization errors during sampling is small, meaning that errors won't nudge points too far away from the true trajectory. Therefore, when evaluating CDL objective, we sample some large-valued ζ s, which corresponds to classifying only small differences in noise levels. Empirically we find that ζ ∼ Uniform [6 , 15] or ζ ∼ logistic [6 , 15] performed equally good.

Denoising, sampling dynamics, and the score connection We have focused so far on denoising and density estimation, but we now want to connect this discussion to the primary use case for diffusion models and the focus of Sec. 4, sampling . There are many choices in how to implement sampling dynamics [11], but all of them rely on the score function , ∇ x log p α ( x ) . The score function points toward regions of space with high likelihood, and by slowly transitioning (or annealing), from the score function of a noisy distribution to one closer to the data distribution, we can build reliable sampling dynamics. To connect denoisers with sampling we must show that a denoising function, ˆ ϵ , that is optimal according to Eq. 1 also specifies the score function.

The derivation is straightforward and is given in Appendix A.1.

4 Sequential and Parallel Sampling with Diffusion Models

Sampling dynamics are typically presented in terms of a stochastic process { x t } T t =1 with timestep, t , rather than in terms of log SNR, α . We will denote x t ≡ x α ( t ) , p t ( x ) ≡ p α ( t ) ( x ) , to connect to our previous notation, with α ( t ) representing a monotonic relationship described in App. B.3. Note that decreasing log -SNR α corresponds to increase timestep t , since smaller log -SNR means there is more noise added to the data.

x

0

The general form of sampling dynamics is a process of slowly transitioning from samples of a simple and tractable distribution to the target distribution. Specifically, start with an isotropic Gaussian x T ∼ N (0 , I ) , the sampler steps through a series of intermediate distributions with noise levels { T, T -1 , . . . , 1 } following the score estimates. Many works [34, 11] interpret diffusion models as stochastic differential equations (SDEs). The forward process is in the form of

where w t is the standard Wiener process/Brownian motion, f and g are drift coefficient and diffusion coefficient of x t separately. The reverse process of Eq. 10 is then used to generate samples

Depending on choices for f, g , we can get either a stochastic or ordinary (deterministic) differential equation. Either way, numerical differential equation solvers are used to approximate the true dynamics. The solver introduces discretization errors at each step, causing the trajectory to deviate into the OOD region where the score (or denoiser) is poorly estimated, further compounding the errors. More discretization steps reduce accumulated error and leads to better sample quality, at the expense of more sequential computation. As a result, a significant limitation of diffusion models is that they require many iterations to produce high quality samples.

Sequential Sampling The influential diffusion sampler DDPM [6] iterates over thousands of discretization steps in simulating the dynamics. Recently, many sequential sampling methods have been developed to take fewer and larger steps while introducing less error [30, 22, 11]. Specifically, Karras et al. 2022 studies the curvature shape of SDE/ODE trajectory and suggests a discretization technique where the resulting tangent of the solution trajectory always points towards the denoiser output. However, speeding up the sequential sampling sacrifices generation quality. Furthermore, the SOTA sequential samplers [18, 11, 30] require hyperparameter tuning and grid search on specific datasets, which poses challenges to the generalization of these samplers to other datasets.

x

1

x

2

x

T

. . .

Figure 1: Computation graph of sequential sam-

p

pling by evaluating

θ

(

x

k

0

x

k

1

x

k

2

x

k

T

x

k

+1

0

x

k

+1

1

x

k

+1

2

x

k

+1

T

. . .

. . .

spective of reverse time. Examining the iterative update rule in Eq. (5), we see that an update at time t depends on all previous timesteps instead of just the previous timestep t -1 . This amounts to introducing skip dependencies in the computation graph (Fig. 2), which enables information to propagate quickly down the chain and accelerate sampling. Parallel Sampling Shih et al. 2024 explores a parallel sampling scheme, where the entire reverse process path is randomly initialized and then all steps in the path are updated in parallel. Parallel sampling is based on the method of Picard iteration, an old technique for solving ODEs through fixed-point iteration. An ODE is defined by a drift function s ( x , t ) and initial value x 0 . In the integral form, the value at time t can be written as their outputs into prefix sums. Given enough parallel processing power, the sampling time then scales

will converge in one step.

with the number of iterations K until convergence, instead of the number of denoising steps T . The number of iterations until convergence depends on the drift function s . More concretely, sequential evaluation can be written as a nested evaluation of functions x /star t +1 = h t ( . . . h 2 ( h 1 ( x 0 ))) on the initial value x 0 where h i ( x ) = x + s ( x , i/T ) /T . If, for all timesteps, the drift at the true solution can be accurately obtained using the drift at the current guess, then the parallel evaluation In other words, the value at time t must be initial value plus the integral of the derivative along the path of the solution. This formula suggests a natural way of solving the ODE by starting with a guess of the solution { x k +1 t : 0 ≤ t ≤ 1 } at initial iteration k = 0 , and iteratively refining by updating the value at every time t until convergence 2

(Proof in Appendix A) (Picard Iteration)

Proposition 1.

s ( x k i , i/T ) = s ( h i -1 ( . . . h 2 ( h 1 ( x 0 ))) , i/T ) ∀ i ≤ t = ⇒ x k +1 t +1 = x /star t +1 To perform Picard iterations numerically, which is shown in Fig. 2, we can write the discretized form of Eq. 12 with step size 1 /T , for t ∈ [0 , T ] :

large empirical speedup.

of denoising steps.

Recall that the drift term using higher-order methods on

Remarkably, Picard iterations are also fully compatible with prior methods for reducing the number approximated using Euler discretization as

p

θ

-

s

(

+1

p

θ

(

x

t

x

|

t

, t/T

x

t

)

-

)

/T

x

t

, but it can also be readily approximated axis of speedups to both reduce the number of denoising steps and compute the steps in parallel.

can be written as

h

t

(

x

t

)

x

t

and

. In our experiments, we demonstrate the combination of the two

3.1

Practical considerations

We see that the expensive computations { s ( x k i , i/T ) : i ∈ [0 , T ) } can be performed in parallel. After some number of Picard iterations, the error difference between two iterates ∥ x k +1 t -x k t ∥ 2 2 drops below some convergence threshold. This converged trajectory, x ∗ t , should be close to the sequential sampler trajectory. Looking at the example in Fig. 1, we show the trajectories of three iterations k = 1 , 2 , 3 . The trajectories before convergence are consistently appearing in the regions with high score error.

5 Experiments

We sample from models fine-tuned on Contrastive Diffusion Loss (CDL) via both parallel and sequential diffusion samplers across a variety of generation tasks, including complex low-dimensional manifold 2D synthetic data and real-world image generation. Our results demonstrate that employing CDL as a regularizer in models trained with standard diffusion loss enhances density estimation and sample quality while also accelerating convergence in parallel sampling. All sampling tests are done on A6000 GPUs. We visualize CDL image generation examples in App. C.

Training configuration Our method is architecture-agnostic. In synthetic experiments, we adopt a simple MLP architecture with positional encoding for timesteps 3 , as it is one of the most versatile models in the literature. In real-world experiments, we consider the standard diffusion loss with two training configurations: DDPM by Ho et al. 2020 and EDM by Karras et al. 2022. For more details on model training, data split, and hyper-parameters, please refer to App. B.2.

Generation quality metrics For real-world data, our intrinsic metric is Fre´ chet Inception Distance (FID) score [5]. The number of images we generated for FID computation follows their baseline models' FID settings, and the FID scores are computed between 5 , 0000 generated images and all available real images.

For synthetic data, to measure how well the generated samples resemble samples from the ground truth distribution, we use the (unbiased) kernel estimator of the squared Maximum Mean Discrepancy (MMD), with Gaussian kernel with bandwidth set empirically as described in App. B.1.

Sampling speed metrics We adopt the following three metrics: (1) Neural function evaluations (NFE) for all settings, i.e. how many times the denoiser is evaluated to produce a sample; (2) For the parallel setting, we report the number of parallel Picard Iterations; (3) Furthermore for the parallel setting, the wall-clock time is reported. While parallel sampling can use fewer total iterations and less wall-clock time than a sequential sampler, this may come at the cost of an increase in the total number of function evaluations. This gap is called the algorithm inefficiency. In the subsequent section, we use contrastive diffusion loss as a training regularizer for standard diffusion losses and refer to the corresponding models as CDL-regularized models.

5.1 Parallel Sampling

In the parallel setting, we use Parallel DDPM sampler [28] with 1000-step diffusion sampling. And for synthetic experiment, to reflect sampling speed by only number of Picard iteration and wall-clock time, we don't use the sliding window technique, and the 2D data is small enough to fit the whole sampler trajectory in GPU memory. While for real-world experiment, sliding window is still applied.

Synthetic Dataset We consider the 2D Dino dataset [19], characterized by its highly nonlinear density concentrated on a low-dimensional manifold. For baselines, we employ the standard DDPM loss [6], as all standard diffusion losses similarly minimize a sum of MSE losses between the actual and estimated denoisers. Both CDL-regularized and DDPM-objective-trained models are trained with a MLP where timestep is encoded by positional encoding. We train it for 2000 epochs to ensure convergence and check the training and validation loss curve to avoid overfitting.

In Fig. 4, it is clearly demonstrated that the parallel generated samples from the CDL-regularized model is much better than the model trained only with the standard DDPM loss, especially around the chin, eyes and paws, where the manifolds are close to each other and difficult to distinguish and learn. From the MMD plot, we see that comparing to the baseline curve trained only with standard DDPM loss, the CDL-regularized curve converges faster with smaller number of Picard iterations and better

(a) Ground-truth dino.

(b) Using CDL loss.

(c) Using DDPM loss.

20

22

24

26

28

30

32

34

36

Number of Picard Iterations

0.001

0.002

0.003

0.004

0.005

0.006

0.007

0.008

0.009

MMD

Contrastive loss

DDPM loss

sample quality (lower MMD scores). Table. 1 shows the sampling speed results, from where we see that CDL-regularized model converges faster with lower final MMD and better sample quality.

Real-world Datasets We select 'DDPM++ cont. (VP)' and 'NCSN++ cont. (VE)' models by [11] trained on CIFAR-10 at 32 × 32 , unconditional FFHQ, and unconditional AFHQv2 [15, 10, 3] as baselines, comparing to the corresponding CDL-regularized models. We adopt the pre-trained models from Ho et al. 2020 4 and Karras et al. 2022 5 . More experimental results can be found in App. B.2. As shown in Tab. 2, CDL-regularized models always outperformed baselines with respect to FID scores.

5.2 Sequential Sampling

While CDL clearly improves parallel sampling quality and convergence speed, we also show that it improves the trade-off between generation speed and sample quality in the sequential setting. As for sequential diffusion sampling choices, we consider the DDPM sampler from Ho et al. 2020, and both the deterministic and stochastic samplers from Karras et al. 2022. To ensure fair comparisons, we adopt the original sampling hyper-parameter settings for all baselines.

Deterministic samplers For FID test, we follow the exact sampling settings outlined in Karras et al. 2022 for each dataset. FID scores are reported in Tab. 3, for sequential deterministic EDM samplers, CDL objective ensures that the generation quality is consistently similar or better.

In principle, increasing NFE has the potential to decrease the overall discretization errors, consequently leading to improved sample quality. However, in practice we observed an unusual behavior 6 with the Karras deterministic sampler - as NFE increases, the FID score deteriorates (Fig. 6). In contrast to EDM models, CDL-regularized models exhibit a more stable FID score. This partially resolves the deterministic sampler sensitivity while improving the quality.

Stochastic samplers In practice, stochastic samplers often yield superior performance compared to deterministic ones. However this is not true in Karras et al. 2022: stochastic samplers outperform

16

32

64

128

1.86

1.88

1.90

1.92

FID

NFE=

(a) Cond. CIFAR-10, VP

16

32

64

128

1.950

1.975

2.000

2.025

FID

NFE=

(b) Uncond. CIFAR-10, VP

16

32

64

128

2.4

2.6

2.8

FID

NFE=

(c) Uncond. FFHQ, VP

16

32

64

128

2.00

2.05

2.10

2.15

FID

NFE=

(d) Uncond. AFHQ-v2, VP

EDM

CDL

16

32

64

128

1.84

1.86

1.88

1.90

FID

NFE=

(e) Cond. CIFAR-10, VE

16

32

64

128

1.95

2.00

2.05

FID

NFE=

(f) Uncond. CIFAR-10, VE

16

32

64

128

2.6

2.8

3.0

FID

NFE=

(g) Uncond. FFHQ, VE

16

32

64

128

2.10

2.15

2.20

2.25

2.30

FID

NFE=

(h) Uncond. AFHQ-v2, VE

deterministic ones only on challenging datasets, for simpler datasets, the introduction of stochasticity not only fails to enhance performance but exhibits image degradation issues, characterized by a loss of detail. They attribute this phenomenon to L2-trained denoisers excessively removing noise at each step (always remove more than it should), and propose to slightly increase the standard deviation ( S noise ) 7 of newly added noise to 1.007. We argue that this approach may not totally resolve the issue and instead complicates the hyperparameter grid search process by introducing an additional parameter, S noise . Also, this S noise logically serves the same function as another hyperparameter γ i , where both of them control the amount of noise to add to reach a higher noise level.

In this experiment, we conducted two stochastic sampling configurations for our baseline EDMtrained models. The first configuration, referred to as EDM-opt, operated at the EDM optimal setting with S noise = 1 . 007 . The second, named as EDM-sub-opt, used a setting with S noise = 1 . 00 , effectively disabling S noise . As for CDL configuration, we exclusively examined the scenario with S noise = 1 . 00 to determine whether CDL could address the problem of excessive noise removal.

The results, as visualized in Figure 7, indicate that CDL outperforms EDM in both S noise configurations. Notably, CDL not only improves upon the EDM-sub-opt configuration (dark blue line) but also surpasses the performance of the EDM-opt configuration (light blue line), even at its optimal setting. This not only demonstrates that CDL robustly provides a better sample quality, but also suggests that CDL can eliminate the need for the hyperparameter S noise . This reduction enables a more efficient grid search for the optimal EDM sampling settings, potentially enhancing the practicality of using such a sampler for other applications.

6 Related Work

The generative modeling trilemma [39] seeks generative models that produce samples (i) quickly, (ii) with high quality, and (iii) good mode coverage. Diffusion models excel at the latter two but a large amount of research has attempted to address the problem of slow sampling speed. From the inception of diffusion models [29], dynamics has been at the forefront, so most work has focused on

0

20

40

60

80

100

S

churn

2.0

2.5

3.0

3.5

4.0

FID

Stochasticity on CIFAR-10

EDM w.

S

noise=1.00

EDM w.

S

noise=1.007

CDL w.

S

noise=1.00

the interpretation of the dynamics as a differential equation that can be sped up through accelerated numerical solvers [30, 34, 42, 9, 17]. Our approach is compatible with any of these approaches as we are sampler agnostic, seeking only to improve the input to the sampler, which is the denoiser or score function estimator, through regularization during the diffusion model training. A separate line of work instead attempts to distill a diffusion model into a faster model that achieves similar sampling quality [27, 35, 21, 38, 39]. In principle, these methods could also benefit from distilling based on a more robust base diffusion model trained with CDL.

Diffusion models admit a surprisingly diverse array of mathematical perspectives, like variational perspectives [6, 7, 12], differential equations [34, 20], and nonequilibrium thermodynamics [29]. Our approach is mostly inspired by connections between the information-theoretic perspective [13, 14] and the score matching perspective [31, 33, 32, 8, 37]. In particular, we point out that score function estimates in traditional diffusion training are sub-optimal, and the information-theoretic perspective leads to a new objective (CDL) that can improve the score estimate.

While previous diffusion models focus on log-likelihood estimation, we consider a different approach based on density ratio estimation and noise contrastive estimation [4, 23], which inspired several notable developments in machine learning [1, 25]. A few works have considered contrastive learning inspired modifications to diffusion either to enforce multimodal data relationships [16, 43], for style transfer [41], or for guidance during generation [24], but none use the diffusion model as a noise classifier to improve diffusion training as we do. Most similar to our approach are methods that use Density Ratio Estimation (DRE) to estimate a ratio between the data density and some simple noise distribution. The density ratio can be estimated by learning to contrast between data samples and samples from the noisy distribution [4, 36]. Recent work generalized the idea to consider classifying between samples along a sequence of distributions between source and target [26, 2]. Our contribution is to relate this approach to diffusion models by noting that standard diffusion models implicitly already implement the required classifiers for distinguishing distributions on the path from the data distribution to a Gaussian distribution. Concurrent work makes a similar connection but while we focus on improving diffusion models by interpreting them as noise classifiers, [40] focused on the converse perspective, improving density ratio estimation by interpreting DREs as denoisers.

7 Conclusion

In this paper, we introduced a novel connection between diffusion models and optimal noise classifiers. While this relationship has a variety of potential applications that could be explored in future work, we used the connection to propose a new self-supervised loss regularizer for diffusion models, the Contrastive Diffusion Loss (CDL). CDL reduces the error of the learned denoiser in regions that are OOD for the standard loss. We showed that CDL improves the robustness of diffusion models across all types of sampling dynamics, and leads to significant speed-ups for a promising new generation of parallel samplers.

References

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Kristy Choi, Chenlin Meng, Yang Song, and Stefano Ermon. Density ratio estimation via infinitesimal classification. arXiv preprint arXiv:2111.11010 , 2021.

Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8188-8197, 2020.

Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 297-304. JMLR Workshop and Conference Proceedings, 2010.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Chin-Wei Huang, Jae Hyun Lim, and Aaron C Courville. A variational perspective on diffusion-based generative models and score matching. Advances in Neural Information Processing Systems , 34:2286322876, 2021.

Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(4), 2005.

Alexia Jolicoeur-Martineau, Ke Li, Rémi Piché-Taillefer, Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105.14080 , 2021.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364 , 2022.

Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2107.00630 , 2021.

Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information-theoretic diffusion. In International Conference on Learning Representations , 2023. URL https://arxiv.org/abs/2302.03792 .

Xianghao Kong, Ollie Liu, Han Li, Dani Yogatama, and Greg Ver Steeg. Interpretable diffusion via information decomposition. In The Twelfth International Conference on Learning Representations , 2024.

Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009. URL https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf .

Chaejeong Lee, Jayoung Kim, and Noseong Park. Codi: Co-evolving contrastive diffusion models for mixed-type tabular synthesis. arXiv preprint arXiv:2304.12654 , 2023.

Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778 , 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 , 2022.

Justin Matejka and George Fitzmaurice. Same stats, different graphs: generating datasets with varied appearance and identical statistics through simulated annealing. In Proceedings of the 2017 CHI conference on human factors in computing systems , pages 1290-1294, 2017.

David McAllester. On the mathematics of diffusion models. arXiv preprint arXiv:2301.11108 , 2023.

Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14297-14306, 2023.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Yidong Ouyang, Liyan Xie, and Guang Cheng. Improving adversarial robustness by contrastive guided diffusion process. arXiv preprint arXiv:2210.09643 , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Advances in neural information processing systems , 33:4905-4916, 2020.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations , 2021.

Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv preprint arXiv:1503.03585 , 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems , 32, 2019.

Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems , 33:12438-12448, 2020.

Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. arXiv preprint arXiv:1905.07088 , 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469 , 2023.

Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Bibliography , page 309-326. Cambridge University Press, 2012.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality.

Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In International Conference on Learning Representations , 2021.

Shahar Yadin, Noam Elata, and Tomer Michaeli. Classification diffusion models: Revitalizing density ratio estimation. 2024.

Serin Yang, Hyunmin Hwang, and Jong Chul Ye. Zero-shot contrastive loss for text-guided diffusion image style transfer. arXiv preprint arXiv:2303.08622 , 2023.

Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. Advances in Neural Information Processing Systems , 34:16280-16291, 2021.

Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan. Discrete contrastive diffusion for cross-modal and conditional generation. arXiv preprint arXiv:2206.07771 , 2022.

A Derivations and Proofs

A.1 Score Connection

We derive the following relation.

To keep track of intermediate random variables and their associated distributions, we will use the cumbersome but more precise information theory notation where capitals represent a random variable and lowercase represents values. Define the channel that mixes the signal, X , with Gaussian noise as Z α ≡ √ σ ( α ) X + √ σ ( -α ) E with E ∼ N (0 , I ) and data distribution p ( X ) , α represents the log of the Signal-to-Noise Ratio (SNR), and σ ( · ) is the sigmoid function. In this notation, the probability that a mixture distribution takes a value, x , would be written p ( Z α = x ) .

We start by re-writing the left-hand side in new notation, and expand the definition based on the noisy channel model, using δ ( · ) for the Dirac delta.

In the second line we expand, and in the third we just move the gradient inside the integral. In the fourth line we use the chain rule to relate the gradient over x to the gradient over ϵ (introducing a sign flip). In the fifth line we use integration by parts to move the gradient (second sign flip). Taking the gradient of the Gaussian in the sixth line gives our third sign flip, and the factor of ϵ . We can conclude by writing the expression in terms of a conditional distribution, and relating that to the optimal denoiser in Eq. 1.

A.2 Mixture Distribution Density

In this section, we derive the expression that shows that the density of a continuum of Gaussian mixture distributions can be written in terms of the optimal denoiser, ˆ ϵ , for the data distribution.

As in the previous section, we will adopt information theory notation. If we define the optimal denoiser for the input distribution, p ζ ( x ) , with a subscript as ˆ ϵ ζ ( · , · ) , we can write the density analogously to Eq. 2.

Note that we now have to keep track of two log SNR values. One indicates how much noise is added to the new 'data' distribution, the other is how much noise we add and then try to remove with our denoiser. The goal is to relate ˆ ϵ ζ to ˆ ϵ . We can formally write down the optimal solution using the relation in Eq. 1.

Now, however, the noise channel is defined differently. The channel mixes the signal, ¯ x ∼ p ζ ( ¯ X ) , with Gaussian noise, ¯ ϵ ∼ N (0 , I ) , as Z ≡ √ σ (¯ α ) ¯ X + √ σ ( -¯ α ) ¯ E . And the noisy variable, ¯ X = √ σ ( ζ ) X + √ σ ( -ζ ) E , where we must be careful to distinguish the two independent sources of Gaussian noise.

We start by expanding definitions.

Now do a change of variables, a 2-d rotation with:

This change of variables leads to the following.

Note in the second line that the expectation of ϵ ′ is zero, and we move the constant for the other term, b , outside the integral. In the third line, we define β which represents the log SNR of the two consecutive noisy channels with ζ, ¯ α . Then we recognize the resulting integral as Eq. 1, the optimal denoiser for recovering samples from from the original (non-noisy) data distribution in Gaussian noise.

A.3 Main Plot 1D Two-mode Gaussian's Analytical Solution

In this section, we calculate the analytical solution to the 1D two-mode Gaussian in Fig. 1.

Fig. 1 is plotting the norm of difference between the ground-truth denoiser ˆ ϵ gt ( · , · ) and the estimated denoiser ˆ ϵ ( · , · ) :

To get this error plot, we need to analytically calculate ground-truth denoiser ˆ ϵ gt ( x , α ) . From score connection Eq. 9, for any intermediate noisy density log p α ( x ) , denoiser function ˆ ϵ ( x , α ) can be derived from score function ∇ x log p α ( x ) . Therefore, ultimately what we need to calculate here is the score function of any noisy distribution p α ( x ) .

The data we used is a mixture of two Gaussians, N ( µ = -5 , I ) and N ( µ = 5 , I ) , and the noise distribution is consists of data plus noise, then the noisy distribution should also be a mixture of Guassians. We just need to relate the parameters of the noisy mixture of Gaussians to the parameters of the mixture of Gaussians.

Start with one mode of the Gaussian mixture for the data. We could represent it in terms of the standard normal random variable, ϵ .

Now call x α the random variable after applying a noisy channel with log-SNR α , here we present sigmoid function as σ ( · ) .

Note that we use a different ϵ ′ here. Now expand this, and then re-arrange.

We want to represent this in a more canonical way to see what the variance and mean of this Gaussian is. Note that for two standard normal random variables, a ϵ + b ϵ ′ , we can represent them as a single random variable with the same variance, √ a 2 + b 2 ϵ ′′ (reparameterization trick).

Now we see that the noisy Gaussian (one component of a mixture) is just a modified version of the original. We have to change the mean (moving it towards zero when adding noise) and the variance.

In our example, we set σ = 1 , so it simplifies further.

So the variance doesn't change, we just slowly shift the two mixtures together to the center.

Therefore, for one mode x ∼ N ( µ, I ) , the intermediate noisy log-density is.

Take the gradient of the log-density via torch built-in function torch.autograd.grad(log-density, samples), we have the ground-truth score function ˆ ϵ gt ( x , α ) .

B Implementation Details

B.1 Synthetic Experiment - Maximum Mean Discrepancy Bandwidth Choice

The Maximum Mean Discrepancy (MMD) is a statistical test used to determine if two distributions are different. It works by comparing the mean embeddings of samples drawn from two distributions in high dimensional feature space. Specifically, if the distributions are the same, the means should be close; if they are different, the means should be far apart. The embeddings are typically constructed using a feature map associated with a kernel function, and here we select the Gaussian kernel:

The bandwidth parameter σ of the Gaussian kernel plays a critical role in the sensitivity and performance of MMD. The better bandwidth choice, the more effective MMD computation is. Intuitively, the bandwidth σ controls the scale at which differences between distributions are detected. A small σ makes the kernel sensitive to differences at small scales (fine details), while a large σ highlights differences at larger scales.

The choice of bandwidth is often related to the variance of the data, and the bandwidth should be on the order of the variance of the data. Through a small experiment where we calculate MMD score between our data and the standard Gaussian under various bandwidths σ s, we pick the one that maximizes the MMD score.

In the synthetic 2D Dino experiment, we plot the relationship between MMD scores and bandwidths (Fig. 8), setting σ = 3 e -02 .

10

3

10

2

10

1

10 0

bandwidth

0.000

0.005

0.010

0.015

0.020

mmd values

MMD between Data and Standard Gaussian

B.2 Details on Model Training

Model Checkpoints We adopt two models as baselines to fine-tune with CDL: DDPM model provided by Ho et al. 2020 and EDM model provided by Karras et al. 2019.

For DDPM model, the checkpoint 8 we used is a ema one pre-trained on unconditional CIFAR-10. The reason we are not using the most frequently used checkpoint ( https://huggingface.co/ google/ddpm-cifar10-32 ) is that, this is not EMA checkpoint and our calculation of the FID score of this model on 50 , 000 generated image via sequential DDPM sampler gives 12 . 43 . This FID score is much higher than what reported on the original paper 3 . 17 . Here we provide the FID scores of this non-EMA pre-trained model, results shown in Tab. 4.

For EDM model, in total eight checkpoints we used are 'DDPM++ cont. (VP)' and 'NCSN++ cont. (VE)' models pre-trained on three datasets (CIFAR-10, uncond-FFHQ, and uncond-AFHQv2 [15, 10, 3]) with two training settings (unconditional and conditional) 9 10 11 12 13 14 15 16 .

As for fine-tuning, we train all models with the same training setting in their original papers. For DDPM model, we train each model for 10 epochs and keep 'learning rate / batch size' ratio to be ' 10 -4 / 64 ', and this training is on two A6000 GPUs. For EDM model, the following table list the exact our fine-tuning configurations, which is still of the same 'learning rate/batch size' ratio. This training is using eight V100 GPUs.

More Experimental results with Parallel DDPM Sampler The previous parallel diffusion sampling paper [28] calculates FID scores by using 5 , 000 generated images and another 5 , 000 randomly selected real images, and to follow the same experimental setting for comparison, we further provide the FID results in Table 6.

More Details on EDM Fine-tuning As the design choices of EDM model is very comprehensive and complicate, here we list the training noise distribution, loss weighting, network and preconditioning choices we make during CDL fine-tuning in Tab. 7.

B.3 Relationship Among Log-SNR, Timesteps, and Noise variance Sigma

To use the pre-trained models in the literature with our CDL loss, we need to translate " t ", a parameter representing time in a Markov chain that progressively adds noise to data in Ho et al. 2020 and Song et al. 2020, or " σ ", the variance scale of the Gaussian noise in Karras et al. 2022, to a log -SNR " α ".

Translation between Timesteps and Log-SNR For time-step t in DDPM and stochastic diffusion notation, we recommend readers check Kong et al. 2023 Appendix B.2 about the mapping between α and t .

Translation between Noise Variance Sigma and Log-SNR For variance scale of the Gaussian noise σ in EDM, referring to Eq.(7) and (8) in Karras et al. 2022, it's easy to translate the preconditioning:

From Eq. 18 and Eq. 17, we see that σ ≡ √ σ ( -α ) σ ( α ) , therefore, the relationship between α and σ should be:

B.4 Contrastive Loss Implementation

To implement contrastive loss, we follow the definition in Sec. 3. First, we generate a random binary label y . Next, conditioned on y, we sample from either data distribution p ( x ) or the noisy data distribution p ζ ( x ) . We calculate the point-wise log-likelihood, then the contrastive loss in Eq. 7.

Algorithm 1 Contrastive Diffusion Loss - Training

B.5 Training cost

As we mentioned, CDL training is more expensive to compute than the standard diffusion loss, and here we analysis and give the reason.

According to Kong et al. 2023, we can write the pointwise standard diffusion loss function as Eq. 19, and therefore the standard diffusion loss is as Eq. 20.

To train the standard diffusion loss, we simply need to optimize Eq. 20 by all the training data. However, to train the contrastive diffusion loss, we need to estimate nll ( x ) and nll ( x + ζ ) term in Algo. 1, and there we estimate Eq. 19 by duplicating a single data point x for N = 100 times and calculate Eq. 20. This pointwise NLL estimation nll ( x ) demands N = 100 times more computational resources compared to the standard diffusion loss.

In principle, the contrastive loss Algo. 1 should be executed for the entire training dataset. However, due to the high computational cost, we optimize only one data point per batch instead of utilizing all the training data.

C Samples Visualization

We provide visualization of the images generated from pre-trained models fine-tuned via CDL loss.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We explain the proposed CDL speeds up parallel sampling convergence and improves generation quality. For sequential sampling, it improves the trade-off between sampling quality and gerneration speed. We provide the claims in both abstract and introduction, and justify them in the Results section.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: The main limitation to our approach is the increased training cost. We explicitly highlight limitations in the methods development in Sec. 3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Every theoretical result is supported by a complete proof and assumptions as well, if applicable.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide anonymous code link and full description of training configuration. Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide anonymous code link and full description of training configuration. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all detailds about training and testing details in both experimental section and also in appendix. Since we test on benchmark, our data split are rigously the same as all other baseline settings.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars wherever possible. One set of error bars for Table 2 is missing due to lack of computing resources, but will be added in the final version.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide sufficient information on the computation resources, e.g. GPU information, to reporduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We do not use human subjects and only used standard datasets for experiments. We acknowledge the potential harms of diffusion models in the broader impacts statement, and do not anticipate that our methodological contributions will alter known issues. We also strive for transparency wherever possible, including full, anonymized code with our submission.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: While we do not expect a direct societal impact from our methodological improvements, we added a broader impacts statement that acknowledges the potential impact of diffusion models.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We used existing standard datasets and public, pretrained models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We give proper attribution for the open license code, data, and models used throughout the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets are provided.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or human subjects research is included.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method to improve the parallel sampling of the diffusion model by improving the denoising network for out-of-distribution evaluation. The paper proposes to finetune a trained model using the log-likelihood ratio of a sample at two different noise scales. The log-likelihood ratio is obtained by integrating the denoising error at all noise scales. The proposed method performs better for parallel sampling and performs comparable for sequential sampling on the CIFAR-10, AFHQ, and FFHQ datasets.\n\nSTRENGTHS:\n- The paper is overall well-written and easy to follow. \n- As the contrastive diffusion loss is expensive to compute, the authors show that finetuning on pretrained model is sufficient for few epochs instead of training using contrastive diffusion loss (CDL) (or linear combination of CDL and denoising error) through out the training.\n\nWEAKNESSES:\n- The contrastive diffusion loss (CDL) contains the integration of denoising error over all noise scales. In practice, I assume that the integration will be replaced by a numerical integration therefore, evaluating the proposed CDL loss is expensive. Considering the training/finetuning using CDL loss is expensive, the benefits from the training don’t seem to be significant in the empirical performance. Additionally, very few/no details are provided on how the integration is approximated in experiments/practice.  \n- Some discussions about the related works need to be included. For example, another work on improving parallel [1] is not mentioned/discussed/compared against. Additionally, the derivation of the CDL loss is somewhat similar to some diffusion classifier papers (for example, [2, 3, 4]) therefore they should be cited/discussed. \n\n[1] Accelerating Parallel Sampling of Diffusion Models\n\n[2] Your Diffusion Model is Secretly a Zero-Shot Classifier\n\n[3] Your Diffusion Model is Secretly a Certifiably Robust Classifier\n\n[4] Robust Classification via a Single Diffusion Model\n\nQUESTIONS:\n- It is not clear why only 5k samples are used for calculating FID for parallel sampling. The standard practice is to use 50k samples. As the FID depends on the number of samples, it is hard to make sense of the significance of improvement. \n- Can you experimentally show/quantify the improvement of the denoiser for out-of-distribution evaluation when trained using contrastive diffusion loss (CDL)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper establishes a connection between the diffusion model denoiser and noise classifier through an examination of log-likelihood ratio estimation. The authors introduce a novel loss function, termed Contrastive Diffusion Loss (CDL), designed to encourage diffusion models to explore OOD regions in noise levels. Empirical results indicate that the proposed loss function enhances performance in parallel sampling and demonstrates robustness to different hyperparameters in sequential sampling.\n\nSTRENGTHS:\n* The paper draws an interesting connection between diffusion model denoiser and noise classifier and introduces a novel loss function based upon. \n* The paper is well-organized and most results are clearly presented.\n\nWEAKNESSES:\n* The clarity of the paper could be further enhanced. For instance, in line 86, the authors introduce the notion $\\zeta$ to indicate the noise level, while in Appendix A.2, they use $\\alpha$ during the derivation. It seems like the authors want to differentiate between noise level and SNR, but the message is not entirely clear here.\n\n* The empirical section of the paper is not as strong as the analytical section. In the parallel sampling example, the method shows good performance, but even with the performance improvement, the results are still far from optimal compared to EDM with the default settings. The experiment on deterministic samplers aims to demonstrate that CDL-regularized models maintain a more stable FID score with changes in NFE, but the authors do not explain why CDL helps in this context. The same issue applies to the experiment on stochastic samplers.\n\nOverall, I am on the positive side for this paper. I would appreciate it if the authors could provide more intuitive explanations to better connect the analytical and empirical sections of the paper.\n\nQUESTIONS:\nWhy did the authors choose fine-tuning with the CDL over pre-training and using CDL as a regularization?\n\nLIMITATIONS:\nThe authors addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBuilding on the denoising score matching loss, the paper introduces a novel contrastive learning loss. This loss estimates the log-likelihood ratio between mixture densities with varying noise levels. The contrastive learning loss is then employed to fine-tune the diffusion model, enhancing its ability to estimate out-of-distribution (OOD) data. To validate this approach, the paper demonstrates experimentally that the fine-tuned diffusion model exhibits improved performance in both sequential and parallel sampling, which typically degrade with poor OOD estimates.\n\nSTRENGTHS:\nThe derivation of the contrastive diffusion loss is solid and novel. The CDL could potentially inspire more contrastive loss and noise-level data augmentation methods for diffusion model training. Additionally, CDL introduced the asynchronous data pair $(z_\\alpha, \\beta)$ for training, improving the robustness of the out-of-distribution region, especially during sampling. Overall, the idea is very sound and novel.\n\nWEAKNESSES:\nThe experimental results are not particularly impressive:\n\n1. Although CDL fine-tuning consistently enhances parallel sampling performance on both synthetic and real-world datasets, the improvements are marginal. Additionally, the baseline chosen for comparison is not state-of-the-art. For instance, in the unconditional generation in Table 2, even with CDL fine-tuning, an FID of around 7.0 on CIFAR-10 falls short of the state-of-the-art FID, which is less than 2.0. The improvement in sequential generation is even less substantial. \n\n2. I believe the proposed CDL has significant potential. For instance, when addressing inverse problems, the diffusion model also encounters OOD issues. The paper could benefit from including more promising experiments to demonstrate how CDL enhances the diffusion model's OOD estimation capabilities.\n\nQUESTIONS:\n1. I think there are issues with the chosen baseline in the paper. In the original parallel sampling method [27], parallel sampling achieves similar performance to sequential sampling in significantly less time. However, the results reported in Table 2 show that unconditional parallel sampling performs much worse than sequential sampling (around 3.0 for the original DDPM, VP, and VE). Could the author provide more clarification on this discrepancy?\n\n2. According to [1], the denoising score matching loss serves as an upper bound for the negative log-likelihood. So minimizing the denoising score matching loss is equivalent to maximizing the negative log-likelihood.  Are there any insights or proofs provided on why minimizing the CDL would also lead to meaningful generation?\n\n\n[1] Song, Yang, Conor Durkan, Iain Murray, and Stefano Ermon. \"Maximum likelihood training of score-based diffusion models.\" Advances in neural information processing systems 34 (2021): 1415-1428.\n\nLIMITATIONS:\nThe authors have covered the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-29"}