Classification Diffusion Models: Revitalizing Density Ratio Estimation
Authors: Shahar Yadin
Noam Elata
Tomer Michaeli
Venue: NeurIPS 2024 poster (2024)
Keywords: Density Ratio Estimation
Primary area: generative_models

Abstract: A prominent family of methods for learning data distributions relies on density ratio estimation (DRE), where a model is trained to *classify* between data samples and samples from some reference distribution. DRE-based models can directly output the likelihood for any given input, a highly desired property that is lacking in most generative techniques. Nevertheless, to date, DRE methods have failed in accurately capturing the distributions of complex high-dimensional data, like images, and have thus been drawing reduced research attention in recent years.  
In this work we present *classification diffusion models* (CDMs), a DRE-based generative method that adopts the formalism of denoising diffusion models (DDMs) while making use of a classifier that predicts the level of noise added to a clean signal. Our method is based on an analytical connection that we derive between the MSE-optimal denoiser for removing white Gaussian noise and the cross-entropy-optimal classifier for predicting the noise level. Our method is the first DRE-based technique that can successfully generate images beyond the MNIST dataset. Furthermore, it can output the likelihood of any input in a single forward pass, achieving state-of-the-art negative log likelihood (NLL) among methods with this property.

Classification Diffusion Models: Revitalizing Density Ratio Estimation
Shahar Yadin
Noam Elata Tomer Michaeli
Faculty of Electrical and Computer Engineering
Technion Israel Institute of Technology {shahar, yadin@campus, noamm noame nwwmelataacama ata@campus, tomer .m@ee) technion. .ac.aaccc. .il
Abstract
A prominent family of methods for learning data distributions relies on density ratio estimation (DRE), where model trained classify between data samples and samples from some reference distribution. DRE-based models can directly output the likelihood for any given input, highly desired property that lacking most generative techniques. Nevertheless, to date, DRE methods have failed in accurately capturing the distributions of complex nigh-dimensional data, like images, and have thus been drawing reduced research attention in recent years. In this work we present classification diffusion models (CDMs), DRE-based generative method that adopts the formalism of denoising diffusion models (DDMs) while making use of a classifier that predicts the level of noise added clean signal. Our method is based an analytical connection that we derive between the MSE-optimal denoiser for removing white Gaussian noise and the cross-entropy-optina. classifier for predicting the noise level. Our method is the first DRE-based technique that can successfully generate images beyond the MNIST dataset. Furthermore, it can output the likelihood of any input in single forward pass, achieving state-of- the-art negative log likelihood (NLL) among methods with this property. Code is available on the project's webpage.
Introduction
A classical family of methods for learning data distributions relies the concept of density-ratin estimation (DRE) [46]. DRE techniques transform the unsupervised task of learning the distribution of data into the supervised task of learning to classify between data samples and samples from some reference distribution [15, 4, 35, 7]. These methods have attracted significant research efforts over the years [27, 14, 35, 47], particularly for their inherent capability directly output the likelihood for any given input. However, to date, they have not succeeded in capturing the distribution of complex high-dimensional data, like natural images. Instead, their generative performance was demonstrated only on low-dimensional toy examples and on the simple MNIST handwritten digits dataset [28. 35, 7]. As illustrated Fig. while the sfate-of-the-art DRE method, telescoping density-ratio estimation (TRE) [35], succeeds in capturing the distribution of the MNIST dataset [28], fails on the slightly more complex CIFAR-10 dataset [26].
As opposed to DRE methods, denoising diffusion models (DDMs) [38, 19] have had unprecedented success in generative modeling of complex high-dimensional data, including images [9, 36), audio [25, 5]. and video [12, 1]. This has made them perhaps the most prominent technique for learning data distributions in recent years, with applications in solving inverse problems [21, 37], image editing [33, 16, 3, 20] and medical data enhancement [44, 8], to name just a few. However, assessing the likelihood of data samples is challenging task with DDMs; it requires many neural function
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

CDM (ours)
TRE
D
6
9
0
9
S
9 6
Figure I: Samples from CDMs (left) trained on CelebA 64 x 64 and on CIFAR-10, compared to samples from TRE models [35] (right) trained on MNIST and CIFAR-10. To date, DRE methods have failed to capture the distributions of complex, high-dimensional data, and have been demonstrated only on toy examples or on the simple MNIST dataset. The right pane shows results from TRE, the state-of oh-the-art DRE method, which fails to capture the distribution of CIFAR-10. CDM the first DRE-basee method that can successfully learn the distribution of images.
evaluations (NFEs) to calculate the likelihood-EL.B) [19), or to approximate the exact likelihood using an ODE solver [43].
DDMs are based on minimum-MSE (MMSE) denoising, while DRE methods hinge on optimal classification. In this work, we develop connection between the optimal classifier for predicting the level of white Gaussian noise added to a data sample, and the MMSE denoiser for cleaning such noise. Specifically, we show that the latter can be obtained from the gradient of the former. Utilizing this connection, we propose classification diffusion model (CDM), generative method that combines the formalism DDMs, but instead of a denoiser, employs a noise-level classifier. CDM is the first instance of a DRE-based method that can successfully generate images beyond MNIST (Fig. 1). In addition, as a DRE method, CDM is inherently capable of outputting the exact log-likelihood in a single NFE. In fact, it achieves state-of-the-e negative-log-Ble likeliiel (NLL) results among methods that use a single NFE, and comparable results to computationally
ODE-based methods.
Our experiments shed light on the reasons why DRE methods have failed on complex high- dimensional data date, and why CDM inherently avoids these challenges. Furthermore, we show that CDM can serve as more accurate denoiser, in terms of MSE, than a DDM with a similar architecture. This typically translates into better FID scores. Representative generated samples are shown in Fig. 1. We hope that our approach will spark new interest in DRE methods and ultimately unlock their full potential.
Background
2.1 Density Ratio Estimation
Learning data distributions via DRE was first proposed by Gutmann and Hyvärinen [15]. Their noise contrastive estimation (NCE) method uses the fact that the ratio between an unknown distribution pa(x) and a known reference distribution Pn (z) can be extracted from the optimal binary classifier for discriminating samples from pa( and (x). Once this ratio is extracted from the classifier, it can be multiplied by the known pn (z) to obtain Pa (x) Specifically, let denote the class of a sample æ, with C' = 1, 0 corresponding
the event that æ is a sample from pa(x), P., (x), respectively. The optimal classifier for predicting C from x outputs both P(C = 1[x) and IPC = 0/æ:). Using Bayes' rule, we use these values compute the density ratio
Pa (z)
PP(C [
(I)
Ph. x
PCC
0|æ)
where we assumed that the classes balanced, so that P(C 1) PCC 0) =

---

Unfortunately, this method fails in practice when pa(x)) and Pa (x( differ significantly from one another, as is the case when Pa(æ) the distribution of images and Pr. (x) corresponds to white Gaussian noise. This is because when training a classifier to discriminate between images and noise, the classifier can achieve very high accuracy even without learning meaningful information about images. When this point is reached, the weights of the classifier practically stop updating. Rhodes et al. [35] referred to this issue as the density-chasm problem, and suggested to overcome it by making the classification problem more difficult. To do so, their TRE method uses sequence of distributions Pxo (zr), Pxx, (z),.... Px... (z), which are closer to one another, such that Px... (a)) is the reference distribution and Pxo (z) is the target distribution. The intermediate distributions (px; (xx ]= do not have to be known; the only requirement is that it would be possible to sample from them. For example, x; can be defined as x; = Vā,Xo + VI- a,Xms where Xo ~ Pxx Xm ~ px.. and āg is a sequence that decreases from I to O. Then, using (I), each ratio Px, (a)/Px... (x) can be extracted by training a binary classifier to distinguish between samples from Px, (z:) and Px.+1 (z)), and the ratio between the target and the reference distributions can be calculated as
Pxo (x:) Px (x)
Pioo (2))
Pxc, (x:)
Pxm 2 (a:)
Px (x)
Pxp (x))
Pxg (x)
Pxm
(a)
Px..
While this method overcomes the denssty-chhumm-prrblem for each pair of consecutive distributions, still fails in learning the distribution of datasets that are more complicated than MNIST, as illustrated in Fig. 1. This is because each ratio Px. (z)/Px.., (x) is extracted from a binary classifier trained only on inputs from the distributions Px, and Px.+: For instance, the classifier producing the ratio Pxo (px) (a:) is trained on inputs close to the real data, while the one producing Px_- (ar) (1x) is trained on inputs close to the reference distribution. This can lead to mismatch between training and test time, since inference, all the ratios are evaluated the same input æ. Moreover, even each individual ratio is nearly accurate, the accumulation of small errors can result in significant overall error. Our method is also based on a classification objective, however it avoids these problems by employing additional loss, which is based our main result (Theorem 3.
2.2 Denoising Diffusion Models
DDMs [38, 19], are class of generative models that sample from a learned target distribution by gradually denoising white Gaussian noise. More formally, DDMs generate samples by attempting to reverse forward diffusion process with T steps that starts from data point X and evolves as X; Va,x,-1 + VT- at&t, t = 1..... T, where } are iid standard Gaussian vectors. Samples along this forward diffusion process can be equivalently expressed as
X Varxo vI atSt, Et ~ (0,11) I),
where āg 1==1 ag.. The coefficients } are taken to be such that {ā²} is monotonic sequence with är R 1. This enforces the density Px, to be close to the normal distribution N (V,0 I).
The reverse diffusion process is learned by modeling the distributions of x;-1 given X Gaussian with mean
1
1- ar -00000
E E|x--1 [xx]
x:
Vat
VI-a az
and covariance III, where co(-..) is neural network and {or} are fixed hyperparameters. Training done by minimizing the ELBO loss, which reduces to a series of MSE terms,
(()) Σ Exo.es [[]Eo(Xe, f) et112]
(5)
At convergence to the optimal solution, the neural network approximates the timestep-dependent posterior mean
=o( t) = E[sy/x; = xz)-
(6)
To generate samples, DDMs sample XT ~ (N. I) and then iteratively follow the learned reverse probabilities, terminating with a sample of xg. In more detail, at each timestep f, the model accepts x, and outputs a prediction of the noise e, (equivalently, prediction of the clean signal xp), from which x,-1 is obtained by sampling from the reverse distribution. The process described above is that underlying the DDPM method [19]. Here we also experiment with DDIM [39] and DPM-Solver [32], which follow a similar structure.
3

---

Denoising Diffusion Model
Classification Diffusion Model
x
Classifler
Figure 2: A diagram of CDM (right) compared with DDM (left). A DDM functions as an MMSE denoiser conditioned on the noise level, whereas CDM operates as classifier. Given a noisy image, a CDM outputs a probability vector predicting noise level, such that the 1-th element in this vector is the probability that the noise level of the input image corresponds to timestep t in the diffusion process. A CDM can be used to output the MMSE denoised image by computing the gradient of its output probability vector w.r.t the input image, show in Theorem 3.1.
Method
We start by deriving a relation between classification and denoising, and then as the basis for our CDM method. A summary of the notations we use can be found in App. A.
Let the random vector X, be defined in (3) for timesteps E (1...., T and set two additional timesteps, and + corresponding to clean images and pure Gaussian noise, respectively. Namely, define āg 1 and .
We denote the density of each x; by px, (ar). Our approach based on training classifier that takes as input a noisy sample x, and predicts its timestep t. Formally, let be discrete random variable taking values in {0, 1, ...... T 1}, with probability mass function (t) Pt = t), and let the random vector x be the diffusion signal at a random timestep t, namely' x,. Note that the density of each x, can be written as Px. (x) (x) (x)() (t) and by the law of total probability, the density of & is equal to
T+1 Σ Px. (x) 1=0
(x
We are interested in classifier for predicting from x. It is well known that given any sample drawn from (7), the optimal such classifier (in terms vector (pijz (0 x), Pij [[)x) i........ PTx(T' + 1|x)). where (t)x(t)z) IP( f x). As we now
cross-entropy loss) outputs the probability
show, the denoiser in (6) corresponds to the gradient of this classifier.
Theorem 3.1. Let F(x, (og(pis 1[x)) (o)((((((())))) with t, x and x, defined above. Then
E| = xx] = vI (V. F( t) + xe)
(8)
regardless of the choice of the probability mass function Pa provided that p(( > ffr all
The proof, provided in App. B. consists of three key steps:
Using Bayes rule, we write Px, (x) as a function of Pxr+1 (x) and the optimal classifier.
Then, we take the derivative of the log of both sides and use the fact that Va log Pxy+ (ar) has a closed form solution.
Finally, we use Tweedie's formula [34, 45, 11] to connect between V log (x) and [[(e]|x
Theorem 3.1 suggests that may train classifier and use its gradient as denoiser according to relation (8). This paradigm illustrated in Fig. Once we have constructed denoiser, we can apply any desired sampling method (e.g. DDPM, DDIM, etc.) to generate images from the learned distribution. However, as we show in Sec. 4.1, naively training such classifier with the standard
Note the distinction between the notations and x.. The former has random noise level t, while the latter has fixed noise level

---

Algorithm CDM Training
Algorithm 2 DDPM Sampling Using CDM
Require: Dataset of training samples D
Require: Noise level classifier fo(-)
1: repeat
1: sample 2T NN(0,
2: To ~ t~ (O,......,TT11 (N(1))
2: for f = ...... do
3: E; Vā, + VI 04E
3:
Fo(2,, t) = Fo (ax )T + - fo(
4: Fo(xr, =
+ (xx)
4:
Ea( = vI dg (V Fo t) xz)
5: Ea( vI ar (Va, Fo (zr, 1) we)
5: if then
(0, I), else 2
6: take gradient step on
2(-1= vou (xr vI-a
1-0 s0 t)) +
7:
Weefce(1, Jo (x,)) + (e, e0 (2(
7: end for
8:
until converged
8: return 270
cross-entropy (CE) loss leads to poor results. This is because a classifier may reach a low CE loss even without learning the correct probability ptx(4(ar) for any l. This phenomenon can be observed in Fig. 3, which illustrates the reason that existing DRE methods fail to capture the distribution of high-dimensional complex data like images. We discuss this in more detail in Sec. 4.1.
To obtain the correct probability (t)()) for any f. we suggest training the classifier with a com- bination of a CE loss on its outputs, and an MSE loss on its gradient, according to relation (8). The network's gradient can be efficiently computed using
Following Ho et al. [19], we use the same weight for all timesteps in the MSE loss. Our full training scheme is described in Algorithm 1. Here, fo
denotes the model's t-th logit, which serves as an approxi- mation for log Pt x (f|z:) (up an additive constant that cancels out in the SoftMax operation), and Fa(at () = [7 + 1] [o(x>(()) [4]. The added timesteps. corresponding to entries 0 and T + 1 of the classifier,
trained only using the CE loss. This is because the prediction of the noise is trivial when t = T + and meaningless when t = O (since there is no noise). Importantly, this behavior automatically achieved without any modification to the algorithm. Specifically, in line of the algorithm, F& (IT+1, T + 1) = O, and in line 5, sq (20, 0) = O, preventing the MSE loss from updating the weights for these timesteps.
Algorithm 2 shows how to generate samples with CDM using the DDPM sampler (a similar approach can be used with other samplers). Note that each step in DDPM sampling using CDM is given by
ai Va, Fo(x,, t) + 0:2;
xt- = Vatxi
(9)
where = ~ (O, Therefore, each step steers the process in the direction that maximizes the probability of noise level while minimizing the probability of noise level T 1. This can be thought of as taking a gradient step with size (1 az)/ (e)///q followed by step in a random exploration direction with magnitude o,, similarly Langevin dynamics.
3.1 Exact Likelihood Calculation in a Single Step
To compute the likelihood of a given sample, DDMs are required to perform many NFEs in order to compute lower bound the log likelihood using the ELBO [19], can approximate the exact likelihood [43] using an ODE solver based on repeated evaluations of the network. In contrast, as a DRE-based method, CDM is able calculate the exact likelihood in single NFE. In fact, CDM can compute the likelihood w.r.t. the distribution Px, of noisy images, for any desired timestep t. Specifically, we have the following (see proof in App. B.3)
Theorem 3.2. For any E {0, 1..,, T 11.
(T + 1) (t)
Pt (1)())
Px. (m))
PTx + +1/x) N(,;0,
(10)
where (x; is the probability density fanction of a standard multivariate Gaussian distribution.
Note that the first term in (10) only depends on the pre-selected probability mass function pt (which we choose to be uniform in our experiments), and the second term can obtained from the I-th and (T' + 1)-th entries of the vector at the output of the classifier (after applying SoftMax). This implies
5

---

t-200 t-400
t-500 t-800
-290
(P(()))))))
-13 -22
-320
-19
-30
thout
125
12:0
Figure 3: Comparison between the log probability of noise-level classifier trained using the CE loss alone and model trained using CE and MSE. Since the SoftMax operator is invariant an additive factor, we subtract the maximal value from the vector (i.e., fo( <- fo(=:) -0ax(mma(()))) fo (x;)) for visualization. We utilize the connection we developed between the optimal classifier and the MMSE denoiser to incorporate the MSE loss in DRE training, as depicted in Algorithm 1. As evident, without considering MSE, the prediction accuracy of the classifier limited to the vicinity of the correct label, unlike the model trained using both CE and MSE, which yields accurate predictions globally. The essence of an optimal classifier lies in its capability to predict the correct probability vector for all entries, rather than solely for the correct label. As can be seen, this necessitates the incorporation of the MSE loss.
that we can calculate the likelihood of any given image æ. w.r.t. to the density Px, of noisy images for any noise level f. In particular, Px, is the density of clean images. Thus, by choosing & = 0 we can calculate the likelihood of clean images. In App. D we present a simple validation of our efficient likelihood computation by applying it toy examples with known densities, allowing us to compute the analytical likelihood and verify that our computations align with theoretical expectations.
4 Experiments
We train several CDMs on two common datasets. For CIFAR-10 [26] we train both a class conditional model and unconditional model. We also train similar model for CelebA [31], using face images of size 64 64. In Sec. 4.1, we demonstrate why existing DRE methods fail on complex high-dimensional data like images, and show how the incorporation of the MSE loss in method, according to Theorem 3.1, overcomes these challenges. In Sec. 4.2, we compare our method with pre-trained DDMs with similar architectures, disentangle the benefits of our method from other variables. We evaluate the performance of CDM as denoiser, assess its generation quality using FID [17], and measure its likelihood modeling capabilities using NLL. Finally, in Sec. 4.3, we demonstrate the use of different noise schedulers, one specifically tuned for likelihood estimation, and one corresponding
the flow matching incorporating these schedulers into our method leads to state-of- ofte-oo--heeare capable of outputting the likelihood a single forward pass.
-trassoot scheme [29, 30]. We show that
the-art art NLL results among methods
4.1 The Importance of Using Both Losses for Achieving an Optimal Classifier
In Theorem 3.1, we established that the MMSE denoiser corresponds to the gradient of the optimal noise-level classifier. A natural question is whether we can train our model only with the MSE loss. Unfortunately, the answer is negative. This is because the MSE achieved by the model does change if we add a function of & to its output, as such an additive term vanishes when taking the gradient with respect to æ. The CE loss important for removing this degree of freedom. Namely, without the CE loss, the model can function as denoiser but is useless for the purpose of outputing the likelihood in a single step.
Can we train the model only with the CE loss, then? In theory, training the model only with the CE loss should be sufficient. However, practice for achieving an optimal classifier.
will demonstrate next, incorporating MSE is crucial in
Table reports the MSE, CE and classification accuracy achieved by models trained with different losses. We emphasize that the model trained using only MSE in this comparison, is CDM model trained using Algorithm I, and is not equivalent DDM model trained using (5). As evident from

---

Table 1: The importance of using both losses in CDM. We demonstrate the importance of using both the CE and MSE losses at training. We report the results for CIFAR-10 test-set. FID is reported on 50k samples which were generated using DDIM scheduler with 50 steps. As shown by Rhodes et al. [35], to avoid the density-chusm problem, the classification problem should be sufficiently hard
to avoid trivial classifier solutions. This leads to low classification accuracy results.
TRAINING CLASSIFICATION Acet
CE J
MSE
FID 4
NLL 4
LOSs
CE
6.97%
4.49
0.225
329
8.27
MSE
0%
16.59
0.028
7.65
9.32
BOTH
8.34%
4.37
0.028
7.56
3.38
the table, when using only the CE loss, the MSE is high, and when using only the MSE loss the CE and classification accuracy are poor. An important point to notice is that even when training with the CE loss, the classifier's accuracy is rather low (though greater than the 0% achieved when training only with the MSE loss). This is a key prerequisite for making DRE methods work. Specifically, as shown in [35], the classification problem should be sufficiently hard in order avoid the density- chasm problem, otherwise the classifier can easily discriminate between the classes even without having learned the correct density ratio. Yet, as we illustrate next, only making the classification problem harder is still insufficient for learning the probability (r)((1())) with only the CE loss.
Figure 3 shows the logits fox for noisy images with different noise levels, comparing a model trained using CE to model trained with both CE and MSE. As can be seen, in both scenarios the prediction near the true label the same, namely the CE works well in the vicinity of the correct noise level. However, the model trained without the MSE loss exhibits significantly higher predicted logits for more distant noise levels compared to the model trained using both CE and MSE. Moreover, the logits of the model trained without MSE do not decrease monotonically the distance from the actual noise level increases, which is in contrast with the expected behavior. This demonstrates the importance of the MSE loss for obtaining good prediction globally. As can be seen in Theorem 3.1, the denoiser at timestep depends on the predictions of the classifier in both the f-th and the (T' 1)-th entries. Therefore, the addition of the MSE loss enforces the classifier to achieve accurate predictions in both entries, thereby ensuring accurate predictions globally.
4.2 Denoising Results, Image Quality and Negative Log Likelihood
We compare our method with pre-trained DDMs of similar architectures. Since CDM is classifier and DDM is a timestep-conditional denoiser, we take the architecture of our CDM to be identical to the DDM, except for altering the last two layers to output vector of logits, and removing all timestep conditioning layers. These changes have negligible effect on the number of parameters in the model. For more details please refer to App.
As shown in Fig. 4, the denoising performance of our CDM surpasses that of pre-trained DDMs at high noise levels, and is comparable to them at lower noise levels. These quantitative results are corroborated by the qualitative examples in Fig. 5, which showcase image denoising results across various noise levels.
The good denoising performance of CDM translates into high quality image generation. This is illustrated qualitatively in Fig. 1, which shows samples from models trained on CelebA and on CIFAR-10 (unconditional). To quantitatively compare the generation quality of CDM to that of pre-trained DDMs, we 50k FID [17] against the train-set. For both CDMs and DDMs, we compare images sampled using the DDPM [19], DDIM [39], and DPM-Solver (DPMS) [32] samplers, using 1000, 50, and 25 timesteps, respectively. The results, shown in Table 2, demonstrate that CDM is at least comparable to pre-trained DDMs in image quality, outperforming them in most cases.
Additionally, we evaluate CDM's effectiveness in applying classifier-free guidance (CFG) [18] for conditional sampling tasks. As expected, incorporating CFG improves image quality beyond unconditional generation, reflected in the conditional 10 FID results of Table 2. More details and qualitative results are provided in Appendix C.5. These results showease the effectiveness of CDM for image generation, showing it to be equal or better than similar DDM.
7

---

CIFARIO
CalebA 64 x
DDM CDM
DDM
1=000
1-700
1-800
t-500
t=1000
logElle E11
CDM
Imagg
-4-
Nolay Imaage
Ground Truth
200
100
600
205
603
100
300
Meen Imsege
CDM
log Elli Raove]
CDM
235 500
Figure Denoising results. The figure depicts comparison between denoising results on the CelebA dataset for several different noise levels, obtained with CDM and with pre-trained DDM with the same architecture. The right column shows the models' predictions for pure Gaussian noise, which should theoretically be the expectation of the prior distribution. As observed, DDM outputs highly noisy image, whereas CDM generates an image much closer the mean of the dataset.
Figure 4: Denoising performance. The plots show the MSEs (top) and the ratio be- tween the MSEs (bottom) achieved by a pre- trained DDM and by CDM with the same architecture, as a function of the noise level (timestep 1). The CDM significantly outper- forms the pre-trained DDM at high noise lev- els, while demonstrating comparable perfor-
mance at lower noise levels.
Finally, calculate liglliikssooods and compare NLL results recent leading methods in Table CDM demonstrates comparable performance on NLL estimation for CIFAR-10 compared DDMs. Notably, CDM stands out more efficient method than existing ones, requiring only a single forward pass for NLL computation. Table 3 also includes CDi(f,if.), and CDM(OT) on which elaborate in Sec. 4.3 below. These variants of model, improve the NLL predictions and achieve state-of-the-aae results among methods requiring a single step.
4.3 Different Noise Scheduling for Better Likelihood Estimation
To see the effect of using timestep scheduler tuned for DRE tasks, we repeat the CIFAR-10 unconditional experiment, with a different noise scheduler. Following Rhodes et al. [35] we use the scheduler defined /1
- and choose T 1000 similarly to our previous experiments. Utilizing this scheduler, we achieve a better NILL of 2.98 at the expense of higher FID of 10.28, when using the DDIM sampler with steps. This trade-off highlights that the scheduler optimal for learning the data distribution may not be ideal for sampling.
To further explore the influence of the noise scheduler, we train and evaluate a CDM with the flow- matching optimal-transport (OT) scheduler [29, 30] on unconditional CIFAR-10. In this scheduler, x, T Xy + Et, where E ~ NNO,I) and t {0. Po.....Ty T). This scheduler leads to state-of the- art single-step NLL of 2.89 and to an FID of 7.07 with 1000 sampling steps. Please see App. B.4 for more details.
Future research could explore schedulers aimed at further enhancing the NLL. Further analysis of the difference between the schedulers from classification perspective be found in App. E.2
5 Related Work
Using the concept of DRE for learning data distributions was initially studied by Gutmann and Hyvärinen [15]. Their noise contrastive estimation (NCE) method approximates the ratio between the density of the data distribution and that of white Gaussian noise. However, it struggles in practical scenarios where the gap between these distributions is large, is the case for natural images [35]. Conditional noise contrastive estimation (CNCE) [4] is slightly improved version of NCE, in which

---

Table 2: Image generation quality. We compare the FID (lower is better) achieved by a DDM and CDM using three sampling schemes for CelebA and CIFAR-10 10. For conditional CIFAR-10 we train a DDM ourselves, as no model in the original im- plementation [19] supports CFG.
Table 3: NLL (bits/dim) calculated on the CIFAR-10 test-set. For each model we specify the number of NFES required for calculating the NLL. CDM achieves state-of-the art NLL
among methods that use a single NFE.
MODEL
NLL 4
NFE
SAMPLING METHOD
MODEL
IRESNET [2]
3.45
100
FFJORD [13]
3.40
~3K
CELEBA 64 64
DDM
CDM
MINTNET [41]
3.32
120
DDIM SAMPLER, 50 STEPS
8.47
4.78
FLOWMATCHING [29]
2.99
142
DDPM SAMPLER, 1000 STEPS
4.13
2.51
VDM [22]
2.65
10K
2ND ORDER DPMS, 25 STEPS
6.16
4.45
DDPM (L) [19]
$3.70
1K
DDPM (Laimpte) [19]
$3.75
1K
UNCOND CIFAR-I0
DDM
CDM
DDPM (SDE) [43]
3.28
200
DDPM+ CONT. [43]
2.99
200
DDIM SAMPLER, STEPS
7.19
7.56
DDPM SAMPLER, 1000 STEPS
4.77
4.74
REALNVP [10]
3.49
2ND ORDER DPMS, 25 STEPS
6.91
7.29
GLOW [24]
3.35
RESIDUAL FLOW [6]
3.28
COND CIFAR-10
DDM
CDM
CDM
3.38
CDM(UNIE.)
2.98
DDIM SAMPLER, STEPS
5.92
5.08
DDPM SAMPLER, 1000 STEPS
4.70
3.66
CDM(OT)
2.89
2ND ORDER DPMS, 25 STEPS
5.87
4.87
the classification problem is designed to be harder. Specifically, CNCE is based on training a classifier to predict the order of a pair samples with closer densities, e.g. achieved by pairing a data sample with its noisy version.
Telescoping density-ratio estimation (TRE), proposed by Rhodes al. [35], avoids direct classification between data and noise. Instead, it uses gradual transition between those two distributions, and trains classifier distinguish between samples from every pair of adjacent densities. Such classifier learns the ratio between adjacent distributions, and the overall ratio between the data and noise distributions is computed by multiplying all intermediate ratios.
Choi et al. [7] extended this idea from a finite set of intermediate densities to an infinite continuum. This accomplished by deriving link between the density ratios for infinitesimally close distribu- tions and the principles score matching [23, 40, 42], motivating the training of a model to predict the time score Ør log Px, In contrast, we draw different connection which shows that an MMSE denoiser can be obtained as the gradient of optimal noise level classifier. Also, obtain the log ratio between the target and reference distributions, Choi et al. [7] need to solve an integral over the time-score using ODE solver, while in method this ratio be calculated in single NFE.
Yair and Michaeli [47] extended the concept TRE, proposing the training of single noise level classifier instead of training a binary classifier for each pair of neighboring densities. While this method is conceptually similar to ours, our approach distinguishes itself by incorporating the MSE loss as outlined in Theorem 1. As demonstrated in our experiments, this proves to be crucial for achieving an optimal classifier and high-quality image generation.
Discussion and Conclusion
We developed an analytical connection between an MSE-optimal denoiser for removing white Gaussian noise and cross-entropy entropy-optimal classifier for predicting the noise level. We used this connection to propose CDM a DRE based generative technique that is based on noise-level classifier. Importantly, our classifier trained using both classification loss (CE) and a regression loss (MSE). We showed that this key component is what sets CDM apart from existing DRE based methods, and makes it the first instance of a DRE-based technique that can successfully generate images beyond MNIST.

---

Our approach is not free of limitations. A key challenge is that CDMs can be more computationally expensive than DDMs. Indeed, while DDMs require single forward pass for each denoising step. CDMs require both forward pass and backward pass. Nevertheless, the computational cost of performing a forward and a backward pass through network depends on its architecture. In this work, we chose to use the same architecture as that used by DDPM [19], in order to isolate the impact of our algorithmic approach from the choice of the model architecture when comparing to DDMs. However, an important future direction would be explore architectures that are particularly optimized for CDMs and that alleviate the gap in computational complexity. Such architectures should have the property that performing forward pass and backward pass through them is computationally similar to performing only a forward pass in a regular DDM. This could potentially be achieved e.g., by relying only the encoder part of the U-Net. However, leave this exploration for future work.
Broader Impact CDM is a generative model and thus may potentially suffer from the same limitations other generative techniques. These include biases in the generated images, well as malicious and offensive use, such as creating Deepfakes for disinformation. However, CDMs may also impact domains that rely generative models in positive way. This is because, different from most generative models, CDMs are able to compute the likelihood for any input in a single step. This may be used e.g., for out-of-distribution detection or for ranking the likelihoods of different restored images image restoration tasks. Such capabilities may be crucial in fields like medical imaging.
Acknowledgments
This research was partially supported by the Israel Science Foundation (ISF) under Grant 2318/22.
10

---

References
[1] Omer Bar- Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401. 12945, 2024.
[2] Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International conference on machine learning, pages 573-582. PMLR, 2019.
[3] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. InstructPix2Pix: Learning to follow image editing instructions. In CVPR, 2023.
[4] Ciwan Ceylan and Michael U Gutmann. Conditional noise-con estimation of unnor- malised models. In International Conference on Machine Learning, pages 726-734. PMLR. 2018.
[5] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, Najim Dehak, and William Chan. WaveGrad 2: Iterative refinement for text-to-speech synthesis. arXiv preprint 11X:622006.0966, 2021.
[6] Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems,
32, 2019.
[7] Kristy Choi, Chenlin Meng, Yang Song, and Stefano Ermon. Density ratio estimation via infinitesimal classification. In International Conference on Artificial Intelligence and Statistics, pages 2552-2573. PMLR, 2022.
[8] Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated MRI. Medical Image Analysis, 80: 102479, 2022.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusson models beat GANs image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[10] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. arXiv preprini arXiv: (6005088003 2016.
[11] Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106,496):162,, 2011.
[12] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu Video: Factorizing text-oo-video generation by explicit image conditioning. arXiv preprint 27X1v22311.17009 2023.
[13] Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint 180.0.78/0 2018.
[14] Aditya Grover and Stefano Ermon. Boosted generative models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[15] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications natural image statistics. Journal of machine learning research, 13(2), 2012.
[16] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen -O. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.011 01666. 2022.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, volume 30, 2017.

---

[18] Jonathan Ho and Tim Salimans. Classifier-free arXiv:2207.12598. 2022.
diffusion guidance.
arXiv preprint
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances neural information processing systems, 33:6840-6851, 2020.
[20] Inbar Luberman-SSiegelle
Vladimir Kulikov, and Tomer Michaeli. An edit friendly DDPM noise space: Inversion and manipulations. arXiv preprint arXiv:233040611 2023.
[21] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593-23606 2022.
[22] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21777, 2021.
[23] Durk P Kingma and Yann Cun. Regularized estimation image statisties by score matching. Advances in neural information processing systems, 23, 2010.
[24] Durk Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. Advances in neural information processing systems, 31, 2018.
[25] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
[26] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. University of Toronto, 2009.
[27] Justin Lazarow, Long Jin, and Zhuowen Tu. Introspective neural networks for generative modeling. In Proceedings of the IEEE International Conference on Computer Vision, pages 2774-2788 2783, 2017.
[28] Yan LeCun. The MNIST database of handwritten digits. 1998. URL httth:://yaa..eeeae lecun comm/xxb/mnnist.
[29] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02777, 2022.
[30] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 030003 2022.
[31] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pages 3730-3738, 2015.
[32] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775 57755557777777 2022.
[33] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[34] Koichi Miyasawa et al. An empirical bayes estimator of the mean of a normal population. Bull. 1961.
Inst. Internat. Statist, 38(181-188)
[35] Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density- ratio estimation. Advances neural information processing systems, 33:4905 2020.
[36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
12

---

[37] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee. Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
[38] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibriu thermodynamics. In International conference on machine learning, pages 1256-22635 PMLR, 2015.
[39] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[40] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances Neural Information Processing Systems, 32, 2019.
[41] Yang Song, Chenlin Meng, and Stefano Ermon. MintNet: Building invertible neural networks with masked convolutions. Advances in Neural Information Processing Systems, 32, 2019.
[42] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 574-584. PMLR, 2020.
[43] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456, 2020.
[44] Yang Song. Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medi- cal imaging with base- generative models. In International Conference on Learning Representations, 2023.
[45] Charles Stein. Estimation of the mean of a multivariate normal distribution. The annals of Statistics, pages 1135-1151, 1981.
[46] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
[47] Omer Yair and Tomer Michaeli. Thinking fourth dimensionally: Treating time as a random variable in ebms. http:///ggrriiieeee//////ttttttm 2022.
13

---

A
Notation
Table 4: Notation
Notation
Description
Comments
The random variable over t E {0, 1........ T + 13 with distribution pt(t)
(t())
The distribution of the random variable
IP(t
Xx
The diffusson signal at a specific timestep =
X,X;
The diffusion signal at a random timestep
Px, (x)
The distribution of noisy images with noise level
Px)((22(t)
t
px(x)), Px. (x)
The joint distribution of noisy images among all the noise levels.
Ets Px: (2)
F(x,t)
log(p)x(7 + 1 log (og()(=((())))
fo(x))
Logits vector that is produced by our model
fo(())) ( ~ log(p(s((t)z)) fo (2x) [T + 1] - fo(2))
Fo(x,,t) t)
Model approximation of F(x,t)
B
Proofs
B.1
Theorem 3.1 Proof
Using Bayes rule,
Px. (x) (x)()((
(11)
(()
In particular, for T + 1, this relation reads
Pxr (xc) P1sx (T' 11)))x(x)
(12)
(T +
Combining (11) and (12) yields
Px, (z) (T 11
(1/x(
(T 1|x) Pxr+i (x).
(13)
Pt(t)
Taking the logarithm both sides, we have
(T + +1)
Ptix (1/()
log(Px. (x)) log
+ log
+ log(Perea
(14)
(1(1)
Pijs (T 1|xx)
Taking the gradient of both sides ær and noting that the first term on the right hand side does depend on æ, we get that
Ve log(px, (x)) V (og((((((()))) V log( (T + 1[x)) + V (og(pxr11(x) (15)
Since Pxr
(x) N(x 0, I), have that V~ log(p>r;> (z)) Therefore, overall we have
V (px. (x)) V (o((p) Vv log( (T' 1|x)) x.
(16)
As for the left hand side of (15), since VorXo + VI ãic and E ~ >/ I), using Tweedie's formula [34, 45, 11] it can shown that
1
Va log(px, (x))
E(E xx x².
(17)
VT- az
For completeness provide the full proof for (17) in App. B.2. Substituting (16) into (17) and multiplying both sides by VI - o₁ gives
E|ey|x; = x| vI āi (Va log(pgx(T 1|x)) Va log(p)x(t)z)) + +x),
(18)
14

---

which completes the proof.
Note that the proof is correct for any choice of ps (provided that (+ > 0 for all / E (1,.,T++ 1)), since the term that depends on does not depend on æ and thus drops when taking the gradient. In practice, mentioned in the main text, chose use U(O,, 1, 1,..., 1}).
B.2 Proof of Tweedie's Formula
Let us show that if y μx + oz with NNO I) statistically independent of x, then Vy log(py())) = - Elz v].
From the law of total probability,
Py (v(
/
Py ix (
1 20
(2m)d/2add exp
lly pall? px(2))dx.
(19)
Taking the gradient w.r.t gives
20 ly mxel? } px px(c))
1
V (y)
02 ( m.) (2th d/2 exp
σ2 (y- μa: Wy xy (x)
σ2 yy μ px px
(20)
Dividing both sides by (y)), we get
V y (oggy(y) = -
02 (y - μox) Pxly (xl
Ely- jixly y)
σ²
Flzly = y).
(21)
Now, substituting μ = Vol, V Oct y = xt, y = T.t and z = Et, leads to
1
V xx log (2::)
Eler X, =xr],
(22)
VI α
demonstrating (17).
B.3 Proof of Theorem 3.2
Substituting Pxr+1 (x) (x; 0, I) into (13) leads to
Px. (x) PT[T + 1)
(t) PT + (1x) 0,
(23)
(t())
which proves Theorem 3.2.
B.4 Proof of CDM for the Flow Matching Optimal Transport Scheduler
In the case of the flow matching aptimal-transppe scheduler [29, [0], x, defined as
T X T X E
(24)
where e, ~ NO and t (O...., T3. In this case, the objective of conditional flow matching is [29, 30]
T
Exo.et vx(xx) 7 ( xo)
(25)
15

---

and the global minimum of this objective is achieved by
U/ (z)
ElE, - Xu X 2².
(26)
T
We will start by expressing this solution in terms of the optimal denoiser [[]e Substituting (24) into (26) and using the fact that E(x, X
2 gives
1 T
Uy a
(Elee) x] =x).
(27)
Next, following exactly the same logic as in App. B.1, we can write
t T
E|er|x, = x|
(V log(p) Va log(p)s(((z)) +)),
(28)
Finally, by substituting (28) into (27) we get
T(1 - t/T) ( - V log(p(s(((z)) 1
t/T
(x(x))
x.
(29)
C Implementation details
C.1 Architectures
For a fair comparison, for each dataset we used the same architecture for our method and for DDMs. As baslines we took the pre-trained model for CelebA 64 x 64 from the DDIM Official Github [39] and the EMA tre-trained model for CIFAR- 10 from the pytorch diffusion repository, who converted the pre-trained model from the Official DDPM implementation from tensorflow to pytorch.
For conditional CIFAR-10 we trained the DDM model by ourselves because there exist no pre-trained models for CIFAR- 10 capable of handling CFG. We used the same architecture from [19] for both models and trained them for the same number of iterations (more details are App. C.2.2). To condition the model on class labels, we learned an embedding for each class using nn. Embedding and injected at all points where the time embedding was originally applied. In the case of DDM, we added the class embedding to the time embedding, while for CDM, we replaced the time embedding with the class embedding.
In contrast to DDM architectures, our model does not need the layers that process the timestep input so we removed them. In addition, our model outputs a probability vector in contrast to DDMs which output an image, therefore, we replaced the last convolution layer that reduces the number of channels to the original architecture by convolution layers outputs 1024 and 512 channels for CelebA 64 x 64 and 10, respectively. Following this layer, we performed global average pooling and added a linear layer with an output dimension of The resulting change in the number of parameters is negligible.
Inspired by Yair and Michaeli [47], we added non-learned linear transformation the output of the network, which performs cumulative-sumin summattin (cumsum). This enforces (for the optimal classifier)
((x(((2) the t-th output of the model before this layer be log re (2) log (t [m] loog (f|ar:
log Pis 1(x) for t * and ro(x) log (0)z), so that after the cumsum layer, the t-th output Ei log (xx = log (t)(a)).
C.2
Hyperparamters
C.2.1 CelebA 64 x 64
We trained the model for 500k iterations with a learning rate of 1- 14-4, We started with linear warmup of 5k iterations and reduced the learning rate by a factor of after every 200k iterations. The typical value of the CE loss after convergence was 3.8 while the MSE loss was ~ 0.0134 so we chose to give the CE loss a weight of 0.001 to ensure the values of both losses have the same order of magnitude. In addition We used EMA with factor of 0.9999, as done in the baseline model.
16

---

Figure 6: Samples from the conditional CIFAR-10 model. The figure depicts samples generated using CFG with parameter of 0.5. Each row corresponds to a different class.
C.2.2 Unconditional and Conditional CIFAR-10
We trained the model for 500k iterations with learning rate of 2 10-4, We started with linear warmup of 5k iterations and reduced the learning rate by a factor of 10 after every 200k iterations. The typical value of the CE loss after convergence was 4.4 while the MSE loss chose to give the CE loss a weight of 0.001 to maintain values at the same order of magnitude. In addition, we used EMA with a factor of 0.9999, as done the baseline model.
0.03
For the CDM(unif.) model we used the same hyperparametes except for learning rate, which we set to 1 10-4
The CDM(OT) model was trained with the same hyperparameters, except for the learning rate, which was set to 10-4. Additionally, we trained the model without a learning rate schedule for 1M iterations, as the NLL continued to decrease beyond 500k iterations.
C.3 Compute Resources
C.3.1
CelebA 64 x 64
Training the model on CelebA 64 x 64 takes 108 hours server of 4 NVIDIA RTX A6000 48GB GPUs. Sampling 50k images for FID calculation takes 16 hours on the same hardware.
C.3.2
Unconditional and Conditional CIFAR-10
Training the model on CIFAR-10 takes 35 hours server of 4 NVIDIA RTX A6000 48GB GPUs. Sampling 50k images for FID calculation takes 9 hours with classifier free guidance and hours without classifier free guidance on the same hardware.
C.4 Data Augmentation
Following the models to which we compared, for all the datasets we normalized the images to the range [- and applied random horizontal flip at training.
C.5 Classifier Free Guidance
We used CFG sample conditioned examples Sec. 4.2 following [18]. We trained our own conditional models on the CIFAR-10 dataset, both for DDM and for CDM, and used label dropout of
17

---

Independent Pixels
Correlated Pixels
2.0
Predicted vs Ground Truth
Predicted vs Ground Truth
yex
1.5
y=x
Analytic Likellhood
Analytic Log Likeelthood
1.0
0.5
-0.5
-1.0
-0.5
-1.5
-1.0
1155
-1.0
-0.5
0.0
0.5
1.0
1.00
-0.5
0.0
0.5
1.0
1.5
2.0
Predicted Log Likelhood
Predicted Log Likellhood
Figure Log likelihood computation using CDM for toy problems possessing closed form expressions. The left subplot corresponds to the densities from Sec. D.1, while the right subplot corresponds to densities from Sec. D.2. The red points on graphs correspond differnet values. The alignment of these points along the diagonal provides evidence supporting our likelihood estimation as stated Theorem 3.2.
0.1. We selected the best parameter w for each method using a grid search over w 0.25, 0.5, 0.75, 1. In Fig, 6 we show samples from the conditional model. Each row corresponds to different class.
C.6 The Addition of The Timesteps and T ++1
As outlined in Sec. 3, we introduce additional timesteps. corresponding to Lo, æy-1- which are present in DDMs. This inclusion is fundamental to our method formulation. Importantly, this addition does not alter the number of sampling steps, as we continue initiating the reverse process from T and finish it with the denoised result of
1, following the approach in DDM.
D Validating the Log Likelihood Computation on Toy Examples
We validate our efficient likelihood computation by applying it to toy examples with known densities, allowing to compute the analytical likelihood and verify that computations align with theoretical expectations.
D.1 Images With Independent Pixels
We start by experimenting with the uniform distributions p. -- 21 3x32x32 (0.25, 0.5, 2, 2,3). 3). We trained CDM separately for each of these densities following the pro- cedure outlined in Algorithm 1. These distributions correspond to 32 32 color images of iid uniform
with y
noise. The probability density function (pdf) of the uniform distribution U 7, 2 is given by
[--
+ if - 3 3,
[(;7)
(30)
otherwise.
The pdf (x) in the d-dimensional space is the product of the individual pdfs for each dimension. Since the dimensions are independent, the joint pdf is given by
2
(x) II f(aon)
(31)
The logarithm of pdf (x) is given by -dln(7). The log-likelihood is the expectation of In (ar). Since In (xx) is constant, its expectation is that constant. Therefore, the log-likelihood normalized by d is In(7). In the left subplot of Fig. we compare the analytical log likelihood computed using (32) with the estimated log likelihood by model.
18

---

1=200
t=400
t=500
t-800
Image
Nolsy Image
CE
GE MSSE
Figure 8: Comparison of denoising between a model trained using only CE and one trained using both CE and MSE. We note that the denoising results for the model trained using CE alone are poor, but are better for high noise levels than for lower ones. This resonates with our conclusion that models trained only with CE are only accurate near the real noise level: As denoising with CDM relies on Theorem 3.1, we would expect deteriorating denoising quality the further the real noise level is from T + 1, as fa(x) (T + 1] is always used for denoising.
D.2 Images With Correlated Pixels
To further validate our likelihood calculation, we tested on toy examples of images with correlated pixels. Specifically, used images of size 3 x 32 x 32 sampled from multivariate Gaussian distributions with μ = and Σ 전 T. The normalized log likelihood per dimension for multivariate Gaussian is given by
In (
0.5 (In(2.)) + 1)
(32)
2d
ECIFAR-O + V where ECIFAR-1O the defined sequence distributions with = 7
empirical covariance matrix of the CIFAR-10 10 dataset, and y E {0, 10 1000 110-3,10 1,10-1.. 3. In the right 1103310-11
subplot of Fig. 7, compare the analytical log likelihood computed using (32) with the estimated log likelihood by our model.
E
More Experiments
E.1
Analysis of the Effect of Training with Different Losses
Figure 8 provides further qualitative analysis of the effect of training with different losses the quality of the model's image denoising capabilities. The results illustrate that the model trained using only CE loss achieves poor denoising quality compared to the CDM model trained with both CE and MSE.
E.2
The Influence of Different Schedulers on the Classifier Performances
First, assess the classifier's performance, we present the confusion matrices of models trained with and without MSE loss in Fig. 9a. Notably, at lower noise levels, the classifier exhibits high confidence, while at higher noise levels, confidence diminishes. This finding corresponds to the DDPM scheduler [19], which partitions the noise levels to be more concentrated for high timesteps and less concentrated for low timesteps. The similarity between confusion matrices underlines that CE loss alone is adequate for accurate predictions around the real noise level. However, as demonstrated in the main text, this does not imply that the classifier is optimal in terms of learning the correct logits for any given t.
19

---

With MIE
0
1.00
Tre Libel
0.0
True Label
400
Prslsiin
Without NISE
600
001
1000
338
200
400
Prediction
800
1000
(a) Confusion matrices evaluated for models trained with both MSE and CE loss (top) and only with CE loss (bottom). The colors indicate proba- bilities. The similarity between the confusion matrices underlines that CE loss alone is adequate for accurate predictions around the real noise level. This is also
(b) Confusion matrix CDM(unif.) model evalu- ated unconditional CIFAR-10 32 32 using the scheduler from TRE. The colors indicate the probabilities. In contrast DDPM noise scheduler, with TDR noise scheduler, the classification difficulty is preserved across all timesteps.
shown in Fig. (3)
In Fig. 9b, we illustrate that the uniform scheduler from [35] induces uniform difficulty in classifi- cation across various noise levels. This is in contrast the DDM scheduler, depicted in Fig. 9a, in which the classification difficulty increases with the noise level.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and the introduction state the claims made and the contribution. All the claims made match the theoretical and experimental results which appear in Sec. and Sec. 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Sec. 6 we discuss the limitation of our method.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification: All the theorems and the full set of assumptions appear in Sec. 3. All the assumptions are clearly stated and the full proofs appear in App. B.I and App. B.3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the experimental details appear in Sec. 4. The full training and sam- pling algorithms appear in details in the main paper (Algorithm I and Algorithm 2). The architecture, the hyper-panee paremmersseterrs and the optimizer we used appear in App. C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
22

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
Code is available on the project's webpage.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/ public/uuides/ watpes//oooSuumissiiiiicc.
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. cc/ cc/public//ee cublic/guides wwwggguddee/C/a /ideee/Comuubmii www.eeaumiisiioooooooooooooo ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental settings presented in Sec. 4, and all the training details (architecture, hyper-paramette and optimizer) are presented in App.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: In order to calculate error-bars, the model need to be trained many times with different seeds, which would be too computationally expensive. In addition, the best of our knowledge, in most of the papers (and specifically the ones we compare to) FID and NLL are reported based on single experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the amount of compute required for each of the experiments in App. C.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps://neune hccppullic/Ehiisuuiiaiii
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Sec. 6 discuss the broader impacts of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We used academic resources for data and models and cited accordingly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /atasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assests.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "yZPKgbxJjt", "reviewer": "Reviewer_15GS", "summary": "This work develops a new generative framework called the classification diffusion models (CDMs) based on the density ratio estimation (DRE), by establishing an interesting connection between the DDPM's denoiser and noise-predictive classifier, which also helps the exact likelihood computation in a single pass. As is claimed by authors, the proposed method is the first DRE-based technique to successfully generate images of the CIFAR-10 dataset.", "questions": "1. Please provide more details of questions raised in the weaknesses section above.\n2. Check minor grammar typos, e.g. in Line 258, \" trains a classifiers...\".", "limitations": "As is stated by authors, it is worthy to explore BP-friendly architectures to alleviate the computation inefficiency of CDMs.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work is well-written and clearly formulated. \n2. The proposed algorithm is theoretically grounded. \n3. The experiments are convincing, which verify the effectiveness of proposed methods.", "weaknesses": "1. The main concern is the capability or potential of DRE-based methods to successfully learn data distributions, particularly when applied to datasets with large scales in practice. As is claimed by authors, the current development of DRE-based modeling is up to the CIFAR-10 dataset (MNIST before), which is too small to indicate the validity compared to more standard methods (e.g. DDPM). \n2. Another major concern is that CDMs can be more computationally challenging than DDPMs in sampling. According to Algorithm 2, CDMs require an extra backward propagation (BP) than DDPMs for each time step when denoising, which is quite expensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The potential of DRE-based methods to learn data distributions of larger scale datasets**\n\nWe acknowledge that the Celeb-A 64x64 and CIFAR-10 32x32 datasets we experimented with are not very large and high-dimensional by today’s standards, and we certainly agree that extending the evaluation to datasets like ImageNet would be valuable. Unfortunately, however, we lack the computational resources to do so. For instance, as reported in [9], Table 10, achieving satisfactory results (in terms of FID) on ImageNet 128x128 requires a large amount of computation, reaching roughly 521 days on a single V100 GPU. We can run on eight A6000 GPUs, with which we approximate it will take us 56 days to train a standard DDM with the same architecture and number of iterations, using a batch size which is smaller by a factor of 2.\n\nNevertheless, we believe that the preliminary exploration we present on CIFAR-10 and Celeb-A is still valuable for the research community, as it analyzes why DRE-based methods have failed to capture distributions of high-dimensional data to date, and suggests a practical algorithm based on our theoretical result for overcoming this problem. Furthermore, we truly believe that with sufficient computational resources, future works will be able to train our method on larger-scale datasets.\n\n**CDMs can be more computationally challenging than DDPMs in sampling**\n\nWe agree with the reviewer and have mentioned it as a limitation in the main text. Indeed, while a DDM requires a single forward pass for each denoising step, a CDM requires both a forward pass and a backward pass. Nevertheless, the computational cost of performing a forward and a backward pass through a network depends on its architecture. In this work, we chose to use the same architecture as that used by DDPM [19] to isolate the impact of our algorithmic approach from the choice of the model architecture when comparing it to DDMs. However, it is an important future direction to explore architectures that are particularly optimized for CDMs and that alleviate the gap in computational complexity. Such architectures should have the property that performing a forward pass and a backward pass through them is computationally similar to performing only a forward pass in a regular DDM. This could potentially be achieved e.g., by relying only on the encoder part of the UNet. However, we leave this exploration for future work.\n \n\n**Minor grammar typos**\n\nThanks. We will fix them in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UtdAa3v8gP", "reviewer": "Reviewer_ftzp", "summary": "This paper introduces a clever connection between Density Ratio Estimation (DRE) and Diffusion Models (DM), showing that optimal denoisers are also optimal noise classifiers. Doing so allows them to construct a new type of loss based on noise classification. This allows DRE methods to inherit the benefits of diffusion models, like strong sampling, while adding new capabilities like single step likelihood estimation.", "questions": "I was really surprised by the fact that your architecture has no timestep conditioning, but matches or outperforms DDMs that do have timestep conditioning in denoising MSE. Have I understood that correctly (i.e. that the DDM in Fig. 4 does have timestep conditioning?) If so, that seems like a really interesting conclusion for diffusion research, as the final appearance of the timestep only shows up in a simple way in Eq. 8, potentially leading to a simplification of other architectures. It would be interesting to see if this effect persists for stronger architectures. On the other hand, maybe the right way to interpret this result is that your architecture essentially outputs the result for all time-steps at once. Eq. 8 selects the correct one gets a score through backprop. Still, that may be an interesting possibility to try to explore for standard diffusion models. \n\nIn Fig. 3, I can understand why the classifier is limited, but it wasn't totally clear to me why MSE helps. But there may not be an easy answer to that (besides the superficial one that it is a regularizer of some sort).", "limitations": "Limitations were well addressed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The mathematical connection is clever and elegant.  \n- I liked the framing / connections to DRE literature.  \n- The single step likelihood calculation gives a distinguishing feature that sets the method apart from diffusion models. \n- Reasonable ablations were considered, and clear and readable notation was used.  \n- The method requires an additional backward pass for each denoising/sampling step. In the spirit of the NeurIPS guidelines, I count this clearly labeled limitation as a strength. It's intellectually interesting and also clearly distinguishes that the computation done by their model is different from regular diffusion models in a non-trivial way.", "weaknesses": "The proposed approach is functionally equivalent to a diffusion model (using Eq. 8 and with necessary incorporation of MSE loss that matches typical diffusion models), so you might hope to see that you can substantially improve on diffusion models by adding the CE loss as a kind of regularizer. But there are several reasons why it isn't clear that this gives a susbtantial improvement. \n- The architecture has to be changed. There's no obvious way to get around that, but it does marginally affect the ease of use for these ideas, and the directness of comparisons (though on this point I agree with the authors' statement that they chose a minimal intervention).  \n- MSE improvements (Fig. 4) are only marginally different in absolute terms, which is what matters for log likelihood. See some questions about this result. \n- The authors only used / compared with relatively old diffusion models, so it's not clear if modest benefits to MSE or FID in some cases would still be seen in more SOTA models. I would expect much more comprehensive experiments to support the idea that using CE loss can improve sampling (Table 2). \n\n\nThe strongest claim of improvement is the single step likelihood calculation. However, there are three weaknesses associated with this result. \nTable 3 is a bit of a dubious comparison for two reasons. \n- Unlike the ELBO, it's not clear if the proposed estimator gives an upper bound on NLL. I suspect it doesn't. If there are errors in p(t|x), then because it shows up with both signs in Eq. 10, it could contribute to the error in either direction. So we are comparing bounds on NLL to an estimate of NLL which could look lower due only to error.  \n- The NFE comparisons are also a bit dubious. Function evaluation complexity varies wildly between methods. We should have a fair comparison to the associated diffusion model with the same architecture, but it wasn't clear which one that is in the table. Though it's a bit moot as I wouldn't trust the result anyway, as we don't have a bound here. \n- A compelling application of the single step log likelihood evaluation was not discussed. If the log likelihood had been used and evaluated for some downstream application, it might have assuaged worries about the accuracy of the estimate. \n\n\nMinor comments:\n- I didn't like the abstract sentence \"directly output likelihood...lacking in most generative techniques\", since most common generative techniques today do output likelihood. You clarify later what you mean by \"directly\" so it is Ok. I just wanted to say that it was off-putting on first read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The architecture has to be changed**\n\nWe agree with the reviewer that the architecture can't be exactly the same since DDM is a denoiser and CDM is a classifier, which may impact the comparisons a little bit. However, as the reviewer notes, the change in the architecture is minor – we replaced only the last layer of the DDM architecture by a convolution, followed by max-pooling and a linear layer. This has a negligible effect on the number of parameters. It is important to note that this architecture was optimized for DDPM and not for CDM, yet CDM still achieves comparable or better results with it. As we mentioned in the text, we believe that an important and interesting future direction would be to optimize the architecture for CDM. \n\n**Old diffusion models**\n\nWe want to emphasize that our main goal is not improving denoising diffusion models, but rather making DRE-based methods work for the first time on datasets more challenging than MNIST. Therefore, we mostly focus on analyzing the reasons that DRE-based methods failed, and show how our theorem leads to a practical algorithm that overcomes these issues. Moreover, we used the same architecture of DDPM only for a fair comparison. In future work, it would be interesting to investigate different architectures which are more suitable for CDM (and in general may be different than the standard denoiser architectures).\n\n\n\n**Upper bound on NLL**\n\nOur method indeed doesn’t upper bound the NLL but rather estimates it. Yet, this is true for all methods in Table 3, which calculate the likelihood and not the ELBO. For example, DDPM-SDE and DDPM++ (which estimate the likelihood and not ELBO) numerically solve an ODE in which each step can accumulate errors. \n\nAs for the fact that $p(t|x)$ appears with both signs, this is a key problem with existing DRE based methods, which our method solves. Without incorporating the MSE loss, the model’s prediction is inaccurate for $t$ values that are far from the real $t$ as we show in Fig. 3. Specifically, when calculating the likelihood of a clean image $x$, the model needs to be accurate for both $p(0|x)$ and $p(T+1|x)$. This is the reason that DRE-based methods failed to date. In our case, the addition of the MSE loss solves this problem and makes the NLL calculation more accurate (see also the answer to the last question below). In App. D we show that our NLL calculation is exact on high-dimensional toy examples for which we can calculate the ground-truth NLL analytically.\n\n**Function evaluation complexity varies wildly between methods**\n\nWhile the evaluation complexity depends on the architecture, in all cases shown in the table, the number of NFEs is at least 100 and in some cases even thousands. In our case, the calculation requires only a single NFE which is much cheaper computationally. In addition, we use the same architecture as DDPM which appears in the table and requires approximately 200 NFEs, and as we show, we achieve a better NLL.\n \n**Validating the log likelihood evaluation on a downstream task**\n\nWe agree with the reviewer that it would be interesting to validate the effectiveness of our single step likelihood computation in the context of downstream tasks. We see the use of our method in applications as an intriguing direction for future work. For example [R1] showed that a generative model which is able to calculate the likelihood, can be used for zero-shot classification. However, they need to perform many NFEs in order to calculate the likelihood, whereas our method can do this with a single NFE.\n\nFollowing the reviewer’s concern, to further validate our NLL evaluation, we will add to the final version more high-dimensional distribution examples (in addition to those already appearing in App. D), for which we can compute the ground-truth NLL analytically.\n\n**The architecture has no timestep conditioning**\n\nExactly; you understood it correctly. Our theorem suggests that, given the optimal classifier that predicts the timestep (and therefore can't be conditioned on it), we can extract the optimal MMSE denoiser. This effect should persist independently of the architecture and specifically for stronger architectures. It is a unique property of CDMs that doesn't exist in DDMs, since in CDMs the time condition implicitly appears by taking the corresponding classifier's output (i.e., if we want to be conditioned on timestep $t$, we will take the derivative with respect to the $t$-th entry of the classifier outputs, as shown in Theorem 3.1). \n\nYour interpretation that our architecture essentially outputs the result for all time-steps at once is correct, but note that it does so in a very efficient manner. Specifically, if we wanted to do so in a diffusion model, it would have to output $T$ predicted noise maps at once. Our model, on the other hand, only outputs $T$ scalars. It is the gradient of each output that gives the predicted noise map.\n\n\n\n **Why does MSE help in Fig. 3?**\n\nThat is a very important point. When training the model using only the CE loss, the model’s prediction $p(t|x)$ may be accurate for the correct noise level but not for more distant noise levels (as shown in Fig. 3). This is why DRE-based methods have failed to capture the distribution of images to date. As Theorem 3.1 shows, the optimal denoiser for timestep $t$ depends on the model prediction for both $\\log p(t|x)$ and $\\log p(T+1|x)$. Therefore, when we add the MSE loss on the derivative of the difference between them (using the formula from Theorem 3.1), we enforce the classifier’s output to be accurate not only in it’s $t$-th entry (as can be achieved using only CE) but also in its $(T+1)$-th entry. By enforcing this, we obtain an optimal classifier that predicts the correct probability $p(t|x)$ both locally around $t$ (thanks to the CE) and globally for distant $t$ values (thanks to the MSE).\n\n\n\t\n[R1] Zimmermann, Roland S., et al. Score-based generative classifiers, NeurIPS Workshops ‘21.‏", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uEO3hFRwxX", "reviewer": "Reviewer_93Vh", "summary": "A common approach for likelihood estimation is the density ratio estimation (DRE). DRE is a method of modeling the density of a target distribution (or data distribution) in the form of a density ratio with a known reference distribution. The standard normal distribution is often chosen as the reference. To train the density ratio models, a classification-based method called noise contrastive estimation (NCE) is generally used.\n\nHowever, in high dimensions, the difference between the reference and the data distribution is large, making the classification problem very easy. Consequently, the model does not learn accurate density ratios. To overcome this saturation problem, some have proposed to create intermediate distributions between the target and reference, similar to diffusion models. For example, the transporting ratio estimation (TRE) first creates a bridge between the target and the reference distribution through their linear combinations. The density ratio between two adjacent timesteps—$t$ and $t+1$-th timestep—is trained using the NCE method. However, despite the diffusion-like modifications, TRE has failed to resolve the saturation problem in high-dimensional data.\n\nTherefore, to overcome the limitation of the previous approaches and estimate the log-likelihood efficiently for the high-dimensional datasets, the paper proposes a novel DRE-based diffusion-based generative model. Most importantly, the paper proposes to predict the timestep $t$ for a given observation $x$—$p(t | x)$—when $x$ is sampled uniformly from all intermediate distributions in the bridge. It means that the model predicts how noisy the given input is, unlike the classification problems proposed in TRE. The paper highlights two important benefits of the proposed method.\n\nFirst, the authors draw the analytical connection between the noise-predicting classifiers and the Bayes-optimal denoisers at each time step. This connection means conventional parameterizations in diffusion models, such as epsilon/score/denoiser, can also be written by the noise-predicting classifiers. Thus, the noise-predicting classifiers can be trained via denoising score matching (DSM) as well as the cross-entropy loss. This training overcomes the saturation problems that originated from classification-based training.\n\nSecond, even compared to the TRE, regardless of its scalability issue, the noise-predicting classifiers provide efficiency at the log-likelihood evaluation. The proposed method can estimate the log-likelihood of the (clean) data in one-g, while the TRE requires $T$-number of model evaluations.\n\nIn addition, the paper mentions a concurrent work that proposes a similar noise-predicting classifier, but the authors emphasize that the concurrent work didn't incorporate the DSM loss, which is the key ingredient of the proposed approach's scalability. \n\nFinally, the authors demonstrate the proposed method's effectiveness via various experiments, including generative modeling of image generation benchmark datasets.\n\n\n------------------------\nUpdate the rating from 7 to 8 after the authors' rebuttal", "questions": "N/A", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "One of the paper's key contributions is the introduction of a novel solution for likelihood estimation. The proposed method addresses significant challenges in the density ratio estimation and offers a fresh perspective on the parameterizations in diffusion-based models.\n\nMoreover, the authors effectively motivate the use of DSM by establishing the connection between the time-prediction classifiers and the conventional parameterizations. This training overcomes the limitations of previous DRE-based approaches.\n\nFurthermore, the proposed method exhibits comparable performance and scalability compared to popular diffusion-based generative models, further underscoring the practical effectiveness of the proposed DRE-based method.", "weaknesses": "While the proposed method's originality and novelty are clear, its significance is still doubtful. While achieving efficient likelihood estimation is the primary motivation, the paper doesn’t provide sufficient discussion about it—instead, it focuses on demonstrating the improved generation performance of the proposed DRE-based method. Thus, the proposed method would be much more impactful if the authors could demonstrate what becomes available when the likelihood estimation for high-dimensional data works.\n\nAdditional discussion about the importance of the classification loss may seem required. It is clear that the time-prediction classifier links to the conventional denoiser parameterization, and thus, this connection equips the use of DSM to train the classifiers, which enables the proposed model's applicability to high-dimensional data. However, this also implies the possibility of training the proposed models without any classification objective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Likelihood estimation is the primary motivation; the paper lacks illustrations of its use**\n\nOur primary motivation was to analyze why existing DRE methods fail to capture distributions of complex high-dimensional data, and to develop theory and a practical DRE method that doesn’t suffer from this limitation. The single-step likelihood evaluation is actually a by-product of our analysis. Nevertheless, we agree with the reviewer that it would be interesting to explore the single step likelihood computation in the context of downstream tasks. We see this as an intriguing direction for future work. For example [R2] showed that a generative model which is able to calculate the likelihood, can be used for zero-shot classification. However, in their case they need to perform many NFEs in order to calculate the likelihood. These applications are out of scope for our paper, but we believe that our method has the potential to be used for solving these tasks in future work.\n\n\n\n**Additional discussion about the importance of the classification loss may seem required**\n\nThat's an important point. As shown in Table 1, when training using only MSE loss, the CE is very high. In this case, we can use the model as a denoiser, but it will not function as a noise level classifier and specifically will be useless for calculating the likelihood of a given input using Theorem 3.2. The explanation is that the MSE is invariant to an addition of a function of $t$ to the classifier’s output (since when we take the derivative with respect to $x$ in Eq. 8, it zeros out). Therefore, we must combine the CE loss to remove this degree of freedom.  \n\n[R2] Zimmermann, Roland S., et al. Score-based generative classifiers. In: NeurIPS Workshops ‘21.‏", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VoUZkg08ME", "reviewer": "Reviewer_fmsZ", "summary": "This paper introduces a class of generative models based on neural networks which learn the conditional probability distribution of the noise level given a noisy image. It is trained by combining a maximum-likelihood (cross-entropy) objective with a denoising score matching objective. The resulting generative model can be sampled from by iterative denoising like in diffusion models, and also allows for direct likelihood evaluation (in principle).", "questions": "- In the proposed approach, $t = T+1$ plays a special role, which seems slightly unsatisfying to me. Having correct log-density estimates using eq. (10) requires an accurate estimation of $\\log p(T+1|x)$ for $x$ a clean image, which is very challenging as natural images are very far from typical realizations of Gaussian noise (unless the explicit form of the Gaussian density is enforced here?). However, the MSE loss (which indirectly involves $\\nabla_x \\log p_\\theta(T+1|x)$) seems to fix this problem. Still, is there a way to avoid it altogether with a more symmetric formulation? I liked how the approach of Choi et al. elegantly solved it by considering a path $(x_t)$ of typical realizations of each $p(x|t)$, though I understand that in their setting log-density estimation requires computing a (one-dimensional) integral.\n\n- Out of curiosity, what does Figure 3 look like when the model is trained with MSE loss only? A first observation is that one can add an arbitrary function of $t$ only to the model outputs without changing the MSE loss, so the predicted conditional distributions cannot be right. However, I wonder if this is the only degree of freedom: does the difference between the \"true\" and predicted $p(t|x)$ depend on $x$? If true, it would show that the CE objective just fixes this constant and is thus not so important.\n\n- I am very puzzled by the DDM denoising result at high noise in Figure 4. Why would they be so bad? Predicting the mean of the dataset is really easy, so this is not a capacity issue but rather a training issue? And why CDM would be specifically better at large noise levels? Is this because $T+1$ plays a special role?\n\nMinor suggestion:\nI understand that the \"classification\" term comes from historical considerations in the density ratio estimation problem. However, it seems ill-adapted here: $t$ is really a continuous variable (which has been discretized, but this does not seem necessary), and therefore $p(t|x)$ is simply a probabilistic model of a conditional distribution that is trained by maximum-likelihood. The title of the paper also sets the incorrect expectation that it is about unifying diffusion models with image classifiers.", "limitations": "As mentioned in the discussion, an important aspect that is not studied in this paper is the architecture of the network to approximate $\\log p(t|x)$. The community has extensively explored architecture to model scores $\\nabla_x \\log p(x|t)$, but architectures for energies have been much less studied. As rightfully pointed out, here $f$ is a UNet, so that $\\nabla f$ is the gradient of a UNet and thus significantly differs from a UNet (which would be the goal to emulate score-based approaches). Let me suggest a few references that consider this problem:\n- Tim Salimans and Jonathan Ho. Should EBMs model the energy or the score? In Energy Based Models Workshop-ICLR, 2021.\n- Samuel Hurault, Arthur Leclaire, and Nicolas Papadakis. Gradient Step Denoiser for convergent Plug-and-Play. In International Conference on Learning Representations, 2021.\n- Regev Cohen, Yochai Blau, Daniel Freedman, and Ehud Rivlin. It has potential: Gradient-driven denoisers for convergent solutions to inverse problems. In Advances in Neural Information Processing Systems, 2021.\n\nAll in all, I very much like this paper, and strongly recommend acceptance. It proposes novel ideas of interest to the generative modeling community, and the strong numerical results demonstrate that this approach has promising potential.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The proposed model is original and very interesting. Such an approach has the potential to significantly improve upon diffusion models.\n\n- The experimental results are very convincing (except for NLLs, see below) and show great promise.\n\n- The analysis of the shortcomings of density ratio estimation is also on point and very clear. Although these issues were already evidenced in the previous work of Choi et al., their results were limited to MNIST while CDMs achieve satisfying results on CIFAR10 and CelebA 64x64, and have the added potential benefit of direct likelihood evaluation.", "weaknesses": "The biggest weakness of the paper to me is the method used for likelihood evaluation, which has several issues:\n- An implicit assumption in the paper is that the marginal distribution of the noise level learned by the model matches the correct one used during training, that is, $p_\\theta(t) = p(t)$. Although reasonable, it is very difficult to verify, as it requires marginalizing over the high-dimensional variable $x$.\n- There is no guarantee that the resulting density function (eq. (10)) is normalized (i.e., integrates to 1). This can lead to arbitrarily small _reported_ NLLs (potentially even smaller than the entropy of the data). Although the soundness of the approach is verified in a synthetic but high-dimensional example, I think it would be necessary to verify at least this property on a natural image dataset, though this is also very hard to do (but see below).\n- There is also no guarantee that the density function in eq. (10) is the density of samples generated by the reverse diffusion (Algorithm 2). CDMs thus contain a priori two models of $p(x)$, one for sampling and one for likelihood evaluation, which could be different. I would also like to see in the final version of the paper (not for the rebuttal) a comparison between the NLLs of these two models on natural images (the latter can be evaluated with an integral along the sampling trajectory using the change of variable formula, as pointed out in the text). If they match, this also verifies that the reported densities are correctly normalized (see previous point).\n- Finally, restricting NLL comparisons to \"models where likelihood can be evaluated in single shot\" seems slightly arbitrary, especially since here sampling is not in a single shot. It would make sense to allow a similar computational complexity for sampling from the model and evaluating the log-density at a data point.\n\nI might be missing subtle details in my assessment though, and I would be happy to update my score if the authors point out misunderstandings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and interesting comments.\n\n**Comparing the one step likelihood computation to an integral along the sampling trajectory**\n\nIn theory Eq. (10) should be normalized, but this is a great idea. We’ll definitely add it to the final version. We’d just like to note that while our one-step likelihood computation depends only on the model's output, the integral approach depends also on the sampling method. Specifically, different noise schedulers correspond to different ODE/SDE discretizations, and may affect the quality of the generated samples. This is even if each denoiser by itself is optimal. For a scheduler that leads to low quality samples, the likelihood of the generated images obtained with our one-step approach would be low (high NLL). For example, as we show in Sec. 4.3, we can choose a scheduler that improves the NLL over the test set but leads to a worse FID of the generated samples. We argue that this doesn’t imply the model learned a worse approximation of $p(x)$, but only that the sampling trajectory is worse than that of DDPM.\n\nThe DDPM scheduler leads to good generated samples, though. This suggests that the discretization associated with this scheduler leads to an accurate solution of the ODE. And therefore, in this case, we agree that your suggested experiment is interesting and we will include it in our final version. Thanks!\n\n**Likelihood evaluation is performed in a single step, but sampling is not**\n\nWe completely agree. However, there may be applications that only require likelihood evaluation for real images, and not sampling of synthetic images. This is the case e.g. in out-of-distribution detection. Therefore, we believe there is merit in separately reporting the NFEs required for likelihood evaluation and for sampling.\n\nThe ability of our method to evaluate the likelihood of real images in one step stands in contrast to methods that calculate the likelihood by solving an ODE through a sampling-like process (with repeated evaluations of the network), as in DDMs.\n\nIn any case, note from Table 3 that our method is competitive also with methods that evaluate the likelihood using many NFEs.\n\n**The role of t=T+1**\n\nThis is a very interesting question. Please note that both in Equation (3) in [34] and in Proposition 1 in Choi’s work [7], the likelihood is extracted from the ratio  $\\log\\frac{p(x|0)}{p(x|T+1)}$, since $p(x|T+1)$ is known (a Gaussian in this case). In order to achieve a correct likelihood estimation, all the intermediate ratios need to be accurate for the same input $x$, and specifically, the ratios between densities which represent high noise levels need to be accurate when the input is a clean image. For this reason, DRE methods failed to learn distributions of datasets which are more complicated than MNIST. When you incorporate the MSE loss using our theorem, it enforces the log ratio $\\log\\frac{p(t|x)}{p(T+1|x)}$ to be accurate for any $t$, which solves this problem. However, it would be very interesting in future work to try solving it in different ways. \n\n**Using only the MSE loss**\n\nWhen training the model using only the MSE loss, the CE is very high (please see Table 1) and Figure 3 is very noisy, which indicates that the model fails to learn the distribution $p(t|x)$ in this case. Your observation is correct and very interesting. The MSE indeed doesn’t change if we add a function of $t$ to the model’s output. This is why the CE loss is important for estimating $p(t|x)$. Without the CE loss, the model can function as a denoiser but is useless for the purpose of outputting the likelihood in a single step. This indeed seems to be the only degree of freedom that the CE loss fixes because the MMSE predictor corresponds to $\\nabla_x \\log p(t|x)$, which uniquely defines $\\log p(t|x)$ up to an additive constant that may depend on $t$.\n\nNote that the difference between the true and predicted $\\log p(t|x)$ may depend on $x$. However, in this case it is the same function of $x$ for all $t$. This is true because this function should cancel out after the softmax and also when we take the difference between $\\log p(T+1|x)$ and $\\log p(t|x)$ in Equation (8).\n\nTo conclude, the MSE loss alone is invariant to adding a function of $t$ to the model’s output, and the CE loss removes this degree of freedom. In addition, both losses are invariant to adding a function of $x$ to the model’s output, but the softmax removes this degree of freedom. Importantly, when we calculate the NLL using Equation (10), this function of $x$ is canceled out even before the softmax, since it is added both to $\\log p(t|x)$ and $\\log p(T+1|x)$.\n\n**Why is the prediction of DDM poor for large $t?$**\n\nWe don’t know why diffusion models fail here, and it would be very interesting to explore it in the future. However, we do know why CDM is particularly good at the higher noise levels. Please note that in Theorem 3.1, we show that the optimal denoiser depends on the model's prediction for $p(t|x)$ and $p(T+1|x)$. At the high noise levels, $t$ is close to $T+1$, and as we show in Fig. 3, in this case the model’s prediction for $p(t|x)$ and $p(T+1|x)$ is accurate even without incorporating the MSE loss. This accuracy is further improved when we include the MSE loss.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focuses on diffusion-based generative models using density ratio estimation along with a classifier that predicts the level of noise that has been added to a data point in a TRE-like manner (transporting ratio estimation). By deriving analytical connections, the paper shows how to train noise-predicting classifiers using denoising score matching with the cross-entropy loss. The benefit is that log-likelihood of data can be estimated in one function evaluation.

The reviewers found the approach original and interesting enough to warrant publication at NeurIPS. The major contribution is a novel way to think about both density ratio estimation and diffusion models which the reviewers found to have a lot of potential. 

While these analytical connections were satisfying to reviewers, the generative performance of the method lags behind regular diffusion model approaches. Hence, the main benefit is to advance density-ratio methods, but this is not the most performant or promising class of methods, and is not the highest bar to compare against. Hence I find acceptance as a poster to be most appropriate.
Decision: Accept (poster)