Information Theoretic Learning for Diffusion Models with Warm Start
Authors: Yirong Shen
Lu GAN
Cong Ling
Venue: NeurIPS 2025 poster ()
Keywords: Generative models
Additive noise
relative Fisher information
density estimation
likelihood
diffusion models
Primary area: deep_learning
TL;DR: Score matching could connect with relative entropy with arbitrary noise perturbation.

Abstract: Generative models that maximize model likelihood have gained traction in many practical settings. Among them, perturbation-based approaches underpin many state-of-the-art likelihood estimation models, yet they often face slow convergence and limited theoretical understanding. In this paper, we derive a tighter likelihood bound for noise-driven models to improve both the accuracy and efficiency of maximum likelihood learning. Our key insight extends the classical Kullback–Leibler (KL) divergence–Fisher information relationship to arbitrary noise perturbations, going beyond the Gaussian assumption and enabling structured noise distributions. This formulation allows flexible use of randomized noise distributions that naturally account for sensor artifacts, quantization effects, and data distribution smoothing, while remaining compatible with standard diffusion training. Treating the diffusion process as a Gaussian channel, we further express the mismatched entropy between data and model, showing that the proposed objective upper-bounds the negative log-likelihood (NLL). In experiments, our models achieve competitive NLL on CIFAR-10 and state-of-the-art results on ImageNet across multiple resolutions, all without data augmentation, and the framework extends naturally to discrete data.

Information Theoretic Learning for Diffusion Models with Warm Start
Yirong Shen Imperial College London ys69228ic. att.922@ic.c.... ac uk
Lu Gan
Cong Ling
Brunel University of London lu. nt.gaanbruuee.c ac uk
Imperial College London c. ww.lin@@mmerii ial ac uk
Abstract
Generative models that maximize model likelihood have gained traction in many practical settings. Among them, perturbation-baser approaches underpin many state-of-the-art likelihood estimation models, yet they often face slow convergence and limited theoretical understanding. In this paper, we derive a tighter likelihood bound for noise-driven models to improve both the accuracy and efficiency of max- imum likelihood learning. Our key insight extends the classical Kullback-Leibeer (KI.) divergence- FFsheer information relationship to arbitrary noise perturbations, going beyond the Gaussian assumption and enabling structured noise distributions. This formulation allows flexible use of randomized noise distributions that naturally account for sensor artifacts, quantization effects, and data distribution smoothing, while remaining compatible with standard diffusion training. Treating the diffusion process as Gaussian channel, we further express the mismatched entropy between data and model, showing that the proposed objective upper-bounds the negative log- likeliool (NLL). In experiments, our models achieve competitive NLL on CIFAR-10 and state-of- of-the-atttte art results on ImageNet across multiple resolutions, all without data augmentation, and the framework extends naturally to discrete data.
Introduction
Likelihood serves fundamental metric for evaluating density estimation and generative models. tight negative log-likelihood (NLL) bound not only indicates a model's ' capacity to capture the fine-grained structure of the data distribution but also facilitates range of downstream applications,
including data compression 226
29
30
33
73 75 87). anomaly detection (7]. out-of-distrbbtion
detection 84 semi-supervised learning 15 classifier 4
[6]. image generation (8] transfer
learning 17 density ratio estimation 6 10, language models
23
and adversarial purification (71).
Rapid advancements in deep generative modelling 74] have led to various families of models achieving strong likelihood estimation performance, including energy-based models [2I], normalizing flows [69 90], variational autoencoders 440
41 55 diffusion models 28
54 62 cascaded models
[46. and autoregressive models 9 (32] A common underlying structure among many state-of-the-an
90] is the transformation of data into noise via distinct functional mappings. These mappings, despite their differing mathematical forms, can be viewed as variants of diffusion
models 40 46 62
69
models operating under the same noisy process, Gaussian channel [14 64
Gaussian diffusion models can be broadly categorized into variance-preserving (VP) 28 67 and
variance-explost exploding (VE) 43 72 processes by the variance of injected noise. These two paradigms likelihood. VP mod-
differ in both the construction of the forward process and the formulation variable models and trained via variational inference 90], while VE models are interpreted 78 that allow for direct likelihood estimation via However, existing likelihood estimation methods for VE models have not matched the performance of ba-based approaches. This raises the natural hypothesis
els are typically treated as Bayesian
using the evidence lower bound (ELBO) 28
40
55
information-theoneei. (IT' channels 25
cotimaaioo-thhorette tools 24 43 69).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

that likelihood performance may be highly sensitive to noise variance. Furthermore, while previous IT bounds 43 69
8T may be slightly looser than ELBO, they often enjoy faster convergence and
greater interpretability B 16
45 69 83 leaving open the question of whether I'T-based bounds can also be improved as competitive likelihood estimators with enjoying the faster speed and robustness.
Addressing this question requires extending existing theoretical frameworks. The extant Shan- non-Fisher connections for diffusion models have largely assumed idealized isotropic Gaussian corruption [5] 69]. yet real-world data rarely align with such simplified assumptions, Imaging data
commonly feature Poisson- Gasossan sensor noise [20]: dequantization [27 31 74 90] and data smoothing [52] often modeled by uniform or symmetric noise (e.g. a Laplacian kernel) addition to improve tail coverage, sharp transitions and robustness. Additionally, recent generative models
deliberately introduce Poisson [5], heavy-tailed a-stable perturbations (88] or structured noise with the data distribution
62 65
emphasizing the practical needs for more generalized frameworks.
In specific, when a continuous-density model is fitted directly to discrete data, the likelihood eval- uation becomes singular and severely degrades performance. The conventional remedies, uniform or variational dequantization [27] 69 (74), inject auxiliary noise but suffer two drawbacks: they require an additional training phase, which hard to train to the optimal, and, in general, introduce pronounced vaining-evaluation gap that inflates the NLL performance [90] via the mismatched noise.
In this work. to eliminate both sources of discrepancy and instabilities in maximum likelihood learning with diffusion models, we propose variance-awar likelihood bounds via arbitrary isotropic warm-up noise perturbation. Our main contributions and findings summarized as followed:
We first prove Theorems and Proposition 2 showing that for any isotropic noise, the Fisher information measures are asymptotically equivalent to their Shannon counterparts, thereby establishing a maximum likelihood learning framework well beyond the Gaussian setting.
Building on this insight, Theorem 2 and Proposition I provide tightened analytic bounds on the likelihood of diffusion models with only minor architectural changes, specifically, a logarithm signal-to-noke ratio based parameterization and additional low-variance noise regime, which together eliminate the train test gap and stabilize optimization near t = 0.
Empirically, ablation studies confirm the effectiveness of our bounds. Using an efficient aaabortance-sampllee scheme, our method achieves 2.50 bits/dim CIFAR-10 10 and new state-of the-art results of 3.01, 2.91, and 2.59 bits/dim on ImageNet-32. -64, and -128, respectively, while requiring only 0.3M training iterations without any data augmentation.
2 Background
2.1 Incremental Gaussian Channel and Maximum Likelihood Estimation
Assume every datapoint x is an i.i.d. sample drawn from unknown distribution p(x) supported on the source space X C R... Diffusion models (28 [4) naturally induce what is known as an incremental signal-to-noise ratio
Let the source dataset with N datapoints be denoted by X (x, J T
x=
(SNR) channel 25 93 which generates sequence of progressively noisier X towards pure noise:
Y, a, + o,N,
(1)
where a, of E IR+ + are smooth, negaavvve, scalar-valued functions with finite derivatives with respect to time over the fixed time horizon t [0. The noise term N consists of independent standard Gaussian entries. Intuitively, the ratio a /o? can be interpreted as the signal-to- noise ratio (SNR) at time t, denoted as SNR(t) == at /o? By enforcing SNR(t) be strictly monotonically decreasing over the time interval / E [0, I]. the output Y, asymptotically approaches well-defined, analytically tractable stationary distribution (xx as +
To recover and estimate the underlying data distribution p(x) from noisy observations Y,, one must solve a density estimation problem under this forward noisy channel. From an information- theoretic perspective, the mismatched Gaussian channel (78) models the scenario where the true input distribution p(x) is unknown, and one instead uses a hypothesis distribution xxx) for estimation.
In this paper, random objects are denoted by uppercase letters, and their realizations by lowercase letters. The expectation (B()) is taken over the joint distribution of the random variables inside the brackets.

---

noise prodiction model miy: m:
y (e:
hypothetical data x(y,, 1/:; 0)
q::
1o plys)
p(x)
00000 +-00
Forward
Figure 1: Toy example illustrating our method (left) and samples generated on CIFAR-IO (right). We apply an identical warm-up channel to both the data p(x) and the model distribution xx: 0): injecting arbitrary noise V produces smoothed data x and a correspondingly perturbed model do. This results in two variance-regime mismatched channels, a low-variance arbitrary- arbitrary-perturbatior perturbation channel and figh-variance Gaussian noise regime, that share identical channel dynamics but differ in their priors, under which training aligns the Gaussian-perturre distributions and learns the denoising map.
While classical methods for Gaussian channels remain applicable, their analytic solutions are typically intractable because it requires sampling from the posterior distribution of the noisy channel. A more tractable alternative involves leveraging the connection between relative entropy and Fisher divergence. Specifically, [5] [69 demonstrated that the weighted score matching (or equivalently, Fisher divergence) objectives could approximate maximum likelihood training of diffusion models"
2.2 Likelihood of Diffusion Models
Average log- le-likkelihood is widely recognized as the default metric for evaluating generative models. Previous work has largely prioritized perceptual quality, emphasizing coarse scale patterns and global consistency of generated images, with common metrics such the Fréchet Inception Distance (FID) and the Inception Score (IS). In contrast, we optimise for likelihood of the model, a criterion that inherently sensitive fine-scale details and the exact values of individual pixels.
To evaluate the likelihood in diffusion models, we define model distribution q( 0). typically parameterized by a neural network with parameters 0 E θ, which aims to approximate the true data distribution p(x). Given a sample x ~ p(x) and noise n (O, I), noisy observation y is generated via the forward channel described in Equation (I. Under the mismatched channel framework, the marginal distribution over y: induced by the model sample q(>;; 0) is given by:
<(y:: 0) = [ a(x;; 0) p(y:)))
In diffusion modeling, the stationary distribution at t is defined to be the stationary distribution T (x) = NV (O,I), which serves as the starting point for the sampling process. Since our primary focus density estimation and probabilistic modeling, defer implementation and details of sampling process to the Appendix. A.3 and optimization algorithms to future work.
Following (68) we parameterize the model distribution using a noise prediction network. Specifically, the model (y 7): 0) is trained to predict the Gaussian noise n that was added during the forward process, where 7h hypothetical data & is then obtained as:
log SNR(t) defines the noise schedule in log-SNR space. A predicted
Yt o, Thi 0)
* Yt, Ne; 0)
Ot
Theoretically, the training objective of likelihood- diffusion models is minimize the KL divergence (see Def. 3 between the true data distribution p(x) and the continuous model distribution q(x; 0), i.e., DR1. (p(x)) 0)). However, as the noise variance o² + the SNR diverges, leading to numerical instability during both training and sampling (38). To address this, practical
II likelihood based generative modeling, this formulation is equivalent to maximizing the expected log- likelihood. In contrast, methods that prioritize sample quality typically optimize the 2- Wassersteii distance.
3

---

Algorithm 1: Training
Algorithm 2: Likelihood Evaluation
Pre-processing Wart Start
Input:
Trained model fa. test data
for each X' do
Output: Estimated loss L(0). NLL {(x; 6) for each in X do W:
4: x + ao x o(0)- u.
1/ V: arbitrary noise
// store &
1/ V: arbitrary moise // store x
a(U)
Training:
Evaluation:
repeat
1 p(t), (0 ..)) t-
NO, 200, (af. e-
(O 1). 4- f(ys. Ne: 0):
+ &
L( 0) 0.5 [|n R(Y.. dri 0)]
Take gradient descent slep on Volin n( 0)
normalizing constant
f(x; 0) + H(p(y) x). + L(o7:0)
until converged:
return f(x:
diffusion models typically start the forward process small positive time t 0 instead of t = 0 for improved stability. Yet, this small time offset introduces additional perturbation, and the corresponding training objective becomes (ply)()))l (ly 0)). which differs from the evaluation objective DxL (p(x)] (x; 8)). This discrepancy causes a mismatch between the training expected log- likkiiihoood (Epty.) (log q(y, o)) and testing expected log- oollikelihood (Ep(x) ($: 0)).
2.3 Dequantization for Density Estimation
When modeling real-world data, care must be taken to ensure that the reported likelihood values are meaningful 774. Since such data is typically discrete, using continuous density models directly can lead to arbitrarily high likelihoods due to singularities. To mitigate this issue, it is now standard practice to add real-valued noise to integer-valued inputs, a process known as dequantization [17 69 76.. For example, in the case of 8-bit image data, the input values in {0, 1..... 255} are typically perturbed by uniform noise, yielding v x + u where u U[00, 1) With this transformation, training continuous density model the uniformly dequantized data can be interpreted as maximizing a lower bound on the log-likelihood quantized inputs 127 (44. However, this introduces training-test gap in diffusion models: during training, the model qvv 0) is fitted to p(yo), which corresponds to a Gaussian distribution centered at each discrete data point; during evaluation, however, is tested uniformly dequantized data. Although this improves numerical stability in practice (38 69], the mismatch between training and evaluation degrades likelihood performance (90). Moreover, while variational methods (27 allow other forms of noise injection, they introduce an additional optimization stage that is computationally expensive and often difficult to train to optimality.
of a discrete model defined over the original
3 Variance-Aware Likelihood Estimation of Diffusion models
In this section, we present an information-theerriice framework for variance-aware likelihood estimation diffusion models. While retaining the standard score matching objective 34 (9].
our method introduces a tighter, pointwise upper bound on the negative log- log-likelihood. We also
incorporate a modified forward process and an importance sampling scheme to reduce the variance of the Monte Carlo estimator. Notations and definitions are deferred to the Appendices. A.Iand A.2
In specific, we analyze the impact of noise variance (schedule) on likelihood estimation through the lens of Fisher divergence via thermodynamic integration 22 56 along the entire noise variance- regime space, which includes low-variance arbitrary noise regime (0- S of < ag) and a high- variance Gasussian channel (oğ < of < of). providing formal connection between score matching objectives to KI. divergence under arbitrary isotropic noise perturbations. This analysis leads to an exact expression of the mismatched entropy H(p(x), q(x; 0)), which improves optimization process, numerical stability and remains compatible with probabilistic modeling under non- Gans-ass noise setting. Our method toy example pipeline is illustrated in Fig. and Algorithms/ l anddd
3.1 Relationship between Score Matching and KI. Divergence
Beyond the pathology discussed in Section 2.2 real-world scenarios rarely align with the idealized Gaussian assumption. Sensor imperfections introduce structured noise such Poisson, uniform

---

quantization errors, or impulsive aalt-and-pepper patterns. Hence, extending KL-Fisher relations beyond Gaussian perturbations is not merely a theoretical extension, but addresses a pressing practical need in robust generative modeling. Motivated by these practical considerations, we investigate how training objectives that incorporate moise-perturbie lihood principle. Specifically, we highlight the role of Fisher divergence (see Def. (5) in characterizing the first-order sensitivity of KL divergence under small additive noise. By doing so, we generalize prior results [50] 51 69] from Gaussian to arbitrary isotropic noise distributions. Importantly, our derived relation ensures that acore-matching losses remain consistent with the first-order KL term, thus preserving the maximum- maximum-likelihood interpretation for broader class of models, including Poisson-flow 85 and Lévy-based diffusion frameworks [88]
distributions relate to the classical maximum like-
Theorem 1 (Score Matching as the Small-Noise Limit of KL Divergence). Ler X p(x) be arbitrary distributed random vector on IR,,, and let q(x: 0) be a parametric model with X q(x;; 0).
Define the perturbed observation
= o. +o
where & is random vector independent of X, satisfying B[ and Cov L = I. Let po? and l denote the densities of X under and with noise variance of, respectively. Suppose that the KL divergence (PL(Po? [0z)) is finite for sufficiently small oî. Then the following limit holds:
[ p(x) || logp(x) V log g(x;; 0)1l dx. 2 22
DKL DKo 1902) do?
(4)
-7-
i..,
DKL. (pI(((()) [|]] 0f- - 2 (p(x))la(x:0))
do?
where ((-lI-) denotes the Fisher divergence (equivalently, score matching objective) between p and q.
Proof. See Appendix. A.5.I
To further investigate the effect of additive noise on training objectives, we consider a second-order expansion of the KL. divergence in terms of the noise variance of. When both the true and model distributions are perturbed by small additive arbitrary noise, the KL divergence between them satisfies:
DKL. (xx 0)) = DKr. (px)(p( 0)) + oi
[(pllg) + o(o
(5)
2
While maximum likelihood aims to minimize the KL divergence directly, according to Theorem, [ score matching, defined through the Fisher divergence [34), does not directly minimize the KL
divergence explicitly. The two objectives coincide only in the limit o; + 0 where score matching captures the first-order sensitivity of KI. divergence additive noise. When of is not infinitesimal, the Fisher term may dominate, potentially leading to biased or unstable solutions. This result significantly generalizes the observation that score matching seeks to eliminate its derivative in the scale space f t= O) into arbitrary noise settings. In contrast, it is known that variational methods are known to be highly sensitive to noisy training data (see Appendix. B.6 for more details), which may give rise to many false extreme values, whereas score matching tends to be more robust to small perturbation in training data, suggesting that seeks parameters which lead to models robust to noisy setting.
Therefore, the generalized result in Theorem [ formalizing its role as a local approximation for likelihood-brie based training beyond the Gaussian setting [51]. To build upon this foundation, next consider the case where the forward process begins at a small but strictly positive variance, satisfying 0 < of < 1. This setting allows to integrate from of rather than from zero, yielding tractable and numerically stable lower bound while preserving consistency with the asymptotic result established above. We formalize this construction in the next section.
3.2 Bounding the Mismatched Entropy with Thermodynamic Diffusion
we can now consider a practical setting where the forward diffusion process (1) begins at small but strictly positive noise level 0 < o & 1. In this regime, we derive the tollowing approximation on the mismatched entropy (see Def. [4) based on denoising score matching
Building on Section 3.1
objectives [79] via thermodynamic integration [22] (6) along the interval [o8, of).
5

---

Proposition (Thermodynamic Decomposition of Mismatched Entropy). Consider the signal model (T) suppose p(x) and ( 0) have continuous second-order derivatives and finite second moments. Denote by p(y1) and (x)) the outpul signals of channel at time 1 = when inpuls are p(x) and q(x;; 0) respectively. Assume π (xx = NO, I), which is independens of A. Let ply:lx) denote the channel for any E [0. 1), then for arbitrary datapoint x and small of with O < og < 1:
of E B 1ogp(v/xx)) o(
H(p(x), 0)) = ((p(()),=(x))) - JoSM 0 (
2 o
(6)
Here, the denoising score matching objective is defined as
JOSM 0: 0² (-)) = ///² B logp(y.(x) s(y:; 0() do
1 i
2 2.
and (y;; V. log ((;;; 0) == - -i(y,, Thi 0)/o, is score network estimator.
Proof. Detailed proof defers in Appendix, A.5.2
The Proposition I preserves the theoretical guarantees of the Fisher-KL expansion while yielding a tractable and computable loss. The resulting bound remains asymptotically tight as of - 0, up to additive residual term of order o( The resulting bound remains asymptotically tight as Proposition[] can be interpreted through the lens of the cost of mismatch in statistical inference [35] In this setting, the true data distribution is p, but the decoder employs an estimator optimized for a mismatched model qo- The resulting mismatched entropy H(p. 9o) quantifies the average code length incurred under this model discrepancy. It shows that this quantity decomposes into three parts: a mismatched output distribution loss, a score approximation error arising from the use of s(:; 0) instead of the true score, and irreducible term linked the Fisher information of the channel.
3.3 Bounding the log-likelihood on Individual Datapoints
In many applications that benefit from likelihood optimization (29 33), it is desirable to evaluate the log-likelln oggliiklihood of individual data points, which can be highly sensitive to fine-scale variations and the precise values of pixel intensities [40]. To account for this sensitivity, we derive a pointwize lower bound that remains tractable and stable under finite noise injection.
Theorem 2.
Let px) = ((axx, T) denote the Gaussian channel at any time t E [0, 1]. With the same notations and conditions Proposition] !
we have
log ((x; 0) < (x)oyi((),,t Losu(o?; 0),
(8)
in which LDSM( : 0) is defined as
1 Ff
LDSM (o;; 0) = 2 2
Extn Vy. logp(y:(x) -Syr;
(9)
and H(ply) (x)x xx))) given by
H(p(y1 (x), (x))) Dr((p(y)(x))m(())) + H(p(y)(x))
(10)
Proof. Detailed proof defers in the Appendix. A.5.3
3.4 Variance Reduction with Importance Sampling
A key challenge in evaluating diffusion models lies in accurately and efficiently estimating the loss integral of Equation ( especially under a chosen noise variance schedule o?. In Appendix. B.2 we show that during training, the noise schedule acts estimation, and therefore plays a crucial role in efficient optimization.
importance sampling distribution for loss
Among such, popular choices include linear (28 cosine [54] and learnable parameterizations
with sigmoid-based schedules of = sigmoid(n(t)) remaining prevalent in state-of-the-art models. Accordingly, employ Monte Carlo estimator of this loss for evaluation and optimization.
40 62.

---

To further improve the efficiency and reduce variance in Monte Carlo estimation, we propose two importance sampling (IS) strategies (see Appendix. [B.3 for details).
Formally, the diffusion training objective can be expressed as an integral over the noise variance. It is shown in Appendix. [B. [B. thha denoising score matching loss (8) could be simplified to noise prediction model y'; ,, Me; 0) that directly infers the noise n that was used to generate y::
LUSM (o): (o; 0) 0)= / En-N
i
[o 2 2| n(y nc 1) do
(11)
In practice, the evaluation of the integral is time-consuming, and Monte-Cart methods are used to unbiasedly estimate it. In particular, the variance of the estimator has direct impact the optimization process, influencing both its stability and convergence speed. To mitigate this issue, it can be beneficial to decouple the integration variable used for loss estimation from the time variable employed during training. To this end, we introduce the negative log-SNR variable and rewrite the training objective (TT) as an expectation over this reparameterized variable, as shown followed. By interpreting the integral above as an expectation over the 7). the Monte Carlo estimation highlights the role of p(1)) as an IS distribution and (1)) as a noise variance- deeeended weighting function:
w(1)) I fu M15 0)13
CDSM (??; 0) = - E En ((0.)
(12)
Equivalently, optimizing p(r)) can be viewed as learning monotone mapping n7t) [0, 1 [no. m. which corresponds to the inverse cumulative distribution function (CDF) of p(n). Thus, the Monte
Carlo estimator achieves reduced variance and admits closed-fome inverse- CDF sampling, which motivates the following intuitively designed proposals.
Here, we sample v) p(T) with a continuous distribution p(n) x w(y) that renders the weight in (12) time-invariant, so that the noise prediction error remains scale-consistent across all r) E [No, m)]
values without any amplification or shrinkage during the estimation. This is analogous to the likliihoodweiighted and flow matching methods which conduct uniform sampling for all r7 [39 90].
Furthermore, to further minimize the variance of the Monte Carlo estimator, we can optimize the sampling distribution p(n) directly 40, 90). Specifically, we parameterize 17(t) as a monotonic neural network and adjust it to minimize the variance of the diffusion loss (see Appendix. B.3.2). While this learned proposal can effectively reduce Monte Carlo approximation variance, it introduces additional optimization overhead and potential instability. Empirically, we find that the hand -raftted proposal achieves better convergence speed and comparable NLL results, without requiring additional training objectives or neural components. As such, we adopt our designed IS as our default strategy.
3.5 Training-Fres Dequantization with Warm-up Noise
Real-world datasets usually contain discrete input, such images or texts, and must be dequantized when training continuous-density models. As discussed in Section 2.3 and more details in Ap- pendix. G while dequantization makes diffusion models applicable to discrete data, the training-test discrepancy originates from the mismatch in noise injection between training and inference/testing. To reduce this discrepancy, into training, inference and testing, that fit scamlessly into continuous density diffusion models, and preserves the original maximum- likelhoood objective without retraining.
introduce an arbitrary isotropic warm-up noise u V and inject
By Theorem D the generalized KL-Fisher identity holds for any isotropic noise, which means that dequantization schemes previously restricted to uniform and Gaussian noise can now be extended to, for instance, logistic and Laplacian perturbations. However, existing average log-likelihood bounds 27 74 are derived under mismatched entropy, defined as the sum of the KL divergence and the differential entropy. This motivates the generalization of the differential entropy formulation to arbitrary noise distributions, as follows, result known the de Bruijn identity in information theory.
oo W where X, & E R are independent random vectors, with V satisfying E O and Cov (VF = I. Assume the probability density function p(x) of X is twice continuously differentiable and decays sufficiently fast at infinity, and that the Fisher information J (X) exists and is finite. Then,
Proposition 2 (de Bruijn identity with arbitrary noise (60). Let X = x.X to,
d H (p(x))
1 5 (p(x)),
(13)
do?
2
007
7

---

where H--)) denotes the differential Shannon entropy (see Def. 7 and J.) denotes the Fisher infor- mation (see Def. 2
The key implication of (3) that the derivative is independent of the detailed statistics of the noise distribution. This suggests that the differential entropy forms a smooth manifold o, whose local geometry resembles an isotropic quadratie bowl. Combining Proposition 2 with the Theorem gives a characterization of the gap between the true discrete cross-entropy and its continuous surrogate. Such dequantization noise can provide tighter likelihood bound yet with additional training costs. Our detailed empirical settings of dequantization noise could be found in Appendix. 2.3
3.6 Numerical Stability
All theoretical developments in this work assume that the forward diffusion process begins at a small but strictly positive variance level o > O. This choice avoids the pathological behaviour of the of + 0 limit, which often leads to numerical instability in both training and evaluation (38] 40 69]. Accordingly, we restrict the variance schedule to the interval (og. of throughout all derivations and implementations. Truncating the lower limit of integration to o introduces an approximation error of order o( as discussed in Section 3.1, which negligible in both theory and practice. This ensures that our likelihood- iikelhhood-weighhedd score matching objective remains a consistent estimator of the log-libe nogliikei while simultaneously improving numerical stability. Empirically, we observe that this design remains robust across diverse noise distributions, further confirming the practical reliability.
Related Works
in diffusion models focus on their role in 92 (e.g., by approximating posteriors or solving reverse-time ODEs [50] 90) ). contrast, our work examines how the forward variance schedule influences both robustness and likelihood estimation. Proposition further generalizes the classical de Bruijn identity 112 from Gaussian to arbitrary noise perturbations. Whereas prior information- theoretic studies [18
Most prior analyses of variance schedules
49 54
reverse-time sampling [2 40 47 55
62 68 91
2
80 94 have characterized the connection of KI. and Fisher divergences under Gaussian smoothing 50 51 69 933, we extend these relationships to non-Gaussian noise families. Finally, [43] studies relative-entrope interpretations of denoising objectives and connects them to in the sense of our framework.
mean-square error, but does not address mismatched estimation 335 78
5 Experiments
In this section, we present our training procedure and experiment settings, and our ablation studies to demonstrate how our techniques improve the likelihood of diffusion models. We report negative log-likeli leeliikl (NLL, in bits/dim) and Fréchet Inception Distance (FID) scores for all experiments, and compare convergence speed across models. Then we evaluate our model on a lossless compression benchmark against several competitive baselines in neural compression literature. Note that we focus here on pushing the state-of-the -ftt in density estimation, and while we report FID for completeness, we defer sample quality optimization to future work. Additional results can be found in Appendix D
Datasets and Implementation We evaluate on CIFAR-10, anti-aliased ImageNet-32 dataset, ImageNet-64 and -128. No data augmentation applied. We adopt the same architecture and hyperparameters as the VDM model [40], including a U-Net consisting of convolutional ResNet blocks without any downsampling. Unlike embeddings of diffusion time t, we embed the log-SNR value y)()) or its reverse CDF time embedding instead. This modification better reflects the underlying noise scale and leads to improved likelihood estimation with importance sampling trick in practice.
Selection of Noise Schedules We consider two representative choices: 1.
(VP): a + t = 1, ensuring unit marginal variance throughout the forward process. 2. Straight- (SP): a. + oo 1, corresponding to linear interpolation data space as in 1 49 We omit VE schedules, since in our preliminary runs the best VE model attained only 3.27 bits/dim on CIFAR-10, substantially worse than VP and SP. Moreover, to further illustrate the effects of the noise variance, consider to evaluate various variance functions of shown in Appendix. B.4
8

---

Table 1: Negative log- likelihoodd (NLL) in bits per dimension (BPD), FID, and number of training iterations (Iter., in million) on CIFAR-10 10 and ImageNet- 32 datasets. "/" indicates results not reported or not applicable. *Denotes results obtained on the original ImageNet-32 release. Boldface denotes the best performance within each column, and blue text marks the second-best.
Models
CIFAR-10
ImageNet-32
4
(The models with IT methods)
ScoreDDE
4.06
ScoreODE (third order)
3.38
2.95
4.04
ScoreFlow [0
2,80
5.34
1.6
3.79
11.20
1.6
Flow Matching
2.99
6.35
0.391
3.53
5.02
0.25
Interp.
8.49
i-DODE with
2.56
3.64
10.31
2.25 2.55"
modele
variational methodss
VDM
3.72'
DiffFin
55
2.62
11,20
3,46
/
Mol JAN
2.55
17.62
3.67'
13.19
W-PC DM
46
(VDM
40
Weight))
2.35
6,25
3.32
10
P.PCMM
weight)
Ours (SP with 1S)
2,49
0.3
3,02
1
0.3
Ours with
2.50
10.18
14.76
Table 2: Comparison between our proposed model and other competitive models in the literature in terms of expected negative log likelihood on the test set computed as bits per dimension (BPD). Results from existing models are taken from the literature. "y" indicates results not reported or not applicable. Boldface denotes the best performance within each column, and blue text marks the second-best bestt.
Model
Type
ImageNet-32
ImageNet-64
ImageNet-128
PixelCNN
M7
Autoregressive
3,83
3.57
Flow
4.02
3.81
FLOW+
Flow
3,86
3.69
Sparse Transformer
0
Autoregrestive
/
3.44
Very deep VAE
8
Autoencoder
3,80
3,52
Improved DDPM
54
Dillusion
3.54
Routing
Autoregressive
Flow Matching
45
Flow
3.53
3,31
2,90
VDM 50
Autocncoder
3.72
3,40
1
LP-PCDM
46
Diflusion
3.52
3.12
2.91
W-PCDM
Diffusion
3.32
2.95
2.64
Diffusion
3.01
2.91
2.59
5.1 Likelihoud and Samples
Table summarizes experimental results CIFAR-10 and ImageNet-32 (more details in Ap- pendix. B.5). At the request of one of the reviewers we also ran our model on additional data sets of higher resolution images baseline (Table. 2 On four V100 GPUs with CIFAR-10, slightly faster (2.34 vs. 2.04 iterations/sec) compared to VDM, due to the removal of the additional networks. Typically, achieving benchmark likelihood estimation performance requires several million training iterations, and the training process usually takes a week, or even a month or longer. Table shows that obtain state-of-the-ar NLL ImageNet-32/64/11 under directly comparable settings, while reducing training cost from millions of iterations in prior work to about 300 thousand.
model trains
Ablation Analysis and Additional Experiments Due to the high training cost, we conduct ablation studies only CIFAR-10. In Table we report both NLL and FID under different noise variance endpoints. We ablate the effect of the different IS weightings and reverse CDF embedding by comparing it standard sinusoidal time encoding in Appendix. B.5 All models trained for 300K steps with identical settings, except for the variance schedule. For fair comparison, we follow the VDM protocol [40] and evaluate models using the ELBO-bassed lower bound (see Appendix, (B6)..
Our proposed bounds achieve state-of-the-ar density estimation on ImageNet datasets and match the best reported results CIFAR-10 10. We find that increasing the warm-up noise improves NLL but

---

Table 3:
Comparison of NLL on CIFAR- 10 and ImageNet-32 with different warm-up noise.
Likelihood Estimation Bound
CIFAR-I0
ImageNet-32
Gaussian
Laplace
Logistic
Uniform
Gaussian
Laplace
Logistic
Uniform
ELBO Our Our (VP+
2.69
2:70
2.71
2.71
3.53
3.53
3.54
3.55
2.49
2.49
2.50
2.52
3,01
3,00
3.02
3.09
2.50
2,51
2.51
2.53
3,00
3.01
3.03
3.08
slightly degrades FID. This trade-off arises because larger noise suppresses pixel-level fluctuations and stabilizes training, while too little noise sufficiently regularize the likelihood objective, leading to worse likelihood estimates, despite improved sample quality. We provide details in the Appendix. C
We also observe that the gap between ELBO and our IT-baase bound enlarges with increasing noise
levels. This discrepancy arises from the reconstruction loss used in ELBO [40 [55] which assumes
conditional independence of pixels given the latent variable yo. As noise increases, the conditional distribution p(yo/x) becomes more diffuse around its mode, thereby amplifying the approximation error. These findings highlight a fundamental intuition between likelihood accuracy and perceptual quality under different noise configurations. We leave the further study of this direction in the future.
5.2 Examining the Warm-up Noise Injection
We further examine the effect of different noise distributions, Gaussian, Laplace, logistic, and Uniform, each scaled to equal variance. As shown in Table Gaussian noise performs best, followed by Laplace and logistic, while Uniform lags notably behind. This supports our intuition that heavier- tailed, exponential-family noises yield more stable training and improved likelihood estimation. Detailed theoretical discussion and additional analyses, including the connection to differential entropy, Fisher information and manifold hypothesis, are provided in Appendix G
5.3 Lossless Progressive Coding
Table 4: Lossless compression per- formance on CIFAR-I0 in bits/dim.
As shown in prior work [88 40). likelihood based generative models can be viewed as latent-variable models for neural loss- less compression. We adopt this perspective and implement a Bits-Back Coding scheme (27 using our proposed model as the latent component. On CIFAR-10, 10, our method achieves shorter average code lengths, measured in bits per dimension, compared to several strong baselines (see Table 4). We leave this avenue of research for further work.
Model
Compression Rate (bits/dim)
FLIF
4 14
LBB
3.12
IDF
53
3.26
VDM 40
2.72
ARDM
2.71
W. PCDM
46
2.37
6 Conclusion
Our generalized KI-Fisher relationship transforms noise injection from a theoretical consideration into a widely applicable practical strategy. This principled framework validates existing approaches employing non-Gaussian perturbations and offers new theoretical tools for tackling real-world generative modeling challenges. Our analysis shows that, in the small-noise regime, the score matching objective asymptotically approximates maximum likelihood. Minimizing this objective yields consistent improvements in likelihood across diverse noise schedules, variance settings, and datasets. When critically combined with importance sampling, our approach achieves on-par likelihood on CIFAR-10 and state-of-the-ar likelihood on ImageNet datasets. Our results also motivate future exploration of information-theorrtiic objectives in generative modeling.
Limitations Our method improves likelihood estimation but does not construct a generative diffu- sion process under alternative noise. As result, our method is limited to likelihood evaluation and cannot be directly used for sampling or generation. Dequantization bound, diffusion/drift coefficient and variance configurations are not fully explored. Due to resource limitations, we didn't explore tuning of hyperparameters and network architectures, which are left for future work. We leave full discussion and future extensions to the Appendix.
10

---

References
[1] Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In The Eleventh International Conference Learning Representations, 2023.
[2] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytie an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In International Conference Learning Representations, 2022.
[3] Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research, 2022. Expert Certification.
[4] Huanran Chen, Yinpeng Dong, Shitong Shao, Zhongkai Hao, Xiao Yang, Hang Su, and Jun Zhu. Diffusion models are certifiably robust classifiers, In The Thirty-eightth Annual Conference on Neural Information Processing Systems, 2024.
[5] Minhua Chen and John Lafferty. Mismatched estimation and relative entropy in vector Gaussian channels. In 2013 IEEE International Symposium on Information Theory, pages 2845-2849, IEEE, 2013.
[6] Wei Chen, Shigui Li, Jiacheng Li, Junmei Yang. John Paisley, and Delu Zeng. Dequanti- fied SnffusiionSccrüdinnee bridge for density ratio estimation. In Forty-second International Conference on Machine Learning, 2025.
[7] Zhaomin Chen, Chai Kiat Yeo, Bu Sung Lee, and Chiew Tong Lau. Autoencoder-basee network anomaly detection. In 2018 Wireless telecommunication. symposium (WTS), pages 1-5, IEEE, 2018.
[8] Rewon Child. Very deep VAEs generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021.
[9] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint orXiv:1904.10500 2019.
[10] Kristy Choi, Chenlin Meng, Yang Song, and Stefano Ermon. Density ratio estimation via infinitesimal classification. In International Conference on Artificial Intelligence and Statistics, pages 2552 2573.. PMLR, 2022.
[11] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1310-1320. PMLR, 09-15 Jun 2019.
[12] M. Costa. A new entropy power inequality. 3 31(6):751-760, 1985.
IEEE Transactions on Information Theory.
[13] M. Costa and T. Cover. On the similarity of the entropy power inequality and the brunn- minkowski inequality (corresp.). IEEE Transactions Information Theory, 30(6):837-83,, 1984.
[14] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[15] Zihang Dai, Zhilin Yang, Fan Yang, William W. Cohen, and Ruslan Salakhutdinov. Good semi-supervised learning that requires a bad GAN. In Proceedings of the 31st International Conference Neural Information Processing Systems, NIPS' 17, page 6513-6523, Red Hook, NY, USA, 2017. Curran Associates Inc.
[16] Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thomton,, Yee Whye Tch, and Arnaud Doucet. Riemannian score-based generative modelling. Advances in neural information processing systems, 35:2406-2427 2022.
[17] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In International Conference Learning Representations, 2017.
11

---

[18] Luyao Fan, Jiayang Zou, Jiayang Gao, and Jia Wang. Differential properties of information in jump-diffusion channels. arXiv preprint arXiv:2501.05708, 2025.
[19] R. A. Fisher. Theory of statistical estimation. Mathematical Proceedings of the Cambridge Philosephical Society, 22(5):700-725, 1925.
[20] Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian. Practical Poissonian-Gausee Gaussian noise modeling and fitting for dingle-iage raw-data. IEEE Transac- tions on Image Processing, 17110):777711744 2008.
[21] Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 7518-7525 2020.
[22] Andrew Gelman and Xiao-Li Meng. Simulating normalizing constants: From importance sampling to bridge sampling to path sampling. Statistical science, pages 163-185 1998.
[23] Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Advances in Neural Information Processing Systems, 36: 16:16693-6775, 2023.
[24] Dongning Guo. Relative entropy and score function: New informaation-esiimatoo relationships through arbitrary additive perturbation. 2009 IEEE International Symposium on Information Theory, pages 814-818, 2009.
[25] Dongning Guo, S. Shamai, and S. Verdu. Mutual information and minimum mean-square error in Gaussian channels. IEEE Transactions on Information Theory, 51(4):1261-1282, 2005.
[26] Leonhard Helminger, Abdelaziz Djelouah, Markus Gross, and Christopher Schroers. Lossy image compression with normalizing flows. In Neural Compression: From Information Theory Applications Workshop @ ICLR 2021, 2021.
[27] Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2722-2730. PMLR, 05 Jun 2019.
[28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in nearal information processing systems, 33:684006851, 2020.
[29] Jonathan Ho, Evan Lohn, and Pieter Abbeel. Compression with flows via local bits-back coding. Advances Neural Information Processing Systems, 32, 2019.
[30] Yung-Han Ho, Chih-Chun Chan, Wen-Hsiao Peng, Anfic: Image compression using augmented normalizing flows. IEEE Open Journal of Circuits and Systems, 2:613-626, 2021.
Hsueh-Ming Hang, and Marek Domañski.
[31] Emiel Hoogeboom, Taco Cohen, and Jakub Mikolaj Tomezak. Learning discrete distributions by dequantization. In Third Symposium on Advances in Approximate Bayesian Inference, 2021.
[32] Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. International Conference on Learning Representations, 2022.
[33] Emiel Hoogeboom, Jomn Peters, Rianne Van Den Berg, and Max Welling. Integer discrete flows and lossless compression. Advances in Neural Information Processing Systems, 32, 2019.
[34] Aapo Hyvärinen and Peter Dayan. Estimation of non-normal statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
[35] Jiantao Jiao, Kartik Venkat, and Tsachy Weissman. Relations between information and estima- tion in discrete-time Lévy channels. IEEE Tronsactions on Information Theory, 63(6):3579- 3594, 2017.
12

---

[36] Norman L Johnson, Samuel Kotz, and Narayanaswamy Balakrishnan. Continuous univariate distributions, volume volume 2. John wiley & sons, 1995.
[37] Tero Karras, Miika Aittala, Samuli Laine, and Timo Aila. Elucidating the design space of diffusion-basee generative models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc.
[38] Dongjun Kim, Scungjae Shin, Kyungwoo Song, Wanmo Kang, and II-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In International Conference on Machine Learning, pages 1 11201-11228. PMLR, 2022.
[39] Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives the ELBO with simple data augmentation. Advances in Neural Information Processing Systems, 36:65444-6556,, 2023.
[40] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21777, 2021.
[41] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Truck Proceedings, 2014.
[42] Durk Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1 1x convolutions. Advances in neural information processing systems, 31, 2018.
[43] Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg, Information -tatomation--horetiic diffusion. In The Eleventh International Conference on Learning Representations, 2023.
[44] Luis A. Lastras-Montaäny Information theoretic lower bounds on negative log likelihood. In International Conference Learning Representations, 2019.
[45] Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 35:22870-2223 22882, 2022.
[46] Henry Li, Ronen Basri, and Yuval Kluger. Likelihood training of cascaded diffusion models via hierarchical oolume-preserving maps. In The Twelfth International Conference on Learning Representations, 2024.
[47] Shigui Li, Wei Chen, and Delu Zeng. Evodiff: Entropy-aware variance optimized diffusion inference. arXiv preprint arXiv:2509.26096, 2025.
[48] Marten Lienen, Marcel Kollovieh, and Stephan Günnemann. Generative modeling with bayesian sample inference. arXiv preprint 2xXiv:2502.07580, 2025.
[49] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference Learning Representations, 2023.
[50] Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In International Conference on Machine Learning, pages 14460. PMLR, 2022.
[51] Siwei Lyu. Interpretation and generalization of score matching. In Proceedings of the Twenty- Fifth Conference Uncertainty in Artificial Intelligence, UAI '09, page 359 359-366, Arlington, Virginia, USA, 2009. AUAI Press.
[52] Chenlin Meng, Jiaming Song, Yang Song, Shengjia Zhao, and Stefano Ermon. Improved autoregressive modeling with distribution smoothing. In International Conference on Learning Representations, 2021.
[53] Krishna Narayanan and Arun Srinivasa. On the thermodynamic temperature general distribution. arXiv preprint arXiv:0711.1460, 2007.
13

---

[54] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162-8171. PMLR, 2021.
[55] Beatrix Miranda Ginn Nielsen, Anders Christensen, Andrea Dittadi, and Ole Winther. Diffenc: Variational diffusion with learned encoder. In The Twelfth International Conference on Learning Representations, 2024.
[56] Yosihiko Ogata. monte carlo method for high dimensional integration. Numerische Mathe- matik, 55:137-157, 1989.
[57] Yidong Ouyang, Liyan Xie, Hongyuan Zha, and Guang Cheng. Transfer learning for diffusion models, Advances in Neural Information Processing Systems, 37:1 (013366666699999 2024.
[58] Daniel P Palomar and Sergio Verdú. Gradient of mutual information in linear vector gaussian channels. IEEE Transactions Information Theory, 52(1):141-154, 2005.
(59) Miquel Payaro and Daniel P. Palomar. Hessian and concavity of mutual information, differ- ential entropy, and entropy power in linear vector gaussian channels. IEEE Transactions on Information Theory, 55(8):3613
2009.
[60] Olivier Rioul. Information theoretic proofs of entropy power inequalities. IEEE transactions on information theory, 57(1):33-55, 2010.
[61] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68, 2021.
[62] Subham Sahoo, Aaron Gokaslan, Christopher M De Sa, and Volodymyr Kuleshov. Diffusion models with learned adaptive noise. Advances Neural Information Processing Systems, 37:105730-105779 2024.
[63] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
[64] Yirong Shen, Matthias Seeger, and Andrew Ng. Fast Gaussian process regression using KD-trees. Advances in neural information processing systems, 18, 2005.
[65] Jyotirmai Singh, Samar Khanna, and James Burgess. Squeezed diffusion models. arXiv preprint ar0iv:2508. 14871, 2025.
[66] Jon Sneyers and Pieter Wuille. FLIF: Free lossless image format based on MANIAC compres- sion. In 2016 IEEE international conference image processing (ICIP), pages 66-70. IEEE, 2016.
[67] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML'15, page 2256-2265. JMLR.org. 2015.
[68] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[69] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415- 1428, 2021.
[70] Yang Song and Stefano Ermon. Improved techniques for training score-basee generative models. Advances in neural information processing systems, 33: 221224244444 2020.
[71] Yang Song. Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In International Conference Learning Representations, 2018.
14

---

[72] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[73] Lucas Theis, Tim Salimans, Matthew D Hoffman, and Fabian Mentzer. Lossy compression with Gaussian diffusion. arXiv preprint urXiv:2206.0888 2022.
[74] Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference Learn- ing Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[75] J T'ownseed, T Bird, and D Barber. Practical lossless compression with latent variables using bits back coding. In 7th International Conference on Learning Representations, ICLR 2019, volume 7. International Conference on Learning Representations (ICLR), 2019.
[76] Benigno Uria, lain Murray, and Hugo Larochelle. Rnade: the real-valued neural autoregressive density-estimator. In Proceedings of the 27th International Conference on Neural Information Processing Systems- Volume 2, NIPS' 13, page 2175-2183, Red Hook, NY, USA, 2013. Curran Associates Inc.
[77] Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. International conference on machine learning. pages 1747-1756. PMIR, 2016.
[78] Sergio Verdú. Mismatched estimation and relative entropy. IEEE Trunsactions on Information Theory, (6()):371223720, 2010.
[79] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1674, 2011.
[80] Ben Wan, Tianyi Zheng, Zhaoyu Chen, and Jia Wang. Enhancing the accuracy of generative adversarial networks with Fokker- Planck equations. Neurocomputingg 638: 130158, 2025.
[81] Ben Wan, Tianyi Zheng, Zhaoyu Chen, Yuxiao Wang, and Jia Wang. Pruning for sparse diffusion models based on gradient flow. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
[82] Andre Wibisono. Varun Jog, and Po-Ling Loh. Information and estimation in Fokker-Plann Planckk channels. In 2017 IEEE International Symposium on Information Theory (ISIT), pages 2673- 2677, 2017.
[83] Andre Wibisono, Yihong Wu, and Kaylee Yingxi Yang. Optimal score estimation via empirical Bayes smoothing. In The Thirly Seventh Annual Conference on Learning Theory, pages 4991. PMLR, 2024.
[84] Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: an out-of-distribution detection score for variational auto-encoder. In Proceedings of the 34th International Conference Neu- ral Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc.
[85] Yilun Xu, Ziming Liu, Max Tegmark, and Tommi S. Jaakkola. Poisson flow generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[86] Shahar Yadin, Noam Elata, and Tomer Michaeli. Classification diffusion models: Revitalizing density ratio estimation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[87] Ruihan Yang and Stephan Mandt. Lossy image compression with conditional diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[88] Eunbi Yoon, Keehun Park, Sungwoong Kim, and Sungbin Lim. Score-based generative models with Lévy processes. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
15

---

[89] Kaiwen Zheng, Yongxin Chen, Huayu Chen, Guande He, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Direct discriminative optimization: Your likelihood- basodd visual generative model secretly a GAN discriminator. In Forty-second International Conference on Machine Learning. 2025.
[90] Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood estimation for diffusion ODEs. In International Conference Machine Learning. pages 22363-42338 PMLR, 2023.
[91] Tianyi Zheng, Cong Geng, Peng-Tao Tao Jiang, Ben Wan, Hao Zhang, Jinwei Chen, Jia Wang, and Bo Li. Non-uniform timestep sampling: Towards faster diffusion model training. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 7036-7045, 2024.
[92] Tianyi Zheng, Jiayang Zou, Peng-Tao Jiang, Hao Zhang, Jinwei Chen, Jia Wang, and Bo Li. Bidirectional Beta-tuned diffusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[93] Jiayang Zou, Luyao Fan, Jiayang Gao, and Jia Wang. Convexity of mutual information along the Fokker-Planck flow. arXiv preprint urXiv:2501.050% 2025.
[94] Jiayang Zou, Luyao Fan, Jiayang Gao, and Jia Wang. A revisit to rate-distortion theory via optimal weak transport. arXiv preprint arXiv:2501 0.0662,, 2025.
16

---

Contents of the Appendix
Introduction
2
Background
Incremental Gaussian Channel and Maximum Likelihood Estimation
2.2
Likelihood of Diffusion Models
2.3
Dequantization for Density Estimation
Variance-Aware Likelihood Estimation of Diffusion models
3.1
Relationship between Score Matching and KL Divergence
3.2
Bounding the Mismatched Entropy with Thermodynamic Diffusion
3.3
Bounding the log-likelihood on Individual Datapoints
3.4
Variance Reduction with Importance Sampling
3.5
Training-Frre Faree Dequantization with Warm-up Noise
3.6
Numerical Stability
Related Works
Experiments
5.1
Likelihood and Samples
5.2
Examining the Warm-up Noise Injection
10
5.3
Lossless Progressive Coding
10
6
Conclusion
10
Contents of the Appendix
17
Preliminaries and Reviews
19
A.1
Notations
19
Detinitions
19
A.3
Standard Diffusion Models
20
A
4
Lemmas
20
Proof and Remark Theorems
25
A.5.I
Proof of Theorem.
25
A.5.2
Proof of Proposition.
27
A.5.3
Proof of Theorem
29
A.5.4
Proof of Proposition]
29
Improve the Likelihood Estimation Bounds
30
B.
Arriace-AAwary Likelihood Bounds
30
17

---

B
The Noise Schedule Matters
31
13.3
Variance Reduction with Importance Sampling
32
B.3.1
Designed IS
32
B.3.2
Learned IS
33
B.4
Log-SNR-Timed Channel with Different Variance Functions
33
B.5
Empirical Results
34
B.6 Likelihood Estimation Comparison for bound and ELBO on CIFAR-I0
36
Samples Quality and FID)
38
Experimental Settings
39
Consistency Across Predictors and Corresponding Objectives
40
Numerical Stability
Randomized Distribution Smoothing and Dequantization
41
18

---

A Preliminaries and Reviews
We summarize the key notations and assumptions used in our theorems. The data distribution is p(x). and the model is q(;; 0). where 0 restricted to a parameter space @ The vector Gaussian channel follows Y, a; X o,N, where dz RR IR controls the time-dependent coefficient, t E [0, 1] represents the time horizon, and N ~ (n; 0, I) is Gaussian noise. The input x ~ p(x), and the output yz ~ p(y.), with both as column vectors of appropriate dimensions.
A.1 Notations
In this paper, we are working on the Euclidean space RD for some D 2 1. We denote the in-inner D
product between vectors u = (UI....; ua). V = (V1...., va) E R as T v = (u, v) Ei=1 u,4.
For a symmetric matrix A E DxD, the notation u Au 2 0 for all u E RR For symmetric matrices A, B E R DxD, the notation A > B means A B z 0 positive semidefinite. Throughout, let I e DD denote the identity matrix.
0 means
positive semidefinite, i.e..
For a differentiable function f IRD + R. let V E I P denote the gradient vector at x E TR of the partial derivatives: (Vf(x)), 0f(x) Let V2 DxD D be the Hessian matrix of second Ux:
((x) Let (x() Tr(v2 be the Laplacian. We C partial derivatives: (V2f(x)),, 8x
to denote all continuous functions, and let C* denote the family of functions with continuous k-th order derivatives.
For vector field v IRD - R with v(x) = (v)(x)..... va(x)) E RDD, let Vv RD + DxD be the Jacobian matrix of mixed partial derivatives: (Vv(x))a,) @vi(x) the divergence of v, defined by
Let V V R R be
8x,
D v) Σ
@vi(x) (v(Vvxx))
Ox:
Let Vr C. R be the region (an D-dimensional ball) bounded by the closed, piccewise-smooth, oriented surface Sr, which is the D-sphere of radius centered at the origin. At any point y S,, the symbol es, (y) denotes the outward-pointing unit normal vector to S,.. Under the notation ds, [s,
A.2 Definitions
Let P)R denote the space of probability distributions p over R. which are absolutely continuous with respect to the Lebesgue measure and have finite second moment [ < co. We identify
probability distribution E P(RD) with its probability density function with respect to the Lebesgue measure, which we also denote by p: RDD + R. so p(x) O and SRD p(x)dx== 1.
We say p is absolutely continuous with respect to another distribution v, denoted by p < v, v(A) 0 implies p(A) = 0 for any A C R²; if p and v both have density functions, then p << means v(x) = 0 implies p(x) = 0 for all x E DR.
Definition 1. Let H(-)) P(B
IR be the differential Shannon entropy:
H(p) = - -E, pl - p(x) log p(x) dx.
(14)
Definition 2. Let J(.): PP( + IR be the Fisher information:
{pp) = E.
= -- -EAIA log pl,
(15)
and we define J(p) = +oo if p does not have differentiable density. The second equality in the definition of J {p) above follows by integration by parts.
Note that the expression in (15) is special case (with respect to translation parameter) which does not involve an explicit parameter as in its most general definition
'The parameterized Fisher information matrix is defined with respect to parameter by J(p)
Ep Vo log p(x; 0) VJ log p(x: 0)).
19

---

Definition 3. For probability distributions & v on RP, the Kullback-Leine (KL) divergence or the relative entropy of p with respect is defined by:
#11 V
- p(x) log p(x) v(x)
DRI(Pll)> Ep log
dx.
(16)
RD
Definition For probability distributions p < v on RD, the mismatched entropy or the cross entropy of p with respect to v is defined by:
dx. Definition 5. 1f p and have differentiable density functions, then the relative Fisher information of with respect to is defined by:
H(p, v) ( =
p(x)
(17)
JRD
log v(x)
-/00 p(x) || V log 2 "112] - 20
p(x) v(xx)
1
[(pl = Ep
V log
dx.
(18)
We recall that the KI. divergence, DRL. (-Il), corresponds to the Bregman divergence of the negative entropy. Similarly, the relative Fisher information (RFT), T(-II),, can viewed as the Bregman divergence of the Fisher information. Moreover, half of the RFI is equivalent to score matching, as defined in 34
A.3
Standard Diffusion Models
Consider Gaussian diffusion process [28). which starts from clean data x and defines a sequence of progressively noisier versions, denoted by channel outputs yt. where runs from (least noisy) I (most noisy). In the sampling process, given T' we uniformly discretise the time interval into T timesteps, each of width 1/T. Let t(i) = i/TT denote the current timestep and s(i) preceding one 40)
1)/T the
Forward Process.
The forward process is defined by conditional Gaussian distribution:
p(yx)x) Na o
and, by the Markov property,
p(y:lys) NN(aaayy. o I),
where
Ot
o..
0
o²
Reverse Process. is also Gaussian:
As established in prior work
28
40
the conditional distribution p(yslye, x)
o.. a, of
2. tl of
x)y.ly,.)) = -N
y.
o7
As the ground truth not available during the reverse process, it is approximated by neural network x(y;, t; f), parameterised by 0. The learned reverse kernel becomes:
!oa of
og y: + of x(y,,
oz
of )
p(y=lye:0)= =N
A.4 Lemmas
In this section, we introduce all Lemmas in order to prove the Theorems in the next section.
Lemma Let X E RD be random vector with density p(x). Suppose Vp RDD is an arbitrary
independent random vector with zero mean and identity matrix I. Define X a,X + og W, and let p(x) denote the density of X. Then for every RDD as of 0+, we have:
do?
dx)
2 Ap(x),
(19)
07=01
20

---

D 8
where Ax )az)) the usual Laplacian in RDD
Σ j=1
Formula (19) allows the derivative w.r.t the energy of the perturbation of to be transformed to the second derivative of the original pdf. what followed we provide the proof for Lemma [ which is slightly different than that in [53]. Note that Lemma does not require the distribution of the perturbation to be symmetric as is required in [53]
Proof. Let p()) denote the probability density function of the random vector X, and define its characteristic function as follows:
o(k, of) = exp (ik (ikTx))
E
(20)
Given that X is defined by
x= a ao,
(21)
where and are independent, the characteristic function factorises follows:
o(k,o,) = E ak (o.X+o, v)
(22)
E ik a.X
IR c 0,
(23)
p(akk,0) 0)
x
E e ik
(24)
charactersstcc
characteristic functtion o
Expanding the exponential function via Taylor's theorem gives
(o,k T 4² +0(0)))
ik a. =1 + io, v)
(25)
2!
Since E[y]
and Cov y = I, it follows that
Eik" T D = O,
(26)
[[(k T 4 4 = k M k = |k|
(27)
Taking expectations, obtain
o, 2
E 1k a.
=
| + o(0?).
(28)
Thus, the characteristic function satisfies
(1
o, 2
o(k of)) = plack, 0)
|||| + o(o?))
(29)
Recalling the inverse Fourier transform, the probability density function is given by
1
Poo(X) (2T)) R
ik "ok,, (o(ko,;)kk
(30)
Substituting the expansion of Φkk, of), we obtain
e -kk xk= (0((ark))) 1 (1-
o² || 2 + o(o7)) dk.
Po;(xx) (2T)) RD
(31)
Since
(x(x)= (2T) JRp x &o 0)dk,
(32)
it follows that the difference Poş (x( corresponds multiplying 6(ack, 0) by
ot | + o(o7))
2
21

---

A standard result from Fourier analysis states that multiplication by -| corresponds to the application of the Laplacian A in the spatial domain, defined
| in the Fourier domain
D
82
dx
j=1
Thus, for small oz we obtain
Px7(x)= = (x( + of
(x) + o
(33)
2
Since (x( corresponds to p(x) in the absence of noise, this completes the proof of Lemma |
Lemma 2 (Vanishing boundary flux). For an arbitrary input distribution p(x). an assumed input distribution q(:; 0) and vector Gaussian channel ply xx) Naax, ofI) in ( denote the corresponding output densities by ply)) and q(y;; 0) at time t E [0, 1]. For r > 0, let B, := {y e Rº +yyll S r} be the closed D-ball and let Sp == dB, be the (D-1) -)-1e-sphere with outward unit normal es, (y) = y y/lly|l- Assume p(y:) and q(y:; 0) are C with well-defined Vy and Dy, and that
Ep log y 0) < oc. Then the boundary integrals
L = lim
doy: 0) Vyp((y,) es, (y.)
(34)
1-000 Js.
{ ply:)
L2 = lim
1-000 Js. 2 ((v;;0) Vy, es, (y:) ds = O.
(35)
Proof. First, that the limits L1 and Ly obviously exist, each of these limits can be expressed as the sum of two converging integrals. We prove (34); the argument for (35) is analogous. For brevity, omit the subscript and write y for yv. Let es, (y) to unit vector normal to S,- the point Under this notation ds, = ||ds,||es, (y). We integrate over 2 the surface integral in (34) and apply Green's identity to find the relations. Set
f(r)
log 4(y) Vply) ds..
Sr
By the coarea formula (spherical coordinates),
[e ((r)dr=
220 Vply) (log ((y) es, (y)) dy.
(36)
Using the product rule and the divergence theorem on B,,
J Vply) (log ((y) es, (y)) dy
log ((y) es, (y) dsr - J p(y) V v-ooog((y) es, (y)) dy.
lim p(y) ro
(37)
s.
Letting oo we bound the two terms on the right-hand side.
First term. By the coarea formula,
1.0 J.J. logey)) [ /
(ply) log (lyg())) r-
0,
because Ep[ log g(y)]l < co. Hence p(y) log g(y) ds along the full sequence - 00 (e.g..
via Cesàro argument).
22

---

Second term. the relation
Now we note that the absolute value of the divergence in the second term satisfies
(logg(y)es,,))) = Vg(y))
es. (y)l
V
q(v)
[Vg(y)]
(38)
yly)
Hence, we have
)2) à
\V((y)l
12
((x)
{ |1 arail
1yy, axx1?
1)
Σ
(2
exp (
dx.
q(y)
20 ((y)
of
20
16
Y 0x
Y = [1
Σ
o
R
Y axxi
Y = y
Y
xx
2
E
Y y=
(39)
of
Integrating w.r.t. p(y) dy and applying Jensen, we can write the chain of inequalities
{
Y (x=x)) 3
ply) [V dy - Epp
E
R"
((y)
of
Y y- ))}
[ xx )
Y
E
E
of
[|
2
Ep
of
00,
(40)
which means that
[I] S {F 1/2 <..
Epp (og
Therefore the right-hand side of (37) is finite, so
J f(r) <0.
Finally, f is locally absolutely continuous in r (by the smoothness of p and q and the coarea formula), hence lim f(r) exists. Since Jo f(r dr < cc, this limit must equal O. Thus L, = 0. The proof of L.2 = 0 follows by the same steps with q Vq in place of log q Vp.
Lemma Fix [0, 1] and consider the Gaussian channel in ( Let p(y:) and q(y;; 0) be the oulput densities induced by inputs p(x) and q(x; f), respectively. Assume p(.) and q(: O) are sufficiently smooth and integrable so that differentiation under the integral sign is justified and the boundary terms in Lemma 2 vanish. Then the derivative of the mismatched output cross-entrop
(ooo))).(yye:0) = =ff p(y:) log g(v:; 0) dy.
with respect to the noise variance of is
d H(ply.). 0)) - JRD dop
1 /
(Vg(yri)) Vp(y.)
+ v( p(y.) vv(xr:)) dy:. (((:))))
4(y::0)
(41)
where a b= (a, b) denotes the Euclidean inner product.
23

---

Proof. By definition of cross-entropy in Def. 4
((x:))).((xx:0)) = - { p(y.) log q(y:; 0) dy.
Differentiating w.r.t. of (justified by the assumed regularity) yields
d (-f plogg) = =-
f log
Op dyt do?
/
i
log q dyy
(42)
do?
where we abbreviate p(y+) and q = g(y;; 0).
Under Gaussian smoothing (heat equation),
Op
dq
i 0o
0g
Av. 2
1
Ay.P,
Ay,
log 4
dog
2
dog
2
q
do?
We now recall Green's identity: If ((x) and (xx are twice continuously differentiable functions in RDD and V is any set bounded by a piecewise smooth, closed, oriented surface in R.D then
J. ovi = / @Vb. ds
VΦ. Vv
(43)
where Vø denotes the gradient of Φ, ds denotes the elementary area vector, and V ds is the inner product these two vectors. This identity plays the role of integration by parts in RR..
To apply Green's identity to (42 and having surface S,. Then we use Green's identity on V, and S, with Φ(y+) log p(y,) and <b(,,) = p(y+) and take the limit as r + co. In Lemma 2 the surface integral over S, is shown to vanish in the limit. Hence, by applying heat equation 12 and Green's s identity for the first term on right hide side, we obtain:
we let V, be the D-sphere of radius centered at the origin
-/ log a(y:: 0) dopp(y)) dy.
d
/ log o(y: 0) 0) dy:
(44)
do?
2
1
/ yy, log q(y:; 0) Vy.P(y)) dy:
(45)
2
1 Vgly;; Vply:)
dy:.
(46)
2 .RP
7(y:;0)
where we used integration by parts in high dimension and the vanishing boundary terms from For the second term,
Lemma 2
- -f plyi) do? log c(yr; 0) dy. = -
/ p(y)) Vozg(y1:0) 0)
dy:
(47)
g(vri 0)
-- 2/ /
1
ply:)
Ay, 0) dy:
(48)
4yy;; 0)
p(yc)
Vg(ye) dy
(49)
q(x.)
where we again applied Green's 's identity and used the vanishing of the boundary flux.
Summing the two contributions yields (41)
Lemma 4. Let (X, S, pı) be probability spaces, and let T X + Y be measurable transformation inducing probability measures (Y, T) such that
GG n4 (T (G)), VG i= 1,2.
Denote the Radon-Nikodyy derivatives of 14 with respect a common reference measure
(x()) = dv
dr
24

---

Then, the KL divergence remains invariant under the transformation T, i.e.
(KI(vil122) = DEL (pill2))
where
(KL(vill2) = (v) log 91(() dy(y). g2(y)
Proof. Let À be a reference measure on X such that μr has densities fi with respect to X, i.e.
fa(x) = djs dA
(x))
By the change of variables under T, the density functions transform as
du: d7
d1c (T)
9 (
())
da
where (v) det de
is the Jacobian determinant of T.
Substituting into the definition of KL divergence:
DRL(Fill22) = / (1) (2(2) dr(1),
92(u)
we expand:
= - (1(TT log (TT-
dy()).
fe (T (3))
Since the Jacobian terms cancel, we obtain:
J [vT log
fi(
fe(T-1(y) dr(
By the change of variables (y), we rewrite this as:
= fz) log f2(2)
f1(2) dx(z) (Ki(e)l
Thus, the KI. divergence is invariant under the transformation T, completing the proof.
A.5
Proof and Remark of Theorems
A.5.1
Proof of Theorem.
Theorem Let ~ p(x) an arbitrary distributed random vector on R.P, and let q(x; 0) be parametric model with X q(<;; 0). Define the perturbed observation
where random vector independent of X. satisfying Ej E[4e] and Cov ( I. Let po? and ds denote the densities of X under and with noise variance of, respectively. Suppose that the KL divergence DKL(Po? (90)) is finite for sufficiently small of. Then the following limit holds:
d
1 / p(x) |logp(x) Flogg($;;0) dx,
(ko(po:lloo)) do?
=-
(50)
07-
i.e.,
DKL(pppp (pxx(())((((()))
2 (p(x))l((x(0))
do?
07-
where I(-11-) denotes the Fisher divergence (equivalently, score matching objective) between p and g.
25

---

Proof. According to the definition of relative entropy (See Def. 3)
we have
p x
0))
p(x)
log
)
dx
(51)
d
d
RD
(X: 0)
= - 0p(x)
p(x)
p(X)
log(
+ p(x)
log(
dx
(52)
do
((x; 0)
do?
(x: 0)
/
0p(x)
log(
p(x)
+ p(x)
(0p(x)
/p(x)
0q(x; 0)
/g(x:0) dx.
(53)
y(>;; 0)
Invoking Lemmal 1 on 53
yields
1
/ (D"(K) logí
p(x)
(xx:; ((xx0))0(8)
DKL. (r))(( 0))
+ +x=
dx.
do?
2
g(x; 0)
y(x;
o7=01
(54)
For convenience, define:
u(x) log p(x) g(x:0) 0)
log p(x) log ($;; 0)
Recall the Green's identity:
J vV di Jap"
0h ds
/ (Vv V1u) dx. JRD
(55)
On
Setting v = p and μ = v = log g. we obtain:
J p"u dx
du ds
/ (Ve Up) dx.
(56)
ann
On
Since p(&) has finite differential Shannon entropy, the boundary integral vanishes, leaving
p" dx
JRD (Vv. Vp) dx.
(57)
RD
For the term d/p/q, apply the same identity:
f I'p 9
d'pdx= /
(pV logq V lo dx.
(58)
JRD
Since SRO px)x) = 0 under appropriate boundary conditions, it does not contribute. Similar
technique was used 5 55,8,50 59). Using v = log log 9. we compute:
Vu Voog logg.
(59)
Thus,
Vp. Ve Vp. (Vlogr log g).
(60)
Since Vp- PV logp, we substitute:
Vp Vp-Vu- Vu-VuupPooo logp (V logp V logg).
(61)
Expanding,
logp:
Vlogp= - Vlogg) q) = || logp log
(62)
Thus,
J puudx = -f BD
p (lVlog - V Vlogp- V log q) dx.
(63)
Using the result for d'p/q:
P'P dx =
- (pV logg Vlogg) dx.
(64)
1 q
20
Summing both contributions,
{ (-p (IIV log pll
V logp- V Vlogg) pl log 2 pV logp- V Vlogg) dx.
(65)
gD
Rearranging, we conclude:
1 p(x)] V Voggp(x) - vlogg(x;00) dx,
/ 2
1
dop Dalx(()))
-
(66)
07=01
which completes the proof.
26

---

Similar to the classical result in 781 the relation in Theorem I holds because both sides quantify the error induced by mismatch between the true distribution p and the prior q provided to the estimator. Naturally, when q. both sides vanish; otherwise, the derivative is strictly negative, indicating that perturbations reduce the relative entropy. This observation also yields the relative entropy version of the data processing inequality:
(KI( 2 Dkr.(ple),
(67)
where
p p= /
Here, W denotes noisy channel. This inequality asserts that the KL divergence between two distributions decreases under the action of common channel, which is consistent with the data processing argument used in [69]. where, the channel W corresponds time-reversed Brownian motion, which can be viewed as a continuous analogue of the Gaussian channel [82] Under the
assumption that p(x) v(x)= (x)), the result further implies that the diffusion process smooths
the discrepancy between inputs. Moreover, the neural network acts as channel simulator, enabling
efficient sampling by integrating its output into the neural ODE solvers (37
A.5.2 Proof of Proposition.]
1
Proposition 1 Consider the signal model (I) suppose p(x)) and q(:: 0) have continuous second- order derivatives and finite second moments. Denote by ply1) and (x) the output signals of channel at time 1 when inputs
p(x) and q(; 0) respectively. Assume x axx) = ( I). which is
independent of f. Let ply
y x) denote the channel
① for any I E [0, 1}, then for arbitrary datapoint
and small of with 0 < d <
of E| V.. logp(y; o(oğ).
([(p(x), H(((y11), (xx)) + JOSM 00 02)
JoB
(68)
Here, the denoising score matching objective is defined
1 ? V y, logplys(x) s(y;;0)1 do?,
JOSM 0 00 (-))
(69)
2 o
and syy:;)) = Vy log a(y;; 0) -n(y,, No: 0)/O, is score network estimator.
Proof. Recall the thermodynamic integration commonly used in statistical physics:
J _ f(E)dE = -f( f())
(70)
Let f(o?) = DrL (p(x))la(() 0)). where Y, = =x, + i.,.
Lo dc
f(E)dL = - f(0),
(71)
becomes equivalent to
J ot d
do? dop flo2)do3 = f(o?) - f(0),
(72)
which yields the following expression,
| o d
do? Dra (rall(v))lat: = DRL (ra. 0)) DRa (xxx)x))la(x 0)).
(73)
This occurs because the KI. divergence remains invariant under measurable transformations shown in Lemma 4 In particular, probability density transformation serves as an illustrative example. Let X p be random vector the measure space with probability density function p(x). We define
27

---

the transformation X' = T(X). Consequently, the probability density function of X', denoted as p(x'). can be determined using:
d
p(x') pxTT
T (x)
(74)
dx
Furthermore, the KL divergence satisfies the following property:
DRL (xl (xx(((x)]lx(x))) Va > 0,
(75)
which implies that the KL divergence depends solely on the relative shape of the two distributions rather than their absolute scale. Invoke the Lemma and results in [51] on (73 yields
DEL (p(x)) (Xx 0)) (pL 1 f
2 S.²
I(p(v))ll(xx
(76)
With the fact that q 0) = Txx), which is independent of 9, we have
L. !
DRL (pL( (q(x: 0)) Dx (PL
())
(77)
The mismatched entropy between data and model becomes equivalent
H(p(x), 0)) DRL (p(x))ll((;x))) + H(p(x)).
(78)
Invoking
77 on 78
yields
1 ( H(p(x), (xx; 0)) (xxx)
[elxx))lll(x::))) H(p(x)).
(79)
Recall de Bruijin's identity [12]
H(p(y:)) [
(80)
do?
Using thermodynamic integration again, we have
H(p(y:))dof = H(r(y))) - H(p(oox)): do?
(81)
which is equivalent to
H(p(oox)) = H(p(y:)) 2 1.a o
H(p(y:))do
(82)
Recall the property of differential entropy (see Theorem 8.6.4 (8.71) in 14] H(aX): H(XX) + log (a)(
(83)
and the connection with denoising score matching (see (11) in (79)
[(p(y))llgyy:: 0)) E logy s(y:: 0)] (p(y:)) E [[[ logp(y,1x)11] (84)
Finally, we have
H(p(x), q(<p(x).((;;))))) 0)) (xa H(p(x))) / [p(Vv)) do? Dog laol
1
o E
V -(y) J(plx:) E [| [oggp(y/)x))]]]lmmm do?
2
(85)
oo? E V y. log (x) D Dog Jool.
HH(pyy))x<<x) + JDSM (0; J.
(86)
Since the identity of + of holds for all f, setting the lower limit of integration such that oi implies oj 1. Consequently, Jaol 1, and thus: log lool log 0. And invoking Theorem
we finally get
H((p(x), 0)) H(p(y1), (0:0((;0
E v., logply lospoo///sl//aotttooo +(o)).
(87)
28

---

A.5.3 Proof of Theorem
Theorem 3 Let p(y+(x) = (a,x, o) denote the Gaussian channel at any time t E [0, 1]. With
the same notations and conditions Proposition]] 0 we have
log q(x: 0) S H(plyi)x), xx)) + LDSM(O?:0)
(88)
in which LDSM ; 0) is defined as
i 1 (o; 2 2
Belrs logp(y)(x) 0(y:;0)l|ddo
and H(p(y))x)) (x)) is given by
H(p(yi1x),x<x) D ((ptx)))))))))))))))))
Proof. From Proposition T we have
/ J,3 (xtx1xxp(x)]] Vy, logp(y:(x)) 0))1;0)12
- p [log a(x; 0)] Ep(rr blo [x)]
o
1
B Bp(xs(x)p(x)) Vy, logr(y;(x)]
(89)
2
1 of
plyxl)))(()))))))))))))))))))))) logply, 2 dxd
2 Jo
1 of
+ 2
p(yx)x))(x))) log ply s(yc: dxx (90)
03
Given a fixed channel ply: x)), we can easily see that Se p(x)dx in both sides of 90) can be canceled to get
logg ((x;0) | p(g))x))log((xdd)) 1 of 1e / p(yx(x))\Vy, log loy 200
2
o / r
+
JEB p(yl|x)//V,, logp(y.(x) s(y,, dy
(1)
The second term in (9T) corresponds to one-half of the integrated Fisher information of the Gaussian distribution, which strictly non-negative [19), we obtain:
of ( [
log qxx; 0) S H(p(yilx), (x))
p(yx|x)) logp(y.(x) s(y,, d)l
(92)
which finishes the proof.
A.5.4 Proof of Proposition] 2
Proposition 2 Ler X + where X, R are independent random vectors, with v satisfying E o = 0 and Cov ( = I. Assume the probability density function p(x) of X is twice continuously differentiable and decays sufficiently fast at infinity, and that the Fisher information T(X) exists and is finite. Then,
d do?
H (p(())
J
1
(93)
107-0-
2
where H((-) denotes the differential Shannon entropy and J(.) denotes the Fisher information.
29

---

Proof. By the smoothing properties established in Lemma I the distribution p(x) is differentiable with respect to of. Since the integrand in equation (T3) is both continuous and differentiable of we may interchange the order of differentiation and integration to obtain:
H(p(x)) = J d
d
d
RD do? x(x) (1 + log)(x)) dx
(94)
(A)p(x)) log p(x) dx.
(95)
RD
Following the same argument as in Theorem ! this establishes the desired expression, which recovers the special case of isotropic additive noise discussed in [60].
B Improve the Likelihood Estimation Bounds
Our theoretical analysis suggests that different variance functions can lead to varying performance in likelihood estimation. To validate this empirically, we evaluate the effect of different noise schedules, variance functions and datasets on likelihood estimation performance.
However, in practice, exact numerical evaluation of the integral is generally intractable. A common approach to use Monte Carlo method to estimate the variance integral in (o? 0) during both training and evaluation. Moreover, reducing the variance of the Monte Carlo estimator for the continuous loss objective generally improves the efficiency of optimization. We next summarise our empirical observations on likelihood estimation across different settings.
B.1 Variance-Aware Likelihood Bounds
denoising score matching [79 is typically used as the training objective for diffusion models, serving as a surrogate for approximating the likelihood function q(x: 0). Specifically, recall that out channel (T) is defined as p(y,(x)
As mentioned in Section 3.3
such that Vy, loy xx) = -n/o,. In the sense that with the score model (y:; 0) == -n(y,, M1; f) oo the objective takes the following integral form, where the loss is evaluated under varying noise levels:
/N(axx,
1 noi = 03 Ep(xix)) 2 n n(y,, m1 0) do?
/
LDSM
(osM((?? (o?: 0)
(96)
We begin by approximating the expectation (a? 0) via Monte Carlo estimation. This involves drawing samples y, p(y:|x) from a tractable Gaussian kernel, which enables efficient estimation of the denoising objective. To estimate the integral over the variance schedule, we apply second Monte Carlo approximation by uniformly sampling of from the interval [o]. o?)- However, we empirically observe that such an estimator can significantly degrade the optimization process, potentially due to increased variance poor convergence behaviour. To address this, we reparameterize the objective terms of a smoother coordinate, namely, the negative log signal-to-noise ratio (log-SNR), which facilitates both numerical stability and analytical tractability.
Even under constrained noise schedules, such the Variance Preserving (VP) formulation 28 which enforces of + o = 1, there remains freedom in how a;, a, evolve over time t. To abstract away from specific parameterizations such VP or VE schedules, we adopt a log-SNR parameterization as the default: 7)t log SNR (1) = log a. This reparameterization simplifies the analysis and unifies various diffusion formulations under a single coordinate system. Under this reparameterization, the change of variable from time t to negative log- JoR-SNR// follows:
dp do? 1 da
(97)
dt
of dt
of dt
For the VP schedule, this simplifies to:
dn of dt
do
(98)
dt
By setting the noisy process to follow an EDM-like variance-exploding (VE) schedule 37.. we let 2 ar = and parameterize the variance as of = exp(n). This configuration is equivalent to that
30

---

of NCSNx2 (70] when follows geometric sequence interpolating between 0.01 and 50, i.e., ntt) 2 (0.001 l0g(500)) t. Under this specification, the objective (96) reduces to the
continuous-time loss proposed in 40
- (t)
(x) 1
1
(99)
where (t) dnp/dt dt acts as a natural weighting function. In the above case, this means that (t) 2og (5000] and thus that weighted function constant. We leave the VE schedule optimization for future works.
Similarly, in the cooommtioon-thorette VE formulation 43 i.e. By v7 x I. our bound recovers the loss integrates over the signal-to-noise ratio y:
SNR
C :=
Eple,
|||
8o 7) d7,
(100)
SNRaie
where instead of integrating w.r.t. time f, now integrate w.r.t. the signal-to- to-se ratio y, and where SNR max = (1), SNR min
-(0).
In more general form, the objective in (96) recovers the weighted continuous-time formulation of (66) [4],, where the weighting function is defined as wtt) := af; equivalently, this corresponds scaling the signal coefficient by a across time:
2 (x)
: Fn-- (1)a || no (y:: 1)))
(101)
2
Moreover, (96) can be further unified under the stochastic differential equation (SDE) framework [72]
dx f(x, dt + g(t) dw,
(102)
where f(x,t) : + RD, g(t) E R are manually designed noise schedules and w E R is a standard Wiener process. When the variance function satisfies do? dt gt),,, the corresponding loss aligns with the likelihood-weighted score-matching objective 669
(0) 1'
T (t) 20
E Exxp(x), ((.1( (1,,, 0) + n| dt.
(103)
The same principle extends to deterministic flow-based formulations, including continuous-time ODEs and higher-order probability flows (50) 69 90 with specific design of the gt)). In this case, the training objective becomes:
Tshe = 2/2 2 1 m
wo.. 0) + nl dn.
(104)
This reformulation highlights that the loss landscape is intimately linked the choice of noise schedule, particularly the functional form of o² as a function of log-SNR Since both the weighting in the integrand and the resulting likelihood bounds depend explicitly of, different scheduling strategies can lead to substantial differences in training dynamics, gradient variance, and overall model performance. In the following, we provide both theoretical and empirical analyses to quantify how the design of of influences estimation accuracy, stability, and convergence behaviour.
B.2 The Noise Schedule Matters
Many practical objectives, including the ones above, are special cases of the likelihood loss 39 40 69). Let = (t) be a differentiable, monotone noise parameterization of time E [o, with fixed endpoints Mmin 9(0) and Mmax g(1). For a per-example loss, define
Lu (x) = 2 / Tmmax
w(n) En> [||n niy m 0) || dn.
(105)
where y, denotes the noisy input at noise level r (e.g., via a log-SNR parameterization). in 40). the integral in (105) is invariant to smooth reparameterizations of (i.e., to the choice of schedule) so long as the endpoints Tmin 7max are kept fixed. This invariance extends to the weighted diffusion loss family [39 because the integrand is measured per unit 7).
shown
31

---

However, this invariance does nor earry over to the Monte Carlo (MC) estimator used in evaluation. we sample 2( 1) and n (O I). then by change of variables,
1 (x) (o., m-- (n(())
(t) I
((t) 0)
(106)
2
More generally, if we sample directly p(n)) over Mmiin Mmax] (with a.c.), then
(())
(x) Bap-p.
n i(y, 27
(107)
p(n)
revealing that the noise schedule induces an importance-sumplinn distribution p(n) over r and corresponding weighting function wx11) related to of simultaneously. Consequently, while the population loss is invariant to reparameterization, the variance of the MC estimator (and of its gradients) depends p. Choosing to better match the integrand substantially reduce estimator variance and thus accelerate optimization. In particular, the variance-minimiziin density satisfies the heuristic proportionality p² (n) x Bx-XX.=-V00.) (())) I n 17: 0)112
B.3 Variance Reduction with Importance Sampling
Monte Carlo approximation offers a computationally efficient alternative to exact integration, but typically introduces variance into the training objective. To mitigate this, we adopt both designed and learned importance sampling (IS) strategies, as outlined in Section 3.4 As mentioned in Section. [B22] the diffusion model conducted for all r in [Too through an integral. In practice, the evaluation of the integral is time-consuming, and Monte-Carlo methods are used to unbiasedly estimate the objective by uniformly sampling v). Thus, continuous importance distribution p(1)) can be proposed for variance reduction. Denote
L(x, n, n7 1; a n(y,, m7: 0)1
2
then
L(x, n, 17; 0)
(x; 0) Eny-p(n) Ex.n
(108)
p(1))
We propose use two types of importance sampling (IS), and empirically compare them for faster convergence.
B.3.1 Designed IS
In particular, propose continuous proposal distributions p(1) over n, which is proportional a Since we have explicit expressions for the density, we utilize inverse transform sampling to design sampling procedure. Concretely, take uniform samples of a number / [o, 1]. and solve the following equation about ?/:
7 a dn,
dy dn t, Z
=
(109)
Z
where we have defined maximum time and is a normalizing constant.
Uniform Weighting in S-Scce As a canonical example, we consider uniform weighting in the log-SNR domain. In this case, the proposal distribution becomes p(1) o yielding:
1 te 1 e
Z
sigmoid(- ( dnj log
Also, for squashed hyperbolic tangent, we have:
m
1 1 220
2
=
sigmoid(-2n) dn = log
1 e e-2m
no
To enable inverse transform sampling, we define the antiderivative g(n) == log(1 + e-"), and set
la g(7o), l (()). The corresponding cumulative distribution function (CDF) is then:
lo-((1)
F(n)
17 [momm]]
Z
32

---

Sampling is performed by drawing u ~ U(0, 1) and solving for 7 such that F(n) = Since F(n) smooth and strictly monotonic, this inverse can be computed via root-finding or precomputed lookup table. The method is similar to the implementation in 69 90
Uniform weighting in I-space. As discussed in Section
B. drawing uniformly from the interval
[0, amounts using the proposal density p(n) = 1/00 "(r)* With this baseline choice the objective
in Eq. (99) can be optimized in its native form: each Monte-Carlo sample is simply re-weighted by the factor w(n) = of (t), which bundles the forward-process weight až and the Jacobian n (t). Although the resulting estimator exhibits higher variance than the importance-sant saapling schemes introduced earlier, it provides a clear reference point for measuring the effectiveness of less sophisticated designs.
B.3.2 Learned IS
The variance of the Monte-Carlo estimator depends on the learned network (): To minimize the variance, we can parameterize the IS with another network and treat the variance as an objective 40 90). Actually, learning p(n) is equivalent to learning a monotone mapping. Thus, we can uniformly sample i, and regard the IS as -hange-of-variable from 17 to
Lw 1b, 0) B Ex.n (v/ (b) ab) nxx,,,;;))
(110)
where (1(t: (b) dn((: /dt and ry(:: ab) is monotonic neural network with parameters bb. Since the variance Varx. (t: 3 f(x, n, 17,0) = Ex (b) [(x, n, 07;
(x; 01)2, where (x; (b 0) is proved [40] invariant to 17(t; v)), we can minimize Ex,n.c(17/ (t; v/) [(x, n, 1;; for variance reduction. At this point, we parameterize ry(t; vb similar to [40]. where the network consists of 3 linear layers with weights that are restricted to be positive 1,, 12, 1z. which are composed as r)(:; $/) = 4 (t) + la (ssigmoid((2((1(())) The l2 layer has 1024 outputs, where and la have single output.
We therefore postprocess the monotonic neural network as
(t; v)) n)(0; ab)
(t: w/ = ijo + (m - 77) 7(1 (b) - r7(0; 4b)
(1I1)
= - log(SNRMin define the target SNR range. This construction ensures that (t; +)) remains monotonic and bounded. By adjusting the network parameters &6 we adapt the time-to- to-soise mapping t + ntt) to match regions with higher loss variance, thereby allocating more samples to informative regions of the diffusion trajectory.
where the constants = - (Sg(SNNMAX).
Overhead. While this approach seeks the optimal IS, learning n) adds lightweight auxiliary network training steps and complex gradient operation through r/(t; v)), but no extra score-network passes. Hence, use it mainly to benchmark the optimality of hand-designed IS; more aggressive adaptive schemes following 339 90] are compatible.
B.4 Log-SNR-Timee Channel with Different Variance Functions
Given the central role of of shaping both the weighting of the loss and the structure of the forward process, it is natural to ask how different functional forms of on affect the overall behaviour of the model. In particular, we investigate how alternative variance functions under the log-SNR parameterization influence the resulting likelihood bound, optimization dynamics, and numerical stability. This section provides both analytical insights and empirical comparisons across several representative schedules. As state in Section B. using 7 timing and denoising score matching estimator, the variance weighted objectives are reformulated as:
ioî En l0,
LosM(o?;
2||n -nny,, mo: ():0): 0)12
do?
(112)
Jo
Due to the timt schedule properties, we replace the time subscript with 7, an and on are deter- ministic functions of n without any hyperparameters.
33

---

Table Specification of related values and objectives under VP schedule. With s(n) sigmoid(
3_ sigmoid( -17), sa/2
Formula
Generalized Logistic Sigmoid
Squashed Hyperbolic tangent
tanh(n/2)
on
+ exp(21)
1
On
k(n)
exp(-2n)
mm1 a
Jmm. no
m s (2n) En [|| - i 77; 0)163 dn
CDSM
s (1) En n ni my 0)1 dn
2 no
Table 6: Specification of related values and objectives under SP schedule. With ((1) = sigmoid(n) sigmoid(-n).
Formula
Generalized Logistic Sigmoid
Squashed Hyperbolic tangent
On
1
(n)
eeepp 2n)
On
si)
1eexp(-24 2n)
M a 2
m s (2n) En [|| niy., 27; 0)1 dn
CDSM
(1) E, n(y m. 1))
no
Logistic Sigmoid. We begin with the benchmark variance function of = sigmoid(1) == T which is widely adopted in VP-based diffusion models [40]. This schedule maps variance smoothly into the unit interval, with most of the variation concentrated around of (90). Its symmetry and boundedness make it a natural default choice, but also raise questions about its flexibility in capturing tail behaviour and class imbalance. We have the analytic form of the derivative of d = (1 d
91 DDSM (0;:0) 2 En (1 7 n 7:: 0)1 dn.
(113)
Generalized Logistic Sigmoid. Then we consider the type generalized logistic sigmoid function
(36] of = sigmoid (1) = (146-1) )r with is strict positive coefficient. We have do = a(1
(f)
CDSM(O (0:
En [iigmoid( (-n) n c. mhi 0)1| dn,
(114)
Squashed Hyperbolic tangent. Then we consider the Tanh squash function of 1 (tanh(n) + 1), we have the analytic form of the derivative do = 2(1 o
CDSM (o?:
En (1 9 n n Me
dnj
(115)
Thus, derive their specific objectives and equivalent predictors using the formula for general noise schedules. We summarize them in Table and Table
B.5
Empirical Results
Tables 8 and report various configurations of the noise variance function and its endpoints in the VP setting. Empirically, increasing the noise variance at the initial stage of the diffusion process yields consistently improved likelihood bounds.
34

---

Table 7: Comparison of NLLs and FID on CIFAR-10 under different noise variance settings.
(a) NLL bounds with different noise variances
(h) FTD of CIFAR-10 10 with different losses
Likelihood Bounds
Yl
8.77
Tl
13.3
Loss
17u
8.7
170
-13.3
ELBO
3.99
2.75
ELBO
14.60
11.9
Ours (SP with IS)
2.49
2.79
Ours (VP)
10.18
9,42
Ours (VP with IS)
2.50
2.78
Table 8: Performance comparison across generalized sigmoid schedules with varying exponent initial log-SNR foo and importance sampling strategies on CIFAR-10. 10. Bold denotes the best NLL and FID within each group.
Schedule a
7/0
Sampling Strategy
NLL (4)
FID (L)
a
-13.3
Uniform in
2.81
9.79
IS: Learned Importance
2.75
9.12
IS: Uniform in
2.78
9.42
a a=1
8.77
Uniform in t
2.62
14.9
IS: Learned Importance
2.50
10.05
IS: Uniform in !)
2.50
10.18
a a=
-13.3
Uniform in t
3.62
IS: Learned Importance
3.47
IS: Uniform in 7j
3.51
--2
-8.7
Uniform in &
3.16
IS: Learned Importance
3.05
IS: Uniform in
3.04
a 0.5
-13.3
Uniform in t
2.54
IS: Learned Importance
2.45
IS: Uniform in r7
2.47
0.5
-8.7
Uniform in t
2.61
IS: Learned Importance
2.31
IS: Uniform r)
2.32
We begin with the benchmark configuration adopted in [4), where of (1)(t) and the initial negative log-SNR value set to = =-13.3 13.3. Under this setting, the model achieves an NLL of 2.81 bits/dim after 300K iterations, which is comparable the ELBO result of 2.75 reported on CIFAR-10 with the same number iterations.
0.5, i.e., of sigmoid(7) the results remain promising. However, for Yo = -877, we note that sigmoid(-8.7) 1.7 1.29 10-2, which falls outside the regime where the assumption of < holds strictly, thus violating the thcoretical conditions underlying our analysis. Nevertheless, this configuration
When using
be regarded practical compromise, yielding strong empirical performance despite the theoretical violation.
We further observe that increasing the warm-up noise variance generally leads to degraded FID scores. This motivates more systematic investigation of such extreme configurations, as summarized in Tables I0 and II Empirically, the likelihood performance appears insensitive to the detailed shape of the variance schedule, but depends primarily on the endpoint values of and of. These findings suggest that careful tuning of the endpoint variances can have larger effect than modifying the overall schedule shape, particularly for likelihood-oriente objectives, an observation consistent with
prior analyses in [28 38 40).
35

---

Table 9: Performance comparison across squashed Hyperbolic tangent schedules with log-SNR endpoints To. and importance sampling strategies on CIFAR-10 10 with VP. Bold denotes the best NLL within each group.
700
711
Sampling Strategy
NLL (L)
13.3
Uniform in
3.83
IS: Learned Importance
3.77
IS: Uniform in
3.79
-8.7
Uniform in
3.35
IS: Learned Importance
3.15
IS: Uniform in ?
3.18
-4.33
2.5
Uniform in
2.61
IS: Learned Importance
2.51
IS: Uniform r
2.53
Table 10: Performance comparison across generalized sigmoid schedules with varying exponent a, initial INR 1/o, and importance sampling strategies on CIFAR-10 with SP. Bold denotes the best NLL and FID within each group.
70
Sampling Strategy
NLL (L)
a
13.3
Uniform in
2.82
IS: Learned Importance
2.77
IS: Uniform in 77
2.79
a=1
-8.7
Uniform in
2.62
IS: Learned Importance
2.49
IS: Uniform in 7
2.50
0.5
-13.3
Uniform in t
2.53
Learned Importance
2.47
IS: Uniform in 7
2.45
0.5
-8.7
Uniform in
2.61
IS: Learned Importance
2.31
IS: Uniform in
2.33
B.6 Likelihood Estimation Comparison for IT-bound and ELBO CIFAR-10
Tables 12 and report the average loss components of models trained on CIFAR-10. We observe that all IT-based models employing standard sigmoid variance schedule achieve likelihood estimates that match or surpass those of the ELBO baseline. Notably, when combined with importance sampling, the IT bound consistently outperforms the ELBO across all configurations. However, when using uniform sampling in t-space, the ITT-bound appears to trade off small amount of likelihood for faster convergence, likely due to reduced variance gradient estimation. We further observe that increasing the warm-up up noise variance leads to notable rise in the total ELBO. This increase is largely driven by the reconstruction term,
Exxoo [log q( yo)),
(116)
which becomes more pronounced p(yo/x) flattens under higher noise levels, making accurate reconstruction more challenging.
Intuitively, the loss structure is grounded in the auto-encoder paradigm 41 where one commonly assumes factorised posterior y()) implying conditional independence across the elements of &. In the context of image data, this assumption translates into the belief that pixel values are
36

---

Table 11: Performance comparison across squashed Hyperbolic tangent schedules with log-SNR endpoints fo- and importance sampling strategies on CIFAR-10 10 with SP. Bold denotes the best NLL within each group.
700
71
Sampling Strategy
NLL (L)
13.3
Uniform in
3.82
IS: Learned Importance
3.76
IS: Uniform in
3.73
-8.7
Uniform in
3.32
IS: Learned Importance
3.14
IS: Uniform in ?
3.18
-4.33
2.5
Uniform in
2.60
IS: Learned Importance
2.51
IS: Uniform r
2.51
Table 12: Decomposition of ELBO and Ceemmatioo-Thhoreti Bound (TT) with initial endpoint Yo
13.3. Values reported on CIFAR-10 10 in 310K iterations (1 lower is better).
Term
Description
Value (bits/dim)
Evidence Lower Bound (ELBO)
Total = 2.794
Term
KI. divergence between p(y: (x) and prior (x)
0.0012358
Term
Reconstruction loss (-Epíyo x)) (xx(x)))
0.0103869
Term 3
Diffusion loss ( Extros DEL. (()) (0 0))
2.7836967
Information-Theoree Bound (IT))
Total 2.805
Term Term 2
Integrated denoising score matching term
0.7622061
Mismatched entropy between p(y, xx) and prior ग (x))
2.0434874
conditionally independent given the latent code yo. and each pixel depends only on its corresponding latent component. follows the form:
(Bo.)) x p(u0,<1z;) = N (yo,s; an2,, oi);
(117)
where we normalize over all possible values of z;. However, we argue that this assumption overly restrictive and does not hold in practice. Within the auto-encoding framework, q q(xly)) can be interpreted as a decoder tasked with reconstructing the data. The approximation may hold when the SNR at 0 is sufficiently high. In this case, the conditional distribution p(yu x) becomes sharply peaked around yo = agx, effectively imposing strong constraints on the reconstruction loss. Specifically, it induces high sensitivity to noise: small deviations in the warm-up noise level can significantly affect the fidelity of reconstructions. Moreover, since likelihood estimation is inherently sensitive the probability of individual pixel values and fine-grained image details, this modelling choice represents a practical compromise rather than principled solution. In effect, the assumption simplifies computation at the expense of capturing complex dependencies among pixels, which can be critical for accurate reconstruction and reliable likelihood evaluation.
Increasing the warm-up noise smooths the sharp concentration of p(yalx), easing reconstruction by reducing the signal-to-noise ratio. the VP setting, this implies ag deviates more from 1, yielding more stable behaviour. Crucially, Theorem [ ensures robustness under arbitrary noise, allowing principled tuning of the starting noise to balance stability and accuracy. See Appendix. G for more related diseussions.
Furthermore, we observe that small perturbations may fail regularise extreme pixel values in- troduced during dequantisation, preventing effective mapping into smooth continuous domain.
37

---

Table 13: Decomposition of ELBO and Information- T Bound (IT) with initial endpoint
Zo -8.7. Values reported on CIFAR- 10 in 310K iterations (7 lower is better).
Term
Description
Value (bits/dim)
Evidence Lower Bound (ELBO)
Total = 3.99
Term 1
KL divergence between p(y [x) and prior (x)
0.0012358
Term
Reconstruction loss (-Epívo x)) [log (x)(v)))
2.7219771
Term 3
Diffusion loss ( (p( Dx. (py (t() (1))) 0))
1.2691765
Tetormation-Theorrttee Bound (IT)
Total 2.51
Term
Integrated denoising score matching term
0.4663643
Term 2
Mismatched entropy between p(y1) xx) and prior πt (xx)
2.0435017
Table 14: Likelihood in bits per dimension (BPD) and sample quality (FID scores) on CIFAR-10 and ImageNet-32, for vanilla VDM, MuLAN and ours, indicates the result from 62 for 10K samples generated using an adaptive-step ODE solver.
Model
CIFAR-10
ImageNet-32
Steps
VLB (L)
FID (4)
Steps
VLB (1)
FID (L)
VDM 40
10M
2.65
7.6
2M
3.72
14.26
+ MuLAN
2.65
18.54
2M
3.71
13.19
Ours (Nj 13.3)
0.3M
2.78
9.42
0.3M
3.28
13.80
Ours (N)o = -8.7)
0.3M
2.50
10.18
0.3M
3.01
14.76
Slightly increasing the initial noise helps suppress such outliers, resulting improved likelihood behaviour and more stable training. Moreover, in score- based models with noise prediction, a very small starting noise can impair residual estimation near degrading likelihood due to poor signal-noise separation. We conjecture that adopting a velocity-based parameterisation [55 90
(v-network) may alleviate this issue.
C Samples Quality and FID
Just as selecting appropriate training and optimization strategies is necessary to achieve strong performance in given application, conclusions. We must stress that the primary focus of this paper is on maximizing the likelihood learning metric, specifically, negative log- likelihood measured in bits- bits-per-dimension (BPD; lower better), rather than on optimizing Fréchet Inception Distance (FID) or Inception Score (IS), for the following reasons.
the choice of evaluation metric pivotal for drawing valid
Model samples undoubtedly serve as a valuable diagnostic tool, often enabling us to form an intuition about why model may underperform and how it might be improved. From this standpoint, generative model ought to produce samples that are indistinguishable from those in the training set, whilst encompassing its full variability. To quantify these properties, a variety of metrics, such as the IS and the FID, have been proposed. However, both qualitative and quantitative assessments based on model samples can be misleading with respect to a model's density-estimane well as its effectiveness in probabilistic modelling tasks beyond image synthesis (74). Consequently, average log- likeeliodood remains the de facto standard for quantifying generative image- modeling performance. For many sophisticated models, the average log- likeeiiolooo is challenging to compute or even approximate. Indeed, it is possible for model with sub-optimal log-likelihood to generate visually impressive samples, or conversely, for model with excellent log-likelihood to produce poor samples, an observation that underlines the lack of a direct relationship between FID and negative log-likelihood (NLL).
capabilities, as
38

---

Table 15: Comparison of the mean FID scores with standard error for our model on CIFAR- 10 with sigmoid noise schedule after 0.3M steps. We provide both FID scores on 10K and 50K samples and with respect to both train and test set.
Model
FID 10K train
FID 10K test
FID 50K train
FID 50K test
Ours (7oo = -8.7)
11.85 + 0.2
12.91 + 0.2
10.18 + 0.2
10.90 + 0.2
Ours (7jo = 13.3)
10.16 + 0.2
11.53 + 0.3
9.41 + 0.2
9.50 + 0.2
DiffEnn [55]
14.6 + 0.8
18.5 + 0.7
11.1 0.8
15.0 + 0.7
Table 16: Comparison of the mean FID scores with standard error for our model on ImageNet-32 with sigmoid noise schedule after 0.3M steps. We provide both FID scores on 10K and 50K samples and with respect to both train and test set.
Model
FID 10K train
FID 10K test
FID 50K train
FID 50K test
Ours (1j0 -8.7)
16.90 + 0.2 15.76 + 0.2
17.91 + 0.2 16.15 0.3
14.72 + 0.2
14.76 + 0.2 13.80 + 0.2
Ours (77 -13.3)
13.21 + 0.2
From an information-theoreee standpoint, it is well known that maximizing the log-likelihood of a probabilistic model is equivalent to minimizing the KL divergence from the data distribution to the model distribution. By contrast, FID operates by fitting multivariate Gaussian distributions to the embeddings of real and generated images and then measuring their discrepancy via the Fréchet distance (equivalently, the 2- Wasserstein or Earth Mover's ss distance). Clearly, the mathematical formulations of these two metrics diverge fundamentally: one corresponds to a mismatched estimation problem under a KL-based criterion, while the other embodies an optimal-transport task.
Moreover, FID conflates both fidelity the real data distribution and the diversity of generated samples into a single score, and its absolute value is highly sensitive to myriad factors, ranging from the number of samples and the particular checkpoint of the feature extractor network to low-level image-processing choices. Consequently, the visual appeal of generated images, as quantified by FID, correlates only imperfectly with a model's log-likelihood performance. In our work, we concentrate on advancing the state of the art in likelihood estimation; although we report FID scores for completeness, we leave the optimization of sample quality to future research.
Although our model was not explicitly optimized for perceptual sample quality, we report FID scores, standard metric for visual realism, for both our model and VDM on CIFAR-10 and ImageNet-32 (Tables 14| [5) T6]] From these results, we observe that both models achieve comparable FID) scores across datasets. Importantly, the reported values vary substantially depending the number of generated samples and whether FID is computed against the training or test set. As expected, using more samples improves FID, and evaluation against the training set consistently yields better scores, likely due to closer distributional alignment.
Furthermore, among models with similar likelihood performance, such as i-DODE 190), MuLAN [62] and DiffEnc (55), our method not only achieves the best negative log-likelihood but also retains the lowest FID despite requiring substantially fewer training iterations. Conversely, methods like W-PCDM 46 that explicitly optimize for FID exhibit a marked reduction in likelihood performance.
we find that models achieving better log-likelihood often exhibit slightly worse FID scores. Nevertheless, we emphasize that this degradation in FID is minor, and qualitatively, the generated samples from both models are visually indistinguishable (see Figs. and 5).
Consistent with prior observations 69
28 54
D Experimental Settings
Datasets We perform all experiments on CIFAR-10 10 and ImageNet datasets. CIFAR-10 contains 50,000 training and 10,000 test images. The ImageNet variant includes 1, 1,281,149 training and 49,999 test images. Among the two known versions of ImageNet32, we adopt the newer, anti-aliased version 44 which facilitates likelihood training and remains publicly available. The older version used in 40 no longer accessible. Furthermore, it is notable that ImageNet contains some
personal sensitive information and may cause privacy concern (69].
39

---

Model Architectures Our model architecture closely follows the design of Variational Diffusion Models (VDMs) [4]]. Specifically, we adopt the original U-Net backbone from VDM for pixel- space diffusion without modification. Our diffusion model is parameterized in terms of the 77-tmmdd normalized noise predictor. This architecture optimized for likelihood-base training and includes key design choices such as the removal of internal downsampling and upsampling, and the use of Fourier feature embeddings to improve fine-scale detail prediction. Consistent with VDM's dataset- dependent configurations, we use a U-Net of depth 32 with 128 channels for CIFAR-10 10, and 256 channels for ImageNet-32. Our model for ImageNet-64 and -128 uses double the depth at 64 ResNet layers in both the forward and backward direction in the U-Net. It also uses constant number of channels of 256. All models apply a dropout rate of 0.1 in intermediate layers.
Hardware For the ImageNet-64 and -128 experiments, we used a single GPU node with A800s or 8 H20-NVLink. For the CIFAR-I0 and ImageNet-32 experiments, the models were trained and evaluated on GPUs spanning several GPUs types like V100, L20ss, A40s, and 3090s with float32 precision.
Training We follow the same default training settings as [40]. For all our experiments, we use the Adam optimizer with learning rate 2 x 10 exponential decay rates of $1 0.9, 82 0.99 and decoupled weight decay coefficient of 0.01. We also maintain an exponential moving average (EMA) of model parameters with an EMA rate of 0.9999 for evaluation.
For CIFAR-10, 10, the training processes are conducted on a cluster of 4 GPU cards NVIDIA V100s. We pretrain the model for 0.3 million iterations using batch size of 128, which takes around 38 hours. Then we finetune the model for 1K iterations using batch size of 256 and accumulate the gradient for every 4 batches. Note that in related works [49]. experiments on ImageNet-32 (new version) are conducted at a larger batch size (512 or 1024), which may improve the results. For ImageNet-32, the training processes are conducted on 4 GPU cards of NVIDIA A800 (80GB). We pretrain the model for 0.3 million iterations using a batch size of 512, which takes around days. Then finetune the model for 1K iterations using a batch size of 1024 and accumulate the gradient for every batches.
FID We report Fréchet Inception Distance (FID) scores computed on 50,000 generated samples, unless otherwise noted. This follows the standard setup used in [40], with ancestral sampling over 1,000 sampling timesteps. FID is evaluated against both the training and test sets for CIFAR-10 and ImageNet-32. While increasing the warm-up up noise level may slightly increase FID scores, we find that the visual quality of generated samples remains comparable (see Figs. and
E Consistency Across Predictors and Corresponding Objectives
As discussed
40 the diffusion model can interpreted from three distinct perspectives: as a
denoising process, a noise prediction model, and a score-based model. Similarly, our model admits four equivalent parameterizations, with the velocity-based vy-prediction 63 90] included in addition
to the three canonical forms, which is summarized in Table 17
Remark Let V, = ogn - o;X and define the instantaneous velocity as V = ú, - n, We consider
four types of predictors, each parameterized by 9, along with their corresponding matching objectives. Each loss is weighted by a positive time-dependent function w()):
Score predictor s(yr; 0) with likelihood-weighted score matching loss
69
72
Jsm(w(()) 0) E, [w](t) Ex... V logp(y:) A(y::
Noise predictor n 0) with standard noise-matching loss 40
28
(w)) 0) == E, [w(t) Ex.n [||n ñ (y::
Data predictor x(yr;; 0) with reconstruction- based data-matching loss
68
(w)(: 0) := E, [w](t) Ex.n [[|x - x 0)]
40

---

Table 17: Analytical relationships between optimal predictors under the Gaussian forward process.
Predictor Type
Symbol
Optimal Expression
Expressed via
Score
8* (y:)
Vy, log p(ye)
Score matching
Noise
i (yr)
-, (y))
Noise prediction
Y: v(n"(y:)
Data
xx (y:)
Denoising reconstruction
Velocity
(y:)
f(t)y:
1 (t)((t)((ye.
Flow parameterisation
uiot
Velocity (alt.)
(y,)
8,
of (vr)
Score-based ODE
at
Velocity predictor v(y;; 0) with flow-matching loss
90
(t)(w(t): == E, (w(t) Ex. [[| (y: 0]]
Under the Gaussian forward process, the optimal solutions to these objectives are analytically related, and yield equivalent predictors when appropriately reparamcterized. Specifically, they equivalent by the following relations:
n' (y;; 0) -0ss" (y:: 0) o2 V.y. logp(y:)
(118)
(y: Yt O, (y::0)
(119)
a
a + of
(y,, 0)
n (y,;
- y:
(120)
at
at
$ (y:; 0) = dy
dioi
+
n (v;;;;;)
(121)
at
0t
The predictor vi (y:, 0) defined as linear combination noise and data components, represents a static velocity target in the latent space. In contrast, the instantancous flow v (yr; 0) = dy./ /dt arises from differentiating the forward process with respect time. The two are related via the temporal dynamics of au and and coincide when the process is linear and velocity is time-invariant.
F Numerical Stability
Finite-precision arithmetic fragile for terms of the form 1 6. our discrete-time objective, several intermediates extremely close to one (e.g., cumulative coefficients and survival factors). With naïve float32 implementation, these values can round to exactly 1, corrupting the computation and yielding incorrect losses/ lusses/gradients. Prior discrete-time time diffusion implementations (28) used float64 to sidestep such issues. In contrast, our formulation is numerically stable enough that float64 is unnecessary; standard float32 suffices.
numerically problematic term, for example, is the sampling variance o. It is straightforward to verify that
of expmm(softplus((((3) softplus(7(t))).
(122)
where (x) = ex and softplus(z) = log(1 + e²) are numerically stable primitives in common numerical computing packages. Evaluating of via (22) avoids catastrophic cancellation near and keeps computations stable in floar32.
G Randomized Distribution Smoothing and Dequantization
Modern generative models often lean the manifold hypothesis 3 52
real-world high-dimensional
data concentrate near low-dimensional manifold. When the hypothesis holds exactly, the data distribution is singular with respect to the ambient Lebesgue measure and its density not well
41

---

Table 18: Comparison of the NLL for ELBO and our bound on CIFAR-10 in training and testing with 0.3 million iterations.
Model
train
test
ELBO
2.75 + 0.002
2.79 + 0.002
Ours (7o 13.3)
2.79 + 0.002
2.80 + 0.003
Ours (7j = 8.7)
2.49 + 0.002
2.50 + 0.003
defined. When it holds approximately, only points in a thin neighborhood of the manifold carry non-negligible mass; elsewhere the density is near zero. Consequently, any ambient-space density that tries to fit such data must exhibit sharp transitions (large first-order derivatives, i.e., a large, possibly unbounded, Lipschitz constant), which is notoriously challenging for likelihood- models.
Furthermore, while natural images are typically stored using 8-bit integers, they are often modeled using densities, i.e., an image is treated as an instance of continuous random variable. Since the discrete data distribution has differential entropy of negative infinity, this can lead to arbitrary high likelihoods even test data. To avoid this case, is becoming best practice add real-valued noise to the integer pixel values to dequantize the data.
To this end, propose to address such issues in the density estimation problem via warm- start process. Inspired by the recent success of randomized smoothing techniques in adversarial defense and distribution smoothing [1 52 generative modeling.
propose to apply randomized smoothing to diffusion
Randomized Distribution Smoothing Unlike [[]] where randomized smoothing is applied to a model, and [52) where symmetric random noise is applied to the data distribution, we inject the arbitrary randomized smoothing into both data p(x) and model ( 0). Specifically,
convolve an arbitrary isotropic noise distribution with the data distribution and model to obtain the new "smoother" distributions. By choosing an appropriate smoothing distribution, easier than the original learning problem: smoothing facilitates learning in the first stage by making the input distribution fully supported without sharp transitions in the density function; generating sample given a noisy one is easier than generating a sample from scratch.
aim to make warm start process
Dequantization Mismatch Another representative issue in dequantization 669 method with diffu- sion models is training-evaluation mismatch. During training, each datapoint is treated as a narrow Gaussian (or logit-normal) [40 (55] centred on the original value, while evaluation typically occurs data perturbed with uniform noise (74). This inconsistency introduces a distributional shift between training and test likelihood evaluation. Variational dequantization (27). conditional autoregressive model [52) and soft truncation (8 alleviate this issue by learning the dequantization noise and finding the optimal e, but incur substantial computational cost and convergence instability.
Remarks Our Theorem and Prppositionnoffeee a theoretical perspective on these issues without adding extra network structures. From information-theoreti standpoint, adding small noise to discrete data corresponds to smoothing data, increasing entropy at rate controlled by the Fisher information of the data distribution. For highly peaked distributions [40] (55) this smoothing is especially effective, substantially increasing entropy and reducing irregularities, thereby providing a better-conditioned target for model fitting. Proposition formally characterizes this entropy increase, while preserving the original KL divergence in the small-noise limit. Table [8 shows the training and evaluation mismatch of ELBO and our methods.
Beyond smoothing effects, Theorem I provides a first-order expansion of the KL divergence with respect to small additive noise. Specifically, when training begins from nonzero noise level og-
of the initial KL objective is reduced by factor proportional to 2 [pllg)) Since this term is always non-negative and strictly positive when p A g, introducing initial noise simplifies the optimization landscape by suppressing fine-scale discrepancies that are otherwise difficult to capture at the outset. Intuitively, the model first learns to match broader statistical structure before refining finer details, stabilizing gradient flow and avoiding early overfitting discrete artefacts.
42

---

Taken together, Theorem and Proposition 2 clarify why choosing a small but nonzero initial noise level is beneficial. From signal processing perspective, of trades off approximation accuracy against numerical tractability. If o is too large, the training target becomes overly blurred, requiring extra effort to recover fine details. If is too small, the model must approximate a distribution with sharp discontinuities or high-frequency details from the outset, which can hinder learning and lead to poor convergence.
Truncated Normal Dequantization We present training-free dequantization strategy example, recently adopted by [90], that naturally fits the diffusion framework and exemplifies our theoretical results. Let Xo E {0, .... 255] denote 8-bit discrete data scaled to [-1, To define discrete density, we use continuous model g(: 0) evaluated on dequantized inputs:
(xx: 0)
(x + u: 0) du.
(123)
Jucl- 250 2t0]
This matches the diffusion-based formulation if we write y = ag + o,6 and choose é ~ a
(O, -T.7) with r 256 Then u - I- 256 256 D
by construction.
ae
Applying change of variables and accounting for Jacobian terms, the variational lower bound becomes:
log Px(x)) 2 BE-TN(O [ogg((x x 2ce) - + D log oe.
(124)
ag
Using the known entropy of the truncated Gaussian [90]. the bound further simplifies to:
D log Po(xo) 2 Be.TN(0.I [log q(x.)) + log(2reeo)) 0.01522 D.
(125)
Choosing the Warm-up Noise We further investigate the influence of different warm-up noise distributions, Gaussian, Laplace, logistic and Uniform, each scaled to have equal variance. As shown in Table 3 Gaussian noise yields the best performance, closely followed by Laplace and logistic, whereas Uniform significantly underperforms. This result aligns with our theoretical intuition that exponential-famie noise distributions, characterized by heavier tails, stabilize training and enhance likelihood estimation. Introducing Laplace noise, previously unexplored in this context, allows us to explicitly examine the impact of heavier-tailed perturbations.
This observation is consistent with the fact that the Gaussian distribution minimizes Fisher information among all distributions with fixed differential entropy [3]. and simultancously maximizes differential entropy among all distributions with the same variance. We attribute the performance gap to tail behavior: Uniform noise has compact support and weakly perturbs extreme values, whereas Laplace and Gaussian assign higher probability mass to large deviations. This results in stronger regularization and more stable gradients. In particular, Laplace noise promotes robustness and sparsity, making it effective for sigh-dimensional or heavy-tailed data.
We consider a baseline that models the data with a mixture of logistic components. Although this parameterization is, in principle, expressive enough to represent multimodal distribution, in practice the baseline fails to recover all modes. We attribute this gap to aptimiization/iiitaliizzzi that arise when the target density exhibits sharp transitions (i.e., a large Lipschitz constant). By contrast, our method is more robust: it captures the distinct modes even with few two mixture components. In this sense, we further generalize the framework of [52] by smoothing both data and model with shared isotropic kernel and training via the score matching to stabilize optimization on high-Lipschin targets.
difficulties
43

---

Figure 2: Random samples from our model trained CIFAR-10 10 for 300000 parameter updates with EMA. The model was trained VDM [40] endpoints, and sampled using 1000 sampling timestep.
Figure 3: Random samples from our model trained CIFAR-10 10 for 300000 parameter updates with
EMA. The model was trained with our endpoints, and sampled using 1000 sampling timestep.
44

---

Figure 4: Random samples from our model trained ImageNet32 for 300000 parameter updates.
The model was trained VDM [40] endpoints, and sampled using 1000 sampling timestep.
Figure 5: Random samples from our model trained on ImageNet32 for 300000 parameter updates.
The model was trained with our endpoints, and sampled using 1000 sampling timestep.
45

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: See our introduction for list of claims including connection with relative entropy with score matching with arbitrary noise.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
46

---

Justification: Yes, our method improves likelihood estimation but does not construct generative diffusion process under alternative noise. See the paper for more details.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Please see our detailed proofs. Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Not only we do show all equations and train on standard datasets, we will open source the code.
Guidelines:
47

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will open source after paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
sot.eeesuammiiioooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https CodeeSumiiii wwwiiisssooosiiiiio
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
48

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: Yes, include all hyperparameters in the paper and will open source code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [TODO]
Justification: We will report the deviatations for NLL in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
49

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httparrrpp..cppllic//cccccccciaa
Answer: [Yes]
Justification: Our paper is just a diffusion model useful for density estimation with standard datasets.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We do not believe our method will have a high risk of abuse as our models are not perceptually SOTA, they only provide for density estimation.
50

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
We are using standard benchmark datasets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the new official source of ImageNet32.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
51

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non sta-srdndard components.
Please refer to LLM policy (nttps://neur/)) http://ppp...oooooeeeeeee/////// for what should or should not be described.
52


Reviews:
[{"review_id": "d5LpW5kDr5", "reviewer": "Reviewer_Nedb", "summary": "This paper presents a novel likelihood bound for noise-driven generative models, aiming to improve both the accuracy and efficiency of maximum likelihood estimation. The main contribution is an extension of the KL divergence, Fisher information connection to general noise perturbations, transforming a theoretical insight into a practical design tool. This allows for the use of realistic noise distributions, such as those reflecting sensor noise or heavy-tailed behavior, while maintaining the standard diffusion training frameworks. By interpreting diffusion as a Gaussian channel, the authors derive an exact expression for the mismatched entropy between data and model. Empirically, their method achieves impressive likelihoods on CIFAR-10 and ImageNet-32.", "questions": "I have mentioned several points in section Strengths And Weaknesses. In addition,\n\n1. Does the framework require condioned score function of nioses have a closed-form expression?\n\n2. In prop 1 and thm 2, how do we obtain theta_1 in real implementations and how is it different from theta. \n\n3. I am a bit confused about the notation $H(p(y_1), \\pi(x))$ in prop 1 (and thm 2). Are you intended to say $p(y_1)$ is something like $q_1^{\\theta_1}(x)$ instead?\n\n4. In (9), \\hat{n} is trained to match n, is n here gaussian?\n\n5. The work you present here is similar to (Paven 1990 Lemma 3.8). Could you comment on their relationship?\n\n\n- Michele Pavon and Anton Wakolbinger On Free Energy, Stochastic Control, and Schrödinger Processes, Progress in Systems and Control Theory, vol 10. Birkhäuser, Boston, MA. 1990", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\nThis paper intends the extend the training framework of regular diffusion to a much more general data corruption process. Several theoretical results seems interesting. For example, in Theorem 1, they showed that the score matching loss flavour can extend to a general addictive disturbance. \n\n## Weaknesses\n\n1. While the authors present a sequence of interesting theortical results, it is still not very clear for me how they used those results to implement their algorithm. It would be helpful if they could provide a pseduocode block to present the entire pipeline. \n\n2. While the authors claim that the proposed method could handle arbitrary noises, it appears that this is somewhat overclaimed. Authors should have a better discussion on what propeties they assume the noise should have.", "was_revised": true, "final_justification": "Most of my initial concerns and questions have been addressed in the rebuttal. I believe much of the original confusion stemmed from the unclear notation and writing in the current version.\n\nWhile I now have a better understanding of how the algorithm works, I do not feel I have sufficient background to fully assess its significance or the comprehensiveness of the empirical study. To reflect this, I have raised my score to 4 while lowering my confidence to 2.", "rebuttal": "We are grateful for the reviewer's appreciation of the clarity and relevance of our work, and for raising insightful concerns. Below we respond to each comments.\n\n---\n**W1:** Thank you for the suggestion. Basically, our approach can be viewed as a concatenation of two distinct noise regimes, each catering to a different range of **noise variance**:\n1. **Low‑variance regime (0 ≤ $\\sigma^2_t$ < $\\sigma^2_0$)**\n\n   * We make **no assumption** on the noise distribution and allow **arbitrary perturbations**.\n\n2. **High‑variance regime ($\\sigma^2_0$ ≤ $\\sigma^2_t$ ≤ $\\sigma^2_1$)**\n\n   * We adopt a **classic Gaussian noise model**.\n   * Leveraging Gaussian noise in this range ensures the standard diffusion training methodologies and the optimisation process.\n\nBy concatenating these two regimes, our method combines flexibility to handle diverse noise types at low intensities with the tighter theoretical guarantees of Gaussian noise at higher intensities. Indeed, a pseduocode block would be helpful to present the entire pipeline, and we appreciate the reviewer’s suggestion to include it and will add the following to the manuscript:\n\n#### — Pre‑processing: Fisher‑consistent arbitrary noise —\n\n```pseudo\nfor each x in X do\n    u ~ Ψ            # Ψ is the ARBITRARY isotropic noise\n    v ← x + σ(0)·u   # 0 < σ(0) ≪ 1\n    store v in X     # no training needed\nend for\n````\n---\n#### — Algorithm 1: Training—\n\n```pseudo\nrepeat\n    v ~ X\n    t ~ U(0,1)              \n    α,σ ← Sched(t)\n    n ~ N(0,I)             # N is the Gaussian noise \n    y ← α·v + σ·n          # Gaussian channel\n    n̂(θ) ← f(y,t;θ)\n    Take gradient descent step on\n        ∇θ‖n − n̂(θ)‖² \nuntil converged\n```\n\n> **Theorem 1**: As σ→0, minimising this loss matches asymptotically equivalent to maximising the log‑likelihood.\n\n---\n#### — Algorithm 2: Likelihood evaluation —\n\n```pseudo\n# 1. identical dequantisation at test time\n    u ~ Ψ;  v ← x + σ(0)·u\n\n    # 2. Monte‑Carlo estimation of DSM integral (Theo.2)\n    η ~ p(η)                      #log-SNR parameterization, Importance Sampling\n    α,σ ← Sched(η)\n    n ~ N(0,I)                    \n    y ← α·v + σ·n                       # Gaussian channel\n    n̂(θ) ← f(y, η;θ)\n    L(θ) ← 0.5·Z·‖n − n̂(θ)‖²            # Z is a normalizing constant\n    \n    # 3. Point‑wise log‑likelihood lower bound (Theo.2)\n    ℓ̂(x;θ) ← H(p(y₁|x), π) + L(θ)       # H(p(y₁|x), π) is pre‑computed\n    return ℓ̂(x;θ)\n```\n> **Theorem 2**: A novel, tighter, noise variance-aware log likelihood bound for standard Gaussian diffusion Models.\n\n---\n**W2:** Thank you for raising this important point. We agree that “arbitrary noise” sounds strong without further clarification, but we have made explicit in the manuscript that our method requires only the following mild, broadly‑applicable assumptions on the noise distribution in Theorems (see, for example, **line 142**). ***No further constraints***: We impose only ***zero‑mean and isotropy***, no additional requirements such as symmetry (unlike [1]), a closed‑form score function (**Q1**), or detailed noise statistics.  In practice, these two conditions are satisfied by the vast majority of noise types encountered in imaging, signal processing, and real‑world measurements.\n\nUnder these minimal assumptions, our method delivers **five** clear benefits:\n1. A new theoretical paradigm for dequantising discrete data. 2. Robustness and stability at $t=0$, resolving train–test mismatch.3. Optimal efficiency and performance in likelihood estimation: Injecting an additional noise block at the start of diffusion improves the efficiency and tightness of our ***likelihood*** bound, yielding state‑of‑the‑art likelihood estimates on **ImageNet-32, -64, -128** without extra computational overhead. 4. A foundation for future extensions to more complex or uncontrollable noise. 5. Exceptionally mild requirements that hold in the wild.\n\nWe hope this clarification demonstrates that our use of “arbitrary noise” is **not** an overclaim, but rather a reflection of the very limited and realistic assumptions underpinning our theoretical and empirical results.\n\n---\n**Questions:** We thank the reviewer for these thoughtful questions. As described in our Algorithms section, we partition the diffusion process into two noise regimes:\n* **Arbitrary‑perturbation regime** for $\\sigma_t^2\\in[0,\\sigma_0^2)$:\n  We impose **no assumptions** on the noise distribution here, and thus **do not** require any closed‑form expression for the conditional score of noise $\\Psi$.\n* **Gaussian channel regime** for $\\sigma^2_t\\in[\\sigma_0^2,\\sigma_1^2]$:\n  In this range, the noise is explicitly modeled as Gaussian. The conditional score for $y=\\alpha_tx+\\sigma_tn$ admits the well‑known analytical form\n\n  $$\\nabla_y\\log p(y|x)=-\\frac{y - \\alpha_tx}{\\sigma_t^2}\\,$$ which our network approximates in the usual fashion.\n\nAccordingly:\n1. **Q1:** We do **not** require a closed‑form score function for arbitrary noise, only the standard Gaussian score in the high‑noise regime.\n2. **Q4:** In (9), $n$ is indeed drawn from $\\mathcal{N}(0,I)$; $\\hat{n}_\\theta$ is trained specifically to match this Gaussian noise within the Gaussian channel.\n\nWe hope this clarifies that our framework leverages analytical tractability where available, without over‑constraining the low‑noise, arbitrary‑perturbation stage.\n\n---\n### Mismatched channel\nBelow is an intuitive statement of our theoretical logic:\n\n> In information theory, a **mismatched Gaussian channel** refers to the scenario where two different inputs:\n>\n> 1. the **true data** $x\\sim p(x)$, and\n> 2. the **assumed input** $q(x;\\theta)$ produced by our network\n>\n> are both passed through the same **Gaussian channel** with variance $\\sigma_t^2$.  In the diffusion framework, any input to this channel will, over infinite time, converge to its **stationary distribution**. However, in practical diffusion models, the true data’s output **never** exactly matches an isotropic Gaussian, whereas our generative **sampling procedure** *does* start from a perfect $\\mathcal{N}(0,I)$. To reconcile this mismatch, we **fix** the channel’s assumed output of $q(x;\\theta)$ to be its true stationary law:\n>\n> $$\n> \\pi(x):=q(y_{1};\\theta_{1}) =\\mathcal{N}(0,I).\n> $$\n>\n> Concretely, we pick $\\theta_{1}$ so that the score network at $t=1$ recovers exactly the isotropic Gaussian equilibrium. Meanwhile, our main parameters $\\theta$ are trained across the entire noise schedule $[\\sigma_{0}^2,\\sigma_{1}^2]$ via denoising score matching.  This ensures that, although real data output never lies exactly in $\\mathcal{N}(0,I)$, our theoretical decomposition (Prop 1, Theo 2) remains valid by anchoring the high‑noise endpoint to the true Gaussian stationary distribution.\n\nAccordingly, below we address **Q2** and **Q3** in the context of the “mismatched Gaussian channel” logic just outlined.\n\n---\n\n### **Q2 – How do we obtain $\\theta_{1}$ in practice, and how does it differ from $\\theta$?**\n| Role  | How it is obtained | Whether it is updated | Purpose |\n| -| -| -| -|\n| **$\\theta_{1}$** (reference parameters) | Fix $\\theta_1$ to ensure the $\\pi(x):=q(y_1;\\theta_1)=\\mathcal{N}(0,I)$ | **Never** updated during the main training loop. | To “anchor’’ the channel’s output at $t=1$ to its true stationary distribution $\\pi(x)=\\mathcal N(0,I)$.  This is needed for the decomposition in Prop. 1 / Thm. 2. |\n| **$\\theta$** (main parameters) | Trained with the denoising score‑matching loss across the full noise schedule $[\\sigma_{0}^2,\\sigma_{1}^2]$.| **Continuously** updated throughout training.    | To predict the noise for every intermediate noise level and drive the likelihood value.|\n\nHence, $\\theta_{1}$ is a *fixed reference* that guarantees $q(y_{1};\\theta_{1})=\\mathcal N(0,I)$, whereas $\\theta$ is the parameter vector we actually optimize.\n\n---\n\n### Q3 \n\n* **$p(y_{1})$** denotes the distribution of the channel output when the *true* input is $p(x)$.\n* **$\\pi(x):=q(y_{1};\\theta_{1})$** is the stationary (isotropic‑Gaussian) output distribution of the same channel under the *assumed* input $q(x;\\theta_1)$ dictated by $\\theta_{1}$.\n* The mismatched entropy term\n\n  $$\n    \\mathcal H\\bigl(p(y_{1}),\\pi(x)\\bigr)=D_{KL}(p(y_{1})\\|\\pi(x) + \\mathcal{H}(p(y_{1}))\n  $$\n\n  therefore measures the mismatch between these two output distributions **after** they pass through the identical channels.\n\nSo the notation is intentional: $p(y_{1})$ is *not* $q(y_{1};\\theta_{1})$; rather, it is the channel’s response to real data, while $q(y_{1};\\theta_{1})$ serves as the fixed Gaussian reference, i.e., stationary distribution of the Markov chain.\n\n---\n\n### Q5\nThank you very much for drawing our attention to [2]; we will cite this result in the revised manuscript. The Lemma 3.8 in [2] is the classical KL–Fisher identity for a **Gaussian** channel, and it underpins several recent studies in deep learning [3], [4]. Prop 1 of our paper indeed builds on the same identity, see Supplementary Material, line 757, so Pavon’s work is certainly relevant.\n\nOur contribution, however, diverges from Pavon’s in two key respects: 1. Theo 1 extends the KL–Fisher connection from Gaussian to aribitrary one. Pavon’s lemma therefore emerges as a special case of our result (line 153). 2. In diffusion models the pathological behaviour at $t=0$ creates a severe train–test mismatch. This issue is outside the scope of [2], but is resolved in our framework by combining Theorem 1 with Proposition 2, which in turn yields the likelihood bound of Theorem 2 and, empirically, state‑of‑the‑art performance.\n\n---\n[1] K. R. Narayanan and A. R. Srinivasa, \"On the thermodynamic temperature of a general distribution.\"\n[2] Michele Pavon and Anton Wakolbinger, \"On Free Energy, Stochastic Control, and Schrödinger Processes.\"\n[3] Siwei Lyu, \"Interpretation and generalization of score matching.\"\n[4] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon, \"Maximum likelihood training of score-based diffusion models.\""}, {"review_id": "wXUYJpxpqv", "reviewer": "Reviewer_1sQW", "summary": "The paper addresses the problem of likelihood estimation in diffusion models. The authors generalize a previously known relationship between the KL divergence and Fisher information to non-Gaussian perturbations. They perform numerical experiments showing likelihood bounds on imagenet 32x32 and cifar10.", "questions": "At the beginning of the paper, you claim that IT bounds may be looser than the ELBO, but it seems like the bound of [30] is slightly tighter than the ELBO. Can you elaborate on that?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "I'll start this review by saying that I am not very familiar with the subject of likelihood estimation for diffusion models. Therefore, it is hard for me to judge novelty and assess some of the claims made in the paper.\n\n**Strengths**\n\nThe authors explain the goal and motivation of the paper very well. The result of Theorem 1 seems novel. Although the experimental results show that Gaussian perturbations still lead to better results, I think this generalization is theoretically interesting and may provide a useful foundation for future analysis. The authors also acknowledge the limitations of their work.\n\n\n**Weaknesses**\n\nI found the paper a bit dense and hard to follow. The proposed method also seems to be computationally expensive, although the authors do not discuss this aspect or compare it to the cost of existing estimation approaches. Also, as mentioned by the authors, the method is limited to likelihood estimation and does not offer a training objective or sampling scheme for diffusion models with more general noise schedules. And finally, the bound does not seem to yield much tighter bounds than existing works.", "was_revised": true, "final_justification": "I read the rebuttal and the authors' explanations. I will keep my score.", "rebuttal": "We are grateful for the reviewer’s thoughtful comments, incisive critiques and detailed questions. We recognise that our work builds on concepts from information theory, statistics and signal processing, domains whose terminology can be dense for readers unfamiliar with them. We hope that our point‑by‑point response has clarified how these principles support our focus on **maximum‑likelihood learning** at the foundations of machine learning.\n\n---\n### Overall \nAverage log‑likelihood is widely recognised as the default metric for evaluating generative image models. Previous work has largely prioritised perceptual quality, *emphasising coarse scale patterns and global consistency of generated images, with common metrics such as the Fréchet Inception Distance (FID) and the Inception Score (IS)*. In contrast, we optimise for likelihood, a criterion that is inherently sensitive to ***fine‑scale details and the exact values of individual pixels***. Mathematically, these objectives **differ markedly** in both form and focus (details see line 20-24 in main text) [1], [2], [3]. The visual appeal of generated images (as measured by e.g., FID) correlates imperfectly with log-likelihood [4]. We focus here on pushing the state-of-the-art in log-likelihood estimation, and while we report FID for completeness, we defer sample quality optimisation to future work (details see supplementary material, *Section C: Samples Quality and FID*).\n\nMeanwhile, modern generative models operate in a continuous domain, yet many real‑world datasets are inherently discrete.  When a continuous‑density model is fitted directly to such data, the ***likelihood evaluation*** becomes singular and severely degrades performance. The conventional remedies, uniform or variational dequantisation [3], inject auxiliary noise but suffer two drawbacks: they require an additional training phase, which is hard to train to the optimal, and, in general, introduce a pronounced train–test gap that inflates the ***negative log‑likelihood (NLL)*** performance. Moreover, another significant issue occurs at the very start of the diffusion process ($t=0$), where numerical instabilities plague both training and sampling. To avoid this, practitioners typically substitute $t=0$ with a small constant $\\epsilon>0$, but this workaround again introduces a discrepancy between training and evaluation.\n\n---\nTo eliminate both sources of discrepancy and numerical instabilities, we prove Theorems 1 and Propositions 2, establishing that, for ***any i.i.d.*** continuous noise distribution with sufficiently small variance, the Fisher divergence is asymptotically equivalent to the KL divergence. Consequently, we may dequantise the discrete data with ***arbitrary*** low‑variance continuous noise and feed the resulting continuous samples into the incremental‑SNR Gaussian channel, that is, the standard diffusion process, without retraining the model or altering its architecture. Building on this insight, Theorem 2 and Proposition 1 yield a tightened analytic bound on the NLL of classical diffusion models (recovering Theorem 1 as the Gaussian special case).  Empirically, our method removes the train–test gap, stabilises optimisation near $t=0$, and delivers ***state‑of‑the‑art*** likelihood estimates across *ImageNet‑32, ‑64, and ‑128* and a competitive performance for CIFAR-10. \n\n---\n### Weakness\nThank you for highlighting the importance of computational cost. \n\nAs you point out, maximum likelihood training in the generative model setting can be extremely demanding, after all, we optimise for every pixel value, which is why most prior likelihood‑based studies have been confined to low resolutions (e.g. 32×32). However, in response to another reviewer request, we have extended our experiments to 64×64 and 128×128 resolutions (see our reply to Reviewer trPf). \n\nTypically, achieving benchmark likelihood estimation performance requires several million training iterations, and the training process usually takes a week, or even a month or longer. However, our likelihood bounds with additional methods largely mitigate this issue: we require only 300 000 iterations to achieve optimal performance. In the main text (*lines 302–305*) and in greater detail in the Supplementary Material (*lines 976–984*), we present comparative tables that reports **training iteration counts** for different likelihood‑estimation methods. Although absolute iteration/seconds vary with hardware and network architecture, iteration count is a robust proxy for relative compute cost (details see Table 1 in main text):\n\n---\n\n### Models with information-theoretic methods\n\n| Model| CIFAR-10 NLL ↓ | ***Iter. (million)*** |\n| -| -: |-: |\n| ScoreODE (second order)  |3.44 |1.3 |\n| ScoreODE (third order) |3.38 |1.3 |\n| ScoreFlow|2.90 |0.3 |\n| Flow Matching|2.99 |0.391 |\n| Stoch. Interp|2.99 |0.5 |\n| i‑DODE|2.56 |6.2|\n| **ISIT (SP with IS, ours)**|**2.49** |0.3 |\n| **ISIT (VP with IS, ours)**|2.50 |0.3|\n\n---\n\n### Models with variational bounds\n\n| Model| CIFAR-10 NLL ↓ | ***Iter. (million)*** |\n| - | -: | -: |\n| VDM |2.65 |10 |\n| DiffEnc|2.62 |8 |\n| MuLAN|2.55 |8 |\n| BSI|2.64 |10|\n| W‑PCDM (VDM weight)|***2.35***|2 |\n| W‑PCDM |10.31|2|\n\n* **previous information‑theoretic bound** typically require on the order of **1 million** training iterations merely to reach competitive likelihoods, but still do not attain state‑of‑the‑art performance (cf. main text, line 39–43).\n* **ELBO methods** converge in *8-10 million* iterations to the SOTA likelihood.\n* ***Our information‑theoretic bound*** converges in only ≈ 300 000 iterations to second best **likelihood (in BPD)** on CIFAR-10 and *SOTA on ImageNet*.\n\n\nMoreover, we discussed our training iteration speed in line 304 of the main text. It is worth noting that in the likelihood‑estimation domain, even a **0.01 bits per dimension (BPD)** improvement represents a substantial advance (e.g. the improvement of MuLAN over i-DODE). While ELBO (W‑PCDM) remains the strongest existing approach on CIFAR-10, Theorem 2 allows us to nearly close the performance gap **at a fraction of the computational effort**.\n\n---\n### Claims of limitation\nAlso, we really appreciate the reviewer for the observation of claims of *limitations*. Let us clarify the intended scope and contributions of our work:\n\n1. **Focus on likelihood rather than perceptual metrics.**\n   When we state that our method is “limited to likelihood estimation,” we mean precisely that our training objective, and hence our strongest empirical gains, are measured in terms of ***negative log‑likelihood (NLL)*** rather than FID. A fuller discussion of this point, and comparative FID results, can be found in our response to Reviewer trPf and line 985-990 in supplementary material.\n\n2. **Sampling schemes for general noise schedules.**\n   It is true that we do not propose a new sampling algorithm optimised for arbitrary noise. Our primary goal was to develop a principled training objective, namely, the novel likelihood bound of diffusion models, rather than to accelerate or improve diversity of sample‑quality under arbitrary noise. Details of our algorithm pipeline appear in our response to Reviewer Nedb. We regard the design of specialised sampling methods as an important avenue for future work and acknowledge this as a current limitation.\n\n3. **Tighter bounds through Theorem 2.**\n   Although our bound is framed as a likelihood objective, it is indeed tighter in the sense that **lower NLL corresponds to a strictly better bound** on the data density. In practice, we achieve noticeably improved NLLs over prior methods without increasing computational cost on ImageNet-32, -64, and -128.\n\n4. **Relation to multiscale approaches.**\n   We note that W‑PCDM attains superior CIFAR‑10 NLL by augmenting its architecture with multiscale modelling components, specifically, Laplacian pyramids and wavelet transforms. We fully expect that integrating analogous multiscale designs into our framework would similarly elevate our performance, and we intend to explore this in future work.\n\nWe hope these clarifications demonstrate both the precise remit of our contribution and the potential for further enhancements.\n\n---\n### Questions\nWe appreciate the reviewer’s insightful question.\n\nReference [5] was indeed one of the key works that motivated our use of information‑theoretic tools in diffusion modelling. In that paper, the authors employ the classical I‑MMSE identity from information theory to derive a novel diffusion bound and report a CIFAR‑10 NLL of **2.90**, which is certainly a promising result. However, this value remains above the best ELBO‑based bound at the time, approximately *2.65 BPD*, so in practice their bound does not yet surpass the ELBO in tightness (recall that lower NLL indicates a tighter log‑likelihood bound). Building on this insight, our work is instead guided by Song et al.’s [3] exposition of the KL–Fisher connection for score matching, which yields substantially tighter likelihood bounds. We discuss these developments in greater detail in the Related Works section (line 281).\n\n---\n[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.\n[2] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696–21707, 2021.\n[3] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415–1428, 2021.\n[4] Subham Sahoo, Aaron Gokaslan, Christopher M De Sa, and Volodymyr Kuleshov. Diffusion models with learned adaptive noise. Advances in Neural Information Processing Systems, 37:105730–105779, 2024.\n[5] Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information-theoretic diffusion. In The Eleventh International Conference on Learning Representations, 2023."}, {"review_id": "jP8AWBxfCW", "reviewer": "Reviewer_trPf", "summary": "The paper introduces a novel lower bound for the log-likelihood on individual datapoints, aiming to improve the training and evaluation of generative models. To address the high variance issue often found in Monte Carlo-based estimations, the authors incorporate importance sampling techniques, leading to a more stable and potentially more accurate optimization process. The proposed formulation is grounded in solid mathematical derivations and presents a theoretically sound contribution to the field of probabilistic modeling.", "questions": "1. Could you provide experimental results on larger datasets, such as LSUN or ImageNet-256, to evaluate how the method scales to higher-resolution or more diverse image distributions?\n2. Please provide qualitative comparisons. The degradation of FID score raises concerns on practical usages.", "limitations": "The primary limitation of the paper is the weak experimental support. Without evaluation on more challenging datasets or broader comparisons to baseline methods, it is difficult to assess the practical relevance and robustness of the proposed method. Additional experiments would significantly strengthen the paper.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper presents a mathematically rigorous approach, with clearly defined objectives and derivations. The proposed lower bound is both theoretically interesting and potentially useful in improving likelihood-based training of generative models.\n2. The incorporation of importance sampling to reduce variance is a well-motivated and technically appropriate choice.\n\nWeaknesses\n1. A significant limitation is the lack of comprehensive empirical validation. The experiments are limited to relatively small-scale datasets such as CIFAR-10 and ImageNet-32. This raises concerns about the generalizability and scalability of the method to larger, more complex datasets.\n2. While improvements in negative log-likelihood (NLL) are observed, the degradation in Fréchet Inception Distance (FID) suggests that the method may prioritize likelihood at the expense of sample quality. This trade-off is not fully explored or explained in the paper.", "was_revised": true, "final_justification": "As the experiments support authors' argument, and now I understand the literature regarding likelihood, I raise my score.", "rebuttal": "We are grateful to the reviewer for the valuable feedback and the opportunity to clarify the objectives, evaluation criteria, and contributions of our work.\n\n---\n### General perspective of evaluation of generative models\nFrom a broad standpoint, we first seek to dispel any confusion regarding the purpose and evaluation criteria of our work before addressing the specific questions in turn. It is essential, for the reviewers’ benefit, to clarify our precise objective, the standards by which we judge success, and the context in which this study is framed. Just as selecting appropriate training and optimisation strategies is necessary to achieve strong performance in a given application, so too is the choice of evaluation metric pivotal for drawing valid conclusions. We must stress that the primary focus of this paper is on **maximising the likelihood learning** metric, specifically, ***negative log‑likelihood measured in bits‑per‑dimension (BPD; lower is better)***, rather than on optimising *Fréchet Inception Distance (FID)* or *Inception Score (IS)*, for the following reasons.\n\nModel samples undoubtedly serve as a valuable diagnostic tool, often enabling us to form an intuition about why a model may underperform and how it might be improved. From this standpoint, a generative model ought to produce samples that are indistinguishable from those in the training set, whilst encompassing its full variability. To quantify these properties, a variety of metrics, such as the IS and the FID, have been proposed. However, both qualitative and quantitative assessments based on model samples can be misleading with respect to a model’s density-estimation capabilities, as well as its effectiveness in probabilistic modelling tasks beyond image synthesis [1]. Consequently, average log‑likelihood remains the de facto standard for quantifying generative image‑modelling performance. For many sophisticated models, the average log‑likelihood is challenging to compute or even approximate. Indeed, it is possible for a model with sub‑optimal log‑likelihood to generate visually impressive samples, or conversely, for a model with excellent log‑likelihood to produce poor samples [1], an observation that underlines the lack of a direct relationship between **FID** and **negative log‑likelihood (NLL)**.\n\nFrom an information‑theoretic standpoint, it is well known that maximising the log‑likelihood of a probabilistic model is equivalent to minimising the ***KL divergence*** from the data distribution to the model distribution. By contrast, FID operates by fitting multivariate Gaussian distributions to the embeddings of real and generated images and then measuring their discrepancy via the Fréchet distance (equivalently, the ***2‑Wasserstein or Earth Mover’s distance***) [2]. Clearly, the mathematical formulations of these two metrics diverge fundamentally: one corresponds to a mismatched estimation problem under a KL‑based criterion, while the other embodies an optimal‑transport task.\n\nMoreover, FID conflates both fidelity to the real data distribution and the diversity of generated samples into a single score, and its absolute value is highly sensitive to myriad factors, ranging from the number of samples and the particular checkpoint of the feature extractor network to low‑level image‑processing choices. Consequently, the visual appeal of generated images, as quantified by FID, correlates only imperfectly with a model’s log‑likelihood performance. In *our work*, we concentrate on advancing the ***state of the art in likelihood estimation***; although we report FID scores for completeness, we leave the optimisation of sample quality to future research.\n\n---\n### W1 & Q1\n\nWe thank the reviewer for emphasising the need for broader empirical validation. To the best of our knowledge, maximum‑likelihood learning in the generative model setting is extremely demanding, after all, we optimise for every pixel value, which is why nearly all prior likelihood‑based studies have been confined to low resolutions (e.g. 32×32) [4][5]. In fact, most density‑estimation papers cap out at 128×128 due to the quadratic growth in pixel count and the resulting computational burden, and to date no likelihood evaluations (bits/dim, BPD) for dataset of higher-resolution (e.g. 256×256) have been systematically reported in the literature.\n\nTo address this gap, we have now conducted additional experiments at 64×64 and 128×128 on ImageNet, achieving state‑of‑the‑art bits‑per‑dimension. For completeness, comparison between our proposed model and other competitive models in the literature in terms of ***expected negative log likelihood*** on the test set computed as **bits per dimension (BPD)** is summarised below:\n\n| Models| ImageNet‑64 |  ImageNet‑128  |LSUN-128|\n|-|-|-|-|\n|Sparse Transformer [Child et al., 2019] |3.44|-|-|\n|Flow++ [Ho et al., 2019]|3.69|-|-|\n|Very Deep VAE [Child, 2020]|3.52|-|-|\n|Improved DDPM [Ho et al., 2020]|3.54|-|-|\n|VDM [Kingma et al., 2021]|3.40|-|**1.44**|\n|Flow Matching [Lipman et al., 2022]|3.31|2.90|-|\n|W-PCDM [Li et al., 2024]|2.95|2.64|-|\n|ISIT (**Ours**)|**2.91**| **2.59** |2.28|\n\n> **Note:** LSUN 128×128 are not yet established benchmarks for density estimation, and downsampling protocols in the literature are inconsistent. Accordingly, we do not attempt direct comparisons to previous methods on these datasets; our results simply provide a ballpark estimate of how well our proposed method scales to higher resolutions.\n\nAdditional qualitative comparisons and  ablation studies on CIFAR-10 could be found in **Tables 6-11** in supplementary material.\n\n---\n### W2 & Q2\nThank you for the detailed review and thoughtful feedback. Below we address specific questions.\n\nOur work is principally concerned with enhancing the likelihood of diffusion models for applications such as lossless compression (details see our reply to  Reviewer BC5c), rather than targeting improvements in FID. Nevertheless, our reported FID metrics remain superior to those of other likelihood‑focused methods (see **Tables 12–14 in the supplementary material**, and Section C: Sample Quality and FID). As noted in [3–6], objectives that favour likelihood often incur poorer FID performance (comparisons are provided in Table 1).\n\nFor example, among models with similar likelihood performance, such as *i‑DODE*, *MuLAN* and *DiffEnc*, our method not only achieves the best negative log‑likelihood but also retains the lowest FID despite requiring substantially fewer training iterations. Conversely, methods like *W‑PCDM* that explicitly optimise for FID exhibit a marked reduction in likelihood performance.\n\nAdditional FID results are presented in **Table 6**. We would be pleased to include FID scores for additional models should the reviewers and Area Chair consider this beneficial. We emphasise that our variance‑aware likelihood bound markedly enhances diffusion models’ likelihood while inducing only a marginal effect on FID, as evidenced by **Tables 12–14**.\n\nMoreover, our work makes a novel contribution by analysing how different noise schedules influence FID. In the *supplementary material (lines 916–942)*, we provide a theoretical account of FID degradation. Briefly, from an autoencoder perspective, the quality of generated samples, reflected by FID, depends critically on the conditional distribution $q(x|y_0)$, which functions as a decoder reconstructing the data. When the signal‑to‑noise ratio (SNR) at $t=0$ is sufficiently high, the posterior $p(y_0|x)$ becomes sharply concentrated around $y_0 = \\alpha_0 x$, thereby imposing stringent constraints on reconstruction loss. This induces high sensitivity to noise: even small deviations in the initial noise level can substantially degrade reconstruction fidelity and thus worsen FID. However, while increasing noise makes signal recovery more challenging, leading to FID fluctuations, the associated trade‑off with negative log‑likelihood is effectively resolved by our Theorem 1 and Proposition 2, as consolidated in Theorem 2.\n\n---\nIt is worth emphasising that our current approach does not employ several techniques known to improve FID, such as truncation, specific weighted score matching/ELBO or other empirical heuristics, and thus our reported FID scores could be further reduced by incorporating these methods. Nonetheless, our primary contribution lies in achieving state‑of‑the‑art log‑likelihood performance. We regard the integration of such sample‑quality‑oriented strategies, in particular explicit optimisation of the Wasserstein distance through tailored noise schedules, as a compelling direction for future work.\n\n---\n[1] L. Theis, A. van den Oord, and M. Bethge. A note on the evaluation of generative models. International Conference on Learning Representations, 2016\n[2] Kynkäänniemi, Tuomas & Karras, Tero & Aittala, Miika & Aila, Timo & Lehtinen, Jaakko. The Role of ImageNet Classes in Fréchet Inception Distance. International Conference on Learning Representations, 2023 \n[3] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.\n[4] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696–21707, 2021.\n[5] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415–1428, 2021.\n[6] Nichol A Q, Dhariwal P. Improved denoising diffusion probabilistic models. International conference on machine learning. PMLR, 2021: 8162-8171."}, {"review_id": "89ZsFKKH69", "reviewer": "Reviewer_BC5c", "summary": "This paper introduces a new likelihood bound for noise, driven generative models, improving both the accuracy and efficiency of maximum likelihood estimation by extending the KL divergence, Fisher information connection, to arbitrary noise perturbations. The method demonstrates competitive results on CIFAR-10 and ImageNet-32 without data augmentation.", "questions": "As we all know that FID is the primary metric for generation, Table 2 shows that only a few baseline methods are evaluated with this metric on ImageNet, which seems unusual. Could you clarify the reason for this?", "limitations": "Authors already list their limitations in the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengthens:\n1:  The paper handles an important limitation of perturbation-based likelihood models by providing a more efficient and theoretically grounded estimation approach.\n\n2:  It achieves competitive or state-of-the-art results on standard benchmarks like CIFAR-10 and ImageNet-32 without relying on data augmentation.\n\n3: The proposed theoretical framework has promising implications for modelling both continuous and discrete data, broadening its practical impact.\n\nWeaknesses:\n1: The practical implementation details for selecting appropriate noise distributions could be elaborated more clearly for readers who doesn’t have strong relevant background.\n\n2: The paper would benefit from more empirical comparisons to alternative state-of-the-art likelihood estimation techniques beyond diffusion models if possible.\n\n3: The results only on image datasets; additional experiments on other modalities would help to strengthen claims about broader applicability.\n\n4: The paper’s overall formatting is not very good, and small errors like these should have been avoided. For instance, at the end of some paragraphs, there are blank boxes.", "was_revised": true, "final_justification": "The author's thoughtful rebuttal (including experiments and questions I have asked) has addressed my concerns.", "rebuttal": "We thank the reviewer for their careful reading and constructive comments. Below, we clarify the main focus of our work and the role of our experiments.\n\n---\n\n### 1. Emphasis on Theoretical Contributions\nWe would like to stress that the primary contribution of our paper is **theoretical**. In particular:\n\n- We develop a **maximum‑likelihood learning** framework for diffusion models under general noise conditions with a novel variance-aware NLL bound. \n- Our analysis yields **strong resutls** for likelihood estimation, extending classical score‑matching theory well beyond the isotropic Gaussian setting.\n\nWhile previous generative-modeling research has predominantly emphasised practical heuristics, whether in network architectures or sample‐quality optimisation, our aim is to establish a **unified, principled learning theory** that rigorously addresses the shortcomings of score matching.\n\n---\n\n### 2. Role of Experiments\n\nWhile we fully acknowledge the importance of empirical validation, our experiments are primarily intended to **verify** the correctness and predictive strength of our likelihood bounds, particularly in relation to noise variance. Although these methods achieve *state‑of‑the‑art* performance, they function chiefly as a **sanity check**. We contend that illustrating concordance between our theoretical analysis and empirical results both reinforces our theoretical claims and attests to their practical significance.\n\n\n---\n\nWe hope this clarification helps to underscore our paper’s focus on **general learning theory** and **parameter‐estimation** guarantees, with experiments playing a supportive, validating role. Thank you again for your insightful feedback.\n\n---\n### General Perspective and aiming on Score Matching\n\nScore matching has become a powerful tool in machine learning for fitting unnormalized statistical models, most notably in score‑based generative modeling [1-3]. However, it still relies on computing the trace of the Hessian matrix, which becomes a major bottleneck in high‑dimensional problems. To address this, researchers have proposed:\n\n- Approximate backpropagation [4]\n- Curvature propagation [5]\n- Reformulating score matching as a denoising objective [6]\n\nThese approaches either reduce the cost of Hessian‑trace computations or bypass second‑order derivatives entirely. Moreover, [7] established a theoretical link between score matching and maximum likelihood estimation under isotropic Gaussian noise, highlighting the method’s robustness to noisy data. Yet, the assumption of noise remains a limiting factor for real‑world applications. In practice, degradations in computer vision and signal processing are far more complex:\n1. **Real‑camera noise** arises from multiple sources, shot noise, thermal noise, dark current, etc., and is further processed by in‑camera pipelines, resulting in spatio‑chromatic correlations and both signal‑dependent and signal‑independent components [8][9].\n2. **Simplistic noise models** used in most score‑matching studies bear little resemblance to these real conditions.\n3. **Blur kernels** in tasks like face super‑resolution cannot be estimated explicitly, which further limits the applicability of Gaussian-assumption.\n\nMotivated by these challenges, we ask:\n\n> _Can the theoretical relationship between likelihood estimation and score matching be extended beyond the Gaussian to any ***arbitrary noise even without explicit forms***?_\n\nWe also note that recent work on score matching with missing data [11] has focused on truncated, denoising, and sliced variants. In contrast, our framework can be viewed as a **highly general form of denoising score matching**:\n- By treating a subset of corrupted observations as “missing,”\n\nWe believe this perspective not only broadens the applicability of score matching to realistic noise environments but also lays the groundwork for future exploration of its connections to missing‑data methodologies.\n\n---\n### Weakness\n**W1:** We thank the reviewer for this suggestion. In practice, we chose our three canonical noise distributions for the following reasons:\n\n* **Uniform noise for dequantization**\n  Uniform noise is the de‑facto standard for dequantising discrete image data, hence its widespread use as a benchmark in likelihood‑based models.\n\n* **Laplace noise as a heavy‑tailed exemplar**\n  The Laplace distribution is the prototypical heavy‑tailed noise, and it is frequently employed in machine learning regularisation and robustness studies.\n\n* **Gaussian noise for its optimal information‐theoretic properties**\n  (see lines 333-335).\n\nWe thank the reviewer for this suggestion. In response, we have augmented our experiments with **logistic noise**, a bell‑shaped symmetric distribution, which, at a fixed variance of $10^{-4}$, delivers performance nearly identical to that of Laplace noise. These additional results further validate the robustness of our variance‑aware likelihood bound across a wide range of noise distributions. We will incorporate these findings into the relevant tables and include full implementation details in the revised manuscript. Crucially, our use of uniform, Laplace, Gaussian and logistic noise is intended solely to illustrate the generality of our theoretical framework; a thorough exploration of application‑specific noise schedules and their optimal tuning will be pursued in future work.\n\n---\n**W2:** Thank you for this suggestion. We agree that a broader empirical comparison would strengthen our claims. As shown in Table 1 in main text, we have already compared our method against several state‑of‑the‑art non‑diffusion models on CIFAR‑10 and ImageNet‑32; however, we did not explicitly indicate each model’s category. Below, we present a summary of the results on ImageNet‑32 and ImageNet‑64, now including the model types:\n\n\n| Model| Type| ImageNet-32| ImageNet‑64|\n| -| -| -| -|\n| **Our Method (ISIT)**    | Diffusion|**3.01**|**2.91**|\n| Very Deep VAE [Child, 2020] | VAE |3.80|3.52|\n| Flow Matching [Lipman et al., 2022]|Flow|3.53|3.31|\n| Sparse Transformer [Child et al., 2019]|Autoregressive|-|3.44|\n| W-PCDM [Li et al., 2024]|Diffusion|3.32|2.95|\n\nBy adding a “Type” column, readers can immediately see which paradigm each baseline represents. We will update and include the fully annotated table in the revised manuscript.\n\n---\n**W3:** To date, likelihood estimation with generative models on truly discrete data (e.g., text) remains under‑explored. This is because most generative frameworks define continuous densities, which become singular when evaluated on a discrete support. As a result, researchers commonly employ **dequantisation** to bound discrete likelihoods, see our response to Reviewer 1sQW for a full discussion. Existing dequantisation methods, however, introduce extra training overhead and suffer from a train–test mismatch.\n\nOur **Theorem 1** and **Proposition 2** show how to resolve both of these issues, yielding tighter likelihood bounds **without** incurring additional training cost. For this reason, we have not included extra experiments on textual data in the current manuscript.\n\nInstead, to demonstrate the real‑world utility of our likelihood bounds, we applied the **bit‑back coding scheme** for **lossless compression** on image data. Concretely, we compressed CIFAR‑10 using our variance‑aware bounds and obtained the following results:\n\n| Model| Compression Rate (bits/dim) |\n| -| -|\n| LBB [Ho et al., 2019]|3.12|\n| IDF [Hoogeboom et al., 2019]|3.26|\n| VDM [Kingma et al., 2021]|2.72|\n|W-PCDM [Li et al., 2024]|**2.37**|\n|ISIT (**Ours**)|2.57|\n\n*(We leave the improvements of this avenue of research for further work.)*\n\nThese experiments confirm that our theoretical improvements to likelihood estimation translate directly into practical gains in lossless image compression.\n\n---\n**W4:** Thank you for pointing out the formatting issues. We will carefully revise the manuscript to eliminate unintended artifacts and improve overall presentation.\n\nRegarding the blank boxes appearing at the ends of paragraphs (e.g. lines 153, 187, 212), these were deliberately included to mark the end of a proof, an established convention (a “□” or QED symbol) in theoretical work. However, we appreciate that this may be unfamiliar to some readers and can be mistaken for a formatting error. In the revised version, we will include a brief note clarifying their purpose.\n\n---\n### Question\n\nThank you for raising this insightful question. In ***maximum likelihood learning*** research, the community’s standard comparison metric is ***negative log‑likelihood (measured in bits‑per‑dimension), rather than FID***. FID is generally reported only as a secondary measure for completeness, not as a benchmark for contribution claims. As we explain in our response to Reviewers 1sQW and trPf, this is precisely why so few prior works compute FID on ImageNet, most simply include it to round out their experimental section. We appreciate your attention to this detail and hope this clarification is helpful.\n\n\n\n---\n[1] Song, Yang, and Stefano Ermon. \"Generative modeling by estimating gradients of the data distribution.\" \n[2] Song, Yang, et al. \"Sliced score matching: A scalable approach to density and score estimation.\" \n[3] Song, Yang, et al. \"Score-Based Generative Modeling through Stochastic Differential Equations.\" \n[4] Kingma, Durk P., and Yann Cun. \"Regularized estimation of image statistics by score matching.\" \n[5] Martens, James, Ilya Sutskever, and Kevin Swersky. \"Estimating the hessian by back-propagating curvature.\" \n[6] Vincent, Pascal. \"A connection between score matching and denoising autoencoders.\" \n[7] Lyu, Siwei. \"Interpretation and generalization of score matching.\" \n[8] Zamir, Syed Waqas, et al. \"Cycleisp: Real image restoration via improved data synthesis.\"\n[9] Yue, Zongsheng, et al. \"Variational denoising network: Toward blind noise modeling and removal.\"\n[10] Givens, Josh, Song Liu, and Henry Reeve. \"Score Matching with Missing Data.\""}]

Author rebuttal:

Meta-review: This paper derives a connection between KL-divergence and Fisher-divergence for diffusions with arbitrary noise distributions. While these results are asymptotic and hold for small noise regimes, they nevertheless address two key issues in likelihood training for diffusion models: (a) handling the degeneracy at $t=0$, and (b) handling the train-test gap that arises when de-quantising. These new results are used to design a method that obtains SoTA or nearly SoTA NLL, using significantly less training time, on both CIFAR-10 and ImageNet datasets. 

Ultimately after the rebuttal/discussion period, the reviews on this paper were overall positive, recognizing both the theoretical and practical merits of this work. For the camera-ready, I ask that the authors incorporate the rebuttal to Reviewer Nedb regarding both the pseudo-code for the ISIT algorithms and the notational improvements. Furthermore, some exposition about how the algorithms arise as a consequence of both Theorem 1 and Proposition 1 would substantially improve the clarity of the manuscript.
Decision: Accept (poster)