Neural Entropy
Authors: Akhil Premkumar
Venue: NeurIPS 2025 spotlight ()
Keywords: Diffusion models
thermodynamics
information theory
statistical mechanics
Primary area: deep_learning
TL;DR: We introduce a measure of the information content stored in a neural network in the context of diffusion models.

Abstract: We explore the connection between deep learning and information theory through the paradigm of diffusion models. A diffusion model converts noise into structured data by reinstating, imperfectly, information that is erased when data was diffused to noise. This information is stored in a neural network during training. We quantify this information by introducing a measure called \textit{neural entropy}, which is related to the total entropy produced by diffusion. Neural entropy is a function of not just the data distribution, but also the diffusive process itself. Measurements of neural entropy on a few simple image diffusion models reveal that they are extremely efficient at compressing large ensembles of structured data.

Neural Entropy
Akhil Premkumar
Department of Applied Physics
Yale University New Haven, CT 06511, USA
akhil. premkumar @yale edu
Work done while at the University of Chicago
Abstract
We explore the connection between deep learning and information theory through the paradigm of diffusion models. A diffusion model converts noise into structured data by reinstating, imperfectly, information that is crased when data was diffused noise. This information is stored in neural network during training. We quantify this information by introducing a measure called neural entropy, which is related to the total entropy produced by diffusion. Neural entropy is function of not just the data distribution, but also the diffusive process itself. Measurements of neural entropy a few simple image diffusion models reveal that they are extremely efficient at compressing large ensembles of structured data.
Introduction
How much information is stored in a neural network? simple example, consider training a neural network to store an 8-bit grayscale image of dimension H x W pixels. The network learns smooth map from pixel co-ordinates to grayscale intensity values from H x W bytes of raw data. This is not the total number of bytes of the parameters that constitute the network, and not every image of size H x W contains the same amount of information. But it is reasonable to expect that if we push images of higher and higher resolutions/detail onto the same network, at some point the network will not be able reproduce the images faithfully
The question is even more pertinent in the context of generative models. These models are capable of producing seemingly endless variations the original training data, say images, but that does not mean the neural network has stored an infinite number of images. Rather, generative models store a distribution of images, call pa, and the generated samples are points that interpolate the training data in Pa- This is similar to how the network from the prior example blends the grayscale intensities between neighboring pixels. So the analogous question to ask is this: how many bytes of data is pa worth? The primary goal of this paper is to answer this question in the context of diffusion- generative models (hint: not simply the Shannon entropy of pa, see App. C.2)
Diffusion models serve as a natural bridge between information theory and machine learning, having been inspired by ideas from non-equilibrium thermodynamies (]. which itself can be viewed as an principles to physical systems 24 Very briefly, samples from
application of information-theod training dataset incrementally noised till they are distributed while a neural network learns to reverse these noising steps. Once trained, the network can transform random Gaussian vector into highly structured output that resembles typical member of the training data. In the continuum limit, the noising and denoising stages become diffusive processes the thermodynamic properties of which are well established [7-9.
generic Gaussian, call it Peq
Diffusion gradually wipes out information from Pa over time (cf. Fig. 6). The information loss is quantified by the total entropy produced during the process, Scot- Within this framework, we can
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

define the information content of Pa in relation to the diffusion process itself-it is the amount of information that must be reinstated drive the process away from its equilibrium state Pog back to Pa- It is precisely Stot (cf. App.[c A well-trained diffusion model retains nearly all of this information in its neural network. Therefore, we can characterize the information content of the network by a quantity we call the neural entropy, SNN R Stot. defined in Eq. (8)..
Before we delve into the details, a few points must be clarified. First, is important to stress that neural entropy quantifies the information stored in a perfectly trained network; it is not the entropy of the phase space density over the neural network's s internal microstates. Second, no diffusion model can reconstruct Pa perfectly because we only have access to a finite number of training samples from it [10), and training is imperfect even with a large dataset. Third, the neural network encodes and interpolates the given information. drawing from its own inductive biases to fill in the gaps between the training data [I]] This is why diffusion models are able to estimate very high-dimensional distributions even from relatively small datasets [12] Consequently, neural entropy is just one part
of a slew of variables, like the choice of network architecture, optimization algorithm, etc. that ultimately affect the overall model performance.
Despite these caveats, empirical measurements of neural entropy reveal interesting insights into the behavior of neural networks. First, in setting where the inductive biases are relatively weak and the data distribution is largely unstructured (e.g. Gaussian mixtures). diffusion models tend to struggle to reconstitute pa accurately as more information is fed into the network (see Fig. [T]. Second, in image diffusion models with U-nets trained on real images, the neural entropy shows a distinct logarithmic scaling with the number of training samples NN (see Figs. and 16). That is, the marginal information gained per sample decreases approximately as I/N. The quality of generated images also reflects this trend (see Fig. [8) Provided that N is sufficiently large for the model to approximate Pa well, diffusion models compress the images with great efficiency, since they encode the ensemble statistics of the training data; storing each image separately in old-fashioned memory would have incurred a linear cost in N.
le4
E
1.25
1.00
0.75
CIFAR-IO
MNIST
0.50
0.25
10a
10
10a
Figure 1: Neural entropy vs. number of samples for im- age diffusion models.
2 Schrödinger's Gedankenexperiment
The link between diffusion and information theory can be traced back to a thought experiment introduced by Erwin Schrödinger in a seminal paper from 1931 [13] Consider diffusion process like the dissolution of an ink drop in water. Common experience suggests that the ink particles would homogenize over the available volume of water and remain in this diffused state indefinitely. However, there is a very small but non-zero probability that the ink particles collect together in some exotic configuration at later time. Schrödinger asks: what is the probability that the particles diffuse back to their original state?
To answer this question in a simpler setting we study random walkers on a one-dimensional lattice. The lattice sites are spaced by and the walkers jump to one of their nearest neighbors at each time step. The density of walkers at x updates as
p(x, + At) qι(x l)p(== l,t) + 92(x + E)p(1 + l,t),
(1)
where qR (x) (or (x)) is the probability that walker at jumps to the right (or left) at time At (see App. B. for notation). But that does not mean exactly (अ) fraction of all walkers at x always jump rightwards in At; over several trials, there will be small fluctuations in the actual number of walkers that make such a transition. Such fluctuations can accumulate to evolve p in a manner different from Eq. (I). albeit with low probability. One may liken this to throwing perfectly fair coin 1000 times: due to fluctuations, we do not always obtain the expected outcome of 500 heads and 500 tails, and in fact there is a minute probability of 2-100 of obtaining all heads (or all tails).
For appropriate choices of yn and (x),,, there exists an equilibrium distribution Peq which satisfies the detailed balance conditions corresponding to Eq. (I. Let I be a large enough time that
2

---

the walkers can equilibrate to very nearly Peq from another state Pa. Starting from Peg at = 0, the probability that the walkers would migrate back to the distribution Pa time is 13
h(rr
P [aa x exp
M Σ
(eg(2o))(2r)zg) log g (ro)
= MDRL(All))
(2)
20.22
Here, M is the total number of walkers on the lattice and g(rr (x)) is the probability that a walker at 2o ends up at 2T under Eq. ( That is, is the transition kernel for Eq. (T). On the other hand h(rr/z)) is kernel that transports Pog (Za) to pa (arr). There are many kernels h that accomplish this, but for sufficiently large M the exponential in Eq. (2) picks out an optimal kernel h, for which the Kullback-Leible divergence D (g) is minimum. can shown that the evolution of Poq > Pa under h2 is reversal (playback) of the transformation pa Pon under Eq.
14 15
With Peq (or g) fixed, P can be understood as a distribution of distributions. sample from P is distribution that Peq can fluctuate into at time f = T. under the dynamics in Eq. (I. If . is large, is sharply peaked at Peqi the probability that the walkers would deviate from this configuration exponentially small. This is true even with n.tnalural processes have preferred direction of time, and they rarely evolve in reverse. Eq. (2 intimately related to the Second Law of Thermodynamics. In fact,
T-At Seot Σ
h. (1r+4rr 2 D
Σ Ze
t) log
(3)
g(re+A1 x4))
where p(z,, t) is the distribution of walkers they evolve between Peq and pa. and Stot is the total entropy generated if pa was subjected to Eq. (I) for time T (cf. App. A.2). In simple terms, Stot quantifies the time irreversibility of the process pa Peg
[9] 16
We discuss the meaning of
Stot in greater detail in the upcoming sections and App.
C
Combining Eqs. 2 and 3 we obtain key relation between the Shannon information content of the outcome Pas per walker, and the total entropy 18 19]:
Scot
log, P [pa].
log 2
M
If we observe a set of random walkers that was initially at equilibrium and find that they are still distributed as Peq we learn nothing new; that was the outcome we expected. However, in the unlikely event that we observe the random walkers distributed as pa. we would gain an amount of information commensurate with the total entropy generated in diffusing Pa + Peq
3 Diffusion models and Maxwell's demon
We can enhance the probability of obtaining the outcome pa by bringing closer to h.. That is, we adjust the jump probabilities in Eq. ① such that the distribution of walkers evolves to Pa after time T. The modified dynamics reshapes the distribution P be peaked around pa rather than Peg
To see how this implemented in diffusion model we convert the discrete random walker setup from Eq. (1) to continuous diffusion process by making the lattice spacing & small. Taylor expanding in and keeping the leading terms, we obtain the Fokker- Planck equation (see App. A
0pp(2,t) t) -0x(b4 +
p((x,
(5)
2
f o² (()),,
64 6x)) At -u
(6)
At
We restrict ourselves to drift terms by (x)) that are confining so that (x) x exp( 2b4//22) exists.
By explicit calculation of Eq. (3). ), can be shown that if a distribution pa subjected to Eq. 5 the
total entropy produced after time T is (cf. Eq. (39p and [9]
Stot = {-
T 0
dt Em(.e) [1ob log Pog Pen - O2 log log
(7)
2
3

---

where the expectation value is taken over the distribution p that interpolates Pa and Peq. The r.h.s. in Eq. is the KL divergence between path measures of two stochastic differential equations (SDEs),
dX, = (X,, - 0²0, ((X, t) + odBi,
(8a)
dX, = b+ bx, + adB,,
(8b)
upto boundary term that vanishes when T is large 12 Eqs. Sa) and (b) that correspond the transition kernels h, and respectively (cf. App. I. That is, if we reset the clock to t = 0 and
apply Eq. (Sa) for time T we can drive Peq back to pa along p. Bringing Eq. (8b) closer to Eq. (8a)
would concentrate ' around pa- In a diffusion model this can be done by changing Eq. (8b) to
dx,- (X,) + er Xx, t))dt odB,,
where (X),,,) is the output of a neural network trained minimize an equivalent of (cf. Eq. (90)
CEM := /0" T o²
dt 2
Kp [10, log Peq 0₂ log + eoll"
(10)
follows from Eq. 4 that, perfectly trained network stores at least the same amount of information as we would learn from observing Peq fluctuate to pa under Eq. (8b). In practice, training is not perfect, so the information absorbed by the network is not exactly Scot. as we discuss below. We define the ideal neural entropy as the information retained by the network under perfect training,
SNN == Stot
(11)
This discussion is reminiscent of Maxwell's demon, a famous thought experiment from physics (1 [2]. The crucial difference that the demon does not perform work on the system: it measures the state of the system to make decisions about closing doors or adjusting potentials [23] Diffusion models do expend work to reconstitute pa from Peq. through the modified drift term Eq. P The additional o² o²ea term reshapes the free energy landscape to make pa the most probable outcome (cf. App. C.2 But these models also measure and store state information from simulations of pa + Peq during training.
true Maxwell's demon would reverse diffusion by waiting for Peg to fluctuate into Pa, an event learns about by measurement, and switch up the potential to lock pa into place. This is an example of an 'information ratchet' [24] 25]. On the other hand, a diffusion model remembers pa in a manner closer how we store, say, image in memory. grayscale image of dimensions H W is sample from a uniform probability distribution over the hypercube [0, 255 HxW The information gained from observing any sample is logg(256) H H x W bytes. This is also the amount of information we need to specify to locate specific sample/image in the hypercube. In the same way. Stot is the information required to locate within the paths generated by Eq. (8b) set of paths that transport Peq + Pd-
4 Entropy matching
Having introduced the total entropy Stot in the context of random walkers on a lattice, we can generalize it to D-dimensional continuous diffusion process with little effort. We will make the drift and diffusion diffusion coefficients time-dependent, but keep the latter isotropic. That is, pa diffuses under
dY, = b+ sY,, (s)dBB.
(12)
where we have introduced a new time variable s:= T - t for the 'frwardd' evolution (see Fig. 6 Let Po be the result of evolving pa for time T with Eq. (12). The SDEs from Eq. 8 are updated to
dX, = - (X,, - -) -) - p(x,, t))dt + oT - t)dB dX, = (X,, t)dt o t)dB,,
(13a)
(13b)
There is no longer static equilibrium state since Eq. (3b) changes over time; if we start with po at time t = 0 and evolve under Eq. (1bb) we will obtain distribution different from po, which we denote as pb+. that depends on Po as well as Eq. (3b)). But is useful to define the quasi- guasi-invariant distribution, pét (I), which satisfies the homogeneous Fokker-Planch equation
1
[
2b+ di
0 V (b4 (x, T - t) + OT - v) = (0) =
Z exp
(14)
σ²

---

Intuitively, pld can be understood as the 'least informative state' at time t. is the distribution that would result we froze b, and o their values f and waited for the system to equilibrate. Therefore Pau depends only on the drift and diffusion coefficients at l and has no memory of the initial state. For example, if 6+ = (x t) and o = 1 the quasi-invariant state would be ptl x exp(-(x 2)²) 77. Thus, Piu is the natural generalization of Peg for time- -eneeedend dynamics. In this paper, we restrict ourselves to forward processes for which the drift and diffusion coefficients have the same dime-dependence, that is, b const. In that case, it can be shown that
0² - Ep Stot L( dt 2
log Peg ||
V loggp = DKL (palleeT) - -DEL (pollz
(15)
This relation is derived in App. B.3 It bears strong likeness to important result in thermodynamies called the Jarzynski equality [26]. specifically the form given in [27]. According to this relation, total entropy is the information gap between p and the maximally ignorant state at given T. Further discussion of the connection to thermodynamics is given in App. C.T Our main goal is to understand
the consequences of Eq. (15) to diffusion models.
(!) Replacing Pod in Eq. 15 with pb. turns it into an inequality,
Sent l
di
E
log (t) V Vlogz
2 Daz (pallpoo)
(16)
This is slight variation of Theorem I from (0 A detailed proof is given in App. B.2 If we modify the drift term in Eq. 13b) to by + eo as we did in Eq. (9) Eq. (16) changes to (cf. Eq. (47)
J," o² Ep
T5z di 2
V log logged v Vlgg eo eol
2 DEL (pa()))po(s
(17)
The l.h.s. is the training objective, LEM, the minimization of which can now be seen as tightening the KL divergence between true and the reconstructed distribution pa. We call this the entropy- matching objective. is nearly the same as the flow-matching objective from (28) except for the factors multiplying the expectation value.
The neural entropy defined in Eq. (II is not always the true measure of the information stored in the network. This is often beneficial; if the neural network stored Scot perfectly for relatively sparse dataset, such images, the diffusion model would learn reconstruct series of Dirac delta functions in pixel space. We propose that the actual value of neural entropy is estimated by
SNN = /"
2 Ep
[Ieol
ds 2
(18)
The time integral has been expressed in terms of here because entropy is produced in the s-direction. Practically Eq. (18) is computed by simulating the forward process Eq. (12) and taking the Monte
Carlo average (cf. Eq. (23) So the expectation still taken with respect the ideal reverse evolution.
relation analogous to Eq. can be derived for score-matching diffusion models by switching the drift term in Eq. 13b to b+ o²so (cf. Eq. (46).
di
[1s0
V Vlogpl? 2 DRL (pa())lpo(: T)).
(19)
However, setting 8g = on the lh..s. does not give us a term that can be interpreted sensibly as an entropy. For example, if we consider the special case where b4, a are time-independent and choose Pa = Peq. we see that Stot vanishes and no information would be stored in the neural network in an entropy-matching model. However, E V log pl
(| A since the score function non-zero over the support of Peq. so the network ends up having to store 'information' to convert Peg to itself! Comparing Eqs. (17) and (19) we see that setting 8g = log p(t + eg makes both approaches equivalent, in principle. However, the score-matching network must put additional effort into learning the quasi-invariant distribution, which complicates the interpretation of score-matching loss as an entropy. See App. for further discussion.
5

---

5 Thermodynamic uncertainty
Returning for a moment to the random walkers on a discrete lattice, is apparent that the walkers are less likely to fluctuate into pa that is far different from Peq. compared to one that is more similar to Peg. This is manifest from Eqs. (2) and (4): a larger KL between pa and Peq: which is Stot, suppresses P(pa] further. In practice pa is often fixed by the training data and ptu changes as we adjust the drift and diffusion coefficients in the forward process, Eq. (12), to speed up the generative process. There is great interest in straightening the trajectories from the Probability Flow (PF) ODE [6] by clever choices of b_ and o, to enable few-shot sampling during the generative stage [29 However, such forward processes often produce more entropy, which means these models may inadvertently be placing a higher information load on the neural network.
As an illustrative example, consider the Straight Line Diffusion Model (SL) introduced in 31 The forward process is
dY, = 1- Y, + V 1- BndB.
(20)
At an intermediate time E (0, 7) (with T = 1), a sample ya ~ pa is propagated to
y (1 - s)ya + Loo 1- (1- S)e,
(21)
where e ~ (0, ID). For small do and a fairly 'wide' Pa, the trajectories in Eq. (1 are nearly straight lines that land at yr of p). This resull of allowing the drift term to dominate the noise in Eq. (20). But that also makes pld x exp(-y²/oj) the KI. to pa and thereby Sune- Or, using the intuition from See. 2 decreasing the randomness in the diffusion process diminishes the chances of an automatic fluctuation into Pa- Using forward process with more noise would lower the entropy, but only to a certain extent. the a is too large Ply becomes too wide compared to Pa and Stot rises again.
very narrow Gaussian, which increases
The above discussion is meant to highlight that Stor depends on the forward diffusion process and pa in a non-trivial way, and that there might be an optimal process that produces the least entropy for a given pa- This intuition is made more precise by the thermodynamic uncertainty relation, which relates the total entropy produced to the 22-Wasserstein distance between pa and Pu 32 33
Stot TT W2 (ps.
(22)
Here a is assumed to be constant for simplicity and TT is the time it takes for Pa to reach po- measured in units of The W/2 depends only on the initial and final distributions. If two processes take the same time to equilibrate (reach po R Peq), the one whose equilibrium state is farther from Pa will generate more entropy. If two process transform pa to the same po. the W22 is the same in both cases, but the faster transformation will produce more entropy to satisfy the bound. Therefore a diffusion model must store more information reverse a faster diffusion process. This is the thermodynamic speed limit: given Pa and po. there is an upper limit to how fast we can diffuse one to the other without exceeding a specific entropy production budget. Equivalently, a faster transformation requires greater amount of information to reverse, which has been found to affect accuracy 34]. These observations are also confirmed in our experiments.
Experiments
Neural entropy, as defined in Eq. (T quantifies the information presented to the network in an idealized setting. In practice, the finiteness of the data, imperfections in training, and strong inductive biases of the network all affect the amount of information stored in the neural network. To address these points we will perform two broad classes of experiments, first to probe the transport properties of diffusion discussed in Eq. 22 and second, to study the storage efficiency of diffusion models.
Transport experiments We work with synthetic datasets sampled from simple multivariate distri- butions for which we have closed-form expressions for both pa and V log p (e.g. Gaussian mixtures). This allows us to produce many samples as we require with high fidelity, compute their exact log densities, and work in arbitrary dimensions. Recall from Fq. (17) that the loss function upper bounds the KL divergence between the data distribution and the generated distribution, D (pa (-)lppe T)).

---

Le3 Entropy prod. rate (VPx)
103
Total entropy (VPk)
Entropy prod. rite (SL)
Total eutropy (SL)
ONN
720
0.5
Mey 200
02
0.2
0.2
0.6
1.0 0.0
0.5
0.0
0.2
1.0
0.2
0.5
0.
Figure 2: Entropy production rate and total entropy as Pa is diffused to Po by the VPx and SL processes from Eq. (24) and Eq, (20) respectively. The dashed lines are the ideal curves for Suot and Stot, while the solid lines are SNN and SNN at the end of the nep-th training epoch.
which means this KL can be used assess the performance of the diffusion model-a smaller KL implies the model reproduces Pa more faithfully. For any sample I we know pa (ax),, so all we need to compute the KI. is log po (z; T). The latter approximated by the method discussed in App. F.2 Finally, the transport experiments will be carried out with diffusion models with a multi-layer percep- tron (MLP) core. Since the inductive biases in such fully connected networks are weak [35], these models enable us to isolate the effects of varying levels of neural entropy on the model performance.
If pa is mixture of Gaussians it is possible compute Eq. (TT) explicitly, which gives the ideal value of neural entropy. However, due to the imperfections mentioned above the actual neural entropy given by Eq. (8). Both these expressions are computed by Monte Carlo averages, and it is useful to examine how they change over time E (0, For example, with some new samples Da pa.
(sw(s)=//
f(ds 0² Ep [|el|?] = Ps 2
[3)3) E Eys-r(sslaa)
[leo ]]]r]]
2
(23)
To explore the considerations raised in Sec. moree thoroughly, we experiment with a few different diffusion processes. We introduce minor generalization of the Variance Preserving (VP) process, given by the SDE
B(s) Y. ds + + 3(s)dB
dY, 2
(24)
which we shall henceforth refer to as VPx. For k this is the same as the VP process from 6 29 If we set k we obtain a process that has the same quasi-invariant distribution as
Eq.
However, the trajectories generated by Eq. 24 are
y. e Jr ((x)da Yd + HVV e 8(3)ds e,
(25)
which 'forgets' ya at an exponential rate as opposed to the linear evolution in Eq. 21 This difference borne out in their respective entropy profiles-phe plots of the entropy production rate Stnt = d.S..e/ /ds, and the total entropy Stot. over time. These are the dashed curves Fig. 2 The final value of Stot very nearly the same for both processes when ς do since they have a common pfP (cf. Eq. (16)). with small discrepancies arising from differences in their respective po. The solid lines in Fig. 2
SNN and SxN, evaluated at various stages of training. As train over more epochs, Rep- the network absorbs more information, bringing SNN closer to its ideal value Stot-
The experiments in Fig. 2 were carried out a model trained N 8192 samples from mixture of five Gaussians in D 6 dimensions, which Pa- The Gaussian components were N(rr, , D). where the means H,- were randomly chosen from D-hypercube of side length 4 centered at the origin. We used / = dg = 0.1, so both processes transform Pa to Po Ry N/ 10 10-211 p) at = 1 (cf. Eq. (40)). The SL process has singularity at T exactly (cf. Eq. 20 this divergence manifests in the SNN curve for SL the early stages of training.
In both VPx and SI. dynamies, the combination of weak noise with relatively strong drift subdues the randomness in the diffusion process, leading to larger Stor (cf. Sec. 5). Equivalently, the pa used above
7

---

lc2
lc2
1c2
5
R.
R. 100
20
Cross-entropy
Rg 1030
(
R. 3000
Loss
R. 6000
-5
-10
200
400
600
200
400
600
200
400
600
Pep
nep
Figure 3:
The evolution of neural entropy, cross-entropy, and loss over training epochs for an unconditional image diffusion model (VP) trained on the MNIST dataset. The different colors correspond to models trained on ne number of samples per class, ne 6000 means the model was trained on the entire dataset. The growth in neural entropy with the number of samples is nearly logarithmic. The values of (T') at the end of training are shown Fig. 16
is too far from their equilibrium state so a larger amount of information must be supplied to transform
Pog RJ Po to Pa- On the other hand, we use a regular VP process, for which Peg = NO, 1D), "distance" to pa is smaller and so is the total entropy. As result, the VP model trains much faster and produces more accurate reconstruction of pa than the VPx or SL. model. This is shown in Fig. TO
Storage experiments We carry out similar experiments on simple image diffusion model with a U-net core, trained on the MNIST dataset without class conditioning [36). In this instance, the training dataset is small relative the dimensionality of pixel space. Therefore the model relies the inductive biases the network to generalize from the given data points rather than memorize them [12] Entropy curves for this dataset also show sharp peak in entropy production (see Fig. 12). This is because the images live on a manifold MM of much lower dimensionality compared to the ambient 784-dimensional space. Seen from the t-direction, the sharp rise in entropy as + tells us that the diffusion model needs to inject a lot information in the final few time steps to locate the sample precisely on Ma. Similar plots for the SL model are given in App.
Fig. 3 we train image model on the first nl, samples from each class, and measure SNN and the cross entropy Eg | log pe] as training progresses. We cannot compute the true Stot or KL since the exact log densities are not known. Nonetheless, we see a similar result as before: the neural entropy and cross-entropy saturates after the model trains for while. Importantly, the model absorbs more information if is presented with a larger number samples, but the growth in SNN with N is not linear, it appears to be logarithmic (see Figs. and 16
10n, samples total,
3.6
We obtain the same behavior from a diffusion model trained on U-net with self- attention layers [5] 38 and apply class conditioning. The image
the CIFAR-10 dataset [37]. This time
quality improves substantially with N but neural entropy scales nearly log N (see Fig. 18). Notably, such trend is absent from the Gaussian mixture experiments performed on an MLP-based diffu- sion model (see Figs. 15). At low N these models concentrate the probability mass around the sparse data points and learn a very different distribution from the true pa- This is why SuN is larger small N: it could also be larger than Stot produced by diffusion of the actual Pa- On the other hand, the combination of structured data and well-matched inductive biases the image models saves us from overfitting even when data density is low, while also com- pressing the details from additional samples efficiently.
2.8
N
10
Figure Neural entropy vs. number of samples for dif- fusion model with an ML. trained Gaussian mixtures.

---

7 Conclusion
In Sec. I we used the example of storing images to motivate neural entropy. We presented two scenarios: the storage of individual images versus the retention of a distribution of images. In a conventional memory the number of bytes required to store N images will be N times the bytes per image, even with efficient compression. Classical compression algorithms make implicit assumptions about the statistical properties of natural images and apply them universally to all inputs. However diffusion model sees a large ensemble of images and can learn non-local correlations between the pixels of similar groups of images; they are infinitely deep autoencoders [39,41] In other words, these models can tailor a compression scheme specific to the properties of the data distribution. This a lossy form of compression of course, since we rarely recover the training images perfectly from these probabilistic models.
How about other generative models? At a conceptual level, Schrödinger's argument can be extended beyond just diffusion. For instance, a very simple-minde approach to creating a sentence of length L would be to choose L words random from corpus. The vast majority of sentences produced by this process would be utterly meaningless, but once in a while, we get a rare fluctuation that counts as a sensible statement. The collection of all such sentences constitutes in a sea of non-sense Peq. Here Peq can be a uniform distribution over all L-word combinations, say. P is the distribution of all distributions on these combinations. The information needed to transform Peq + Pa is proportional to the negative logarithm of the probability of fluctuating into pa automatically. A mechanism that enhances this probability to near certainty would need store that much information to effect the transformation. If we start with a Peq that assigns greater probability to more frequently used words it would be easier, albeit still very improbable, to auto-fluctuate to pa- This is manifestation of thermodynamic uncertainty, for words.
island of meaning pa.
Diffusion- based LLMs 42 4
offer the most direct path to defining neural entropy in the context of language. They are based on a discrete diffusion process very similar to the lattice random walks considered in Secs. 2
and 3 The ideal neural entropy in that case should have form similar to Eq. (37, with transitions that extend beyond nearest neighbor jumps. The discrete analog of the thermodynamic uncertainty relation, Eq. (22), is discussed in [33]. It would be interesting to investigate how the choice of the forward process in diffusion LLMs affects the training efficiency
and model performance. More work will be needed to extend the entropic picture to transformer
models, but investigations into the compressive abilities of such models are underway
Limitations Our definition of neural entropy is limited to continuous diffusion models at present. In App. [[we also point out the difficulties in defining neural entropy with score-matching models. With respect to the experimental results, the logarithmic growth of neural entropy with N is stated as an empirical observation with little explanation. A deeper investigation of this phenomenon is relegated to future work. In particular, it would be interesting to check whether this trend is repeated in more sophisticated network architectures like diffusion transformers [45). or if there is a connection to the scaling laws [66.. As noted in Figs. 12 and 17 the calculation of neural entropy in the image models requires some care due to divergent entropy production near = O. This behooves us to investigate the neural entropy in diffusion processes with momentum components which soften such singular
behavior 47 48
Related work The original work that introduced diffusion models took inspiration from the Jarzynski equality and fluctuation theorem from non-equilibrium thermodynamics [ 49 50). The relation between these ideas becomes apparent through [40] 7], both of which use the Feynman- Kac formula and Girsanov's i theorem to develop similar results separately for diffusion models and thermodynamics. These results can also be understood in the language of stochastic optimal control [5] 51). We illustrate both approaches in App. B.4 Several authors have also developed the connection between these models and the Schrödinger's 's bridge problem (3). The consequences of the thermodynamic speed limit on diffusion model accuracy are studied in greater detail in [34]. ceformmttion-theoree aspects of diffusion models are also explored their ability to capture mutual information is given in (55 It has also been noted that diffusion models
[54) and a deeper discussion of
are a form of energy-based memory [56), which concurs with the discussion in Sec. and App. C.2
since the generative process in these models is descent along learned free energy landscape.

---

Acknowledgments and Disclosure of Funding
We thank Lorenzo Orecchia, William Coturell, Austin Joyce, Maurice Weiler, and Ryan Robinett for valuable discussions, and Yuji Hirono for bringing [34 to our attention. We are especially grateful to William Cottrell and Lorenzo Orecchia for their comments on an earlier version of this paper. The author was supported in part by the Kavli Institute for Cosmological Physics at the University of Chicago through an endowment from the Kavli Foundation. Computational resources for this project were made available through the Ac+Snience research funding from the Data Science Institute at the University of Chicago. Code available https://gitubb.com/akiiii I/NeuralEntropy
10

---

References
[1] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Machine Learning Reseurch, pages 2236-2265, Lille, France, 07-09 Jul 2015. httptntttttttooeeeeeeeeeeeeeeeee https:////ccceeigeeee mlr oit
PMLR. URLhttpss
html 7
23
[2] C. Shannon. A mathematical theory communication. The Bell System Technical Journal, 7733):3794223 1948. doi: 10.
[3] E. T. Jaynes. Information theory and statistical mechanics. Phys. Rev. 106:620-630, May 1957. doi: 10.1 101110/y/ee..166.000 URL
620
26 27
[4] E. T. Jaynes. Gibbs vs Boltzmann Entropies. American Journal of Physics, 33(5):391-398 05 1965.
ISSN 0002-9505. doi: 10.1119/1.1911577, URL http:t/tss://oo org/10. wttg//.111/// 1119/1
1971557
I 26
[5] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffuusio probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12. 2020, virtual, 2020. URL https://prroeeddnggs neurips cc/ http/ppapr2222//aaeeeee 4c5bcf (aabbeeecceaaff0009f1/1011101111 http:/999ab---bbtaaaccao Abstract.
[6] Yang Song, Jascha
Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th In- ternational Conference on Learning Reprexentations, ICLR 2021, Virtual Event, Austria, May
7. 2. ReenReview.net, 2021. URL.htttps://ppereeiiee wttp:eewwwwwweeefffeeeeee hete/forum?id-PPIIG12RR
I 6
22
28
32
35
[7] Hao Ge and Da-Quan Jiang. Generalized jarzynski's equality of inhomogeneous multi- dimensional diffusion processes. Journal Statistical Physics, 131(4)6675-689, 2008. doi: 10. 10000//19955000--55200000 URL https://doi. ortpa/..10/00100
ISSN 1572-9613.
810955-008-98 95200444999-----
22
[8] Raphaël Chetrite and Krzysztof Gawędzki. Fluctuation relations for diffusion processes. Communiccations in Mathematical Physics, 282(2):469-518, 2008. ISSN 1432-0916. doi:
10.1007/:002200000-00500 URL http://///00o..gg//00.0000002020
[9] Udo Seifert. Entropy production along a stochastic trajectory and an integral fluctuation theorem. Phys. Rev. Lent. 95:040602, 2005, doi: 10. 10.1103//hssRewLett L5........000022 URL
httpss//iaakk......r///////.....
040602
26
[10] William Bialek, Curtis G. Callan, and Steven Strong. Field theories for learning probability distributions. Phys. Rev. Lent. 77:4693 Dec 1996. doi: 10. 10.110//hhsseeeLett.77....
URL https://iink .pt.app...oooooooo http://///........////eeeeeeeeee
4693 2
[11] Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4697-4700 Curran Associates, Inc., 2020. URL http::ttt////oceediiige wewpiinnnsnnnuuiiipe neur ipur.ps..r cc cc/ htcppaper_fiee____ fiwess/pperrrr pdf
[12] Zahra Kadkhodaie, Florentin Guth, Eero Simoncelli, and Stéphane Mallat. Generaliza- tion in diffusion models arises from geometry-adaptive harmonic representation. CoRR, abs/2310.02557,, 2023. doi: 10.44550//AXX112211002555 URL. hhttps://doi. http///oi.oo////////////////////
48550/ 18550/arXiv arrii 2310. 02557

---

[13] Raphaël Chetrite, Paolo Muratore- GiniooreGi,annschci,, and Kay Schwieger. E. Schrödinger's 1931 paper "on the reversal of the laws of nature" ["über die umkehrung der naturgesetze" sitzungs- berichte der preussischen akademie der wissenschaften, ahysikallich-mmathmatiich klasse, n9 144-153]. The European Physical Journal H, (621)) Nov 2021. ISSN 2102- org/ 10. 11j0/epjh/
6467. doi: 10. 8 118129-021-00033
3
[14] Akhil Premkumar. Generative difflusion from an action principle, 2023. URL https://arxil org/abs/2310.000 04490
[15] Michele Pavon. Stochastic control and nonequilibrium thermodynamical systems. Applied Mathematics and Optimization, 19(1):187-20)
1989.
doi:
10. 10.100///00448 URL
22 23
25
26
[16] Udo Seifert. Stochastic thermodynamics. fluctuation theorems and molecular machines. Reports on Progress in Physics, 75(12):126001, nov 2012. doi: 10. 00.1080000444488850
440000444444/5//5//22000000 URL
https://ds doi
10. /ttp:///.....00/003444//////////
20
[17] Edgar Roldán and Juan M. R. Parrondo. Estimating dissipation from single stationary trajectories. Phys. Rev. Letr. 105:150607, Oct 2010. doi: 10.1 Los/PhysRevLett.... 1t.. URL https
//link. aps org/ doid wtt.//......///aaaa...ee. www.////y///eeett...ooo 105. 150607
[18] David C. MacKay. Information Theory, Inference Learning Algorithms. Cambridge University Press, USA, 2002. ISBN 0551162981.. 3
[19] Kim Sharp and Franz Matschinsky. Translation of Ludwig Boltzmann's Paper "On the Re- lationship between the Second Fundamental Theorem of the Mechanical Theory of Heat and Probability Calculations Regarding the Conditions for Thermal Equilibrium". En- tropy, 17(4):1971-2000 2015. ISSN 1099-4300. doi: 00339///11041971 URL https //ww:./dww.ccommm/o/////
Kaiserlichen Akademie der Wissenschaften. Originally published Sitzungsberichte der Secimaariiih-aaaawwisss.s Classe. Abt. 11, vol. 76, pp. 37333333333 (Wien, 1877); reprinted in Wiss. Abhandlungen, Vol. II, reprint 42, pp. 164-223,
Barth, Leipzig, 1909. S
[20] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likeli- hood training of score-based diffusion models. In M. Ranzato, A. Beygelzimer, Y. Dauphin. P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural In- formation Processing Systems, volume 34, pages 1415-1428. Curran Associates, Inc..
2021. URL https://proceeeee httpeeedddiiiinnnnnnseeeeeeeeeee newrips. cc/ paper Sapper_fiie fww.s/ille/pelle ppppp/pp22////llee/ / Oafffbbf7fadccceeeee666666222222
pdf
21 22 37
[21] J. C. Maxwell. Ii. illustrations of the dynamical theory of gases. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 20 20(130):21-377 1860. doi: 10.1080/ 14786446008642902.
https://:://d www.oo////...... 1080/ 1000004478644000664222233
[22] Koji Maruyama, Franco Nori, and Vlatko Vedral. Colloquium: The physics of maxwell's demon and information. Rev. Mod. Phys., 81:1 Jan 2009. doi: 10. (0.1103/eewoodPyy..... URL
https:///ise aps org httg://o/1100 1100 w11.////M//PP.......oe 81.
[23] J M R Parrondo, C Van den Broeck, and R Kawai. Entropy production and the arrow of time.
New Journal of Physics, 1(77)0073088, jul 2009, doi: 10. 101110///177-263111111111111 1/7/07308. URL https://dx.c doi http:///0000000/06626000/11/1010
[24] Takahiro Sagawa and Masahito Ueda. Generalized jarzynski equality under nonequilibrium feedback control. Phys. Rev. Lett, 104:090602, Mar 2010. doi: 1..100//hyseeeLett... 104. 090602. URL httph:////iin aps www.ooor//ooooo 1103/PhysseeLettt 1... 090602
4
[25] Shoichi Toyabe, Takahiro Sagawa, Masahito Ueda, Eiro Muneyuki, and Masaki Sano. Ex-
perimental demonstration of informaation-to-eeergy conversion and validation of the gener- alized jarzynski equality. Nature Physics, 6 6(12):988 992, 2010. ISSN 1745-2481. doi:
10. 10.1038/nphys182) URL htttp:///ai..orr/1111033333
org/ 10tg//1...0///p/s11112.2
4
12

---

[26] Christopher Jarzynski. Rare events and the convergence of exponentially averaged work values. Phys. Rev. E. 73:046105, Apr 2006. doi: 10.1 10.110//hyyseee..3..6666//. URL https
//link. aps wwtg/oo////eooo.ooo 1ttp:///....100//yyeee.... 73. 046105
[27] S. Vaikuntanathan and C. Jarzynski. Dissipation and lag in inreversible processes. Europhysics Letters, 87(6):60005, oct 2009. doi: 10. 120000295-50555777600000 URL https://dx. org/10. 120./ 000000099--555////
5 26
[28] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. CoRR, abs/2210.027477 2022. doi: 10448550/ARXIV.2110 02747. URL httpp:://// org/ 10. 48550/arXiv......// 02747
[29] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the de- sign space of diffusion-based generative models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural In- formation Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 Decem- http::/paperrr nips. wwtpcccpppppppeeeeeeeeeeeeeeeeee silerrffilee/__eee hwwp:/appe///// httll///ppper/22/////aaeee Abstract----ttttt--eeeaa Coife--Cooeeeeeo html
ber 9, 2022, 2022. URL
30
[30] Nikita Kornilov, Alexander V. Gasnikov, and Alexander Korotin. Optimal flow matching: Learning straight trajectories just one step. CoRR, abs/2403.13117, 2024. doi: 10.48550/
ARXIV.2403.1111 13117. URL http::ttt/////// htw.doi...ggg/e/oooo
org /0/ 485500/arXii 2403. 13117
[31] Yuyan Ni, Shikun Feng, Haohan Chi, Bowen Zheng, Huan ang Gao, Wei-Ying Ma, Zhi-Ming Ma, and Yanyan Lan. Straight-line diffusion model for efficient 3d molecular generation, 2025.
URL httph:://ttttttii http:/awwwww..ooraaaa//a abs///ab///// 2503. 02918
35
[32] Jean-David Benamou and Yann Brenier. A computational fluid mechanics solution to the moong--aantrovieer mass transfer problem. Numerische Mathematik, 84(3):375-395 Jan
2000. ISSN 0945-3245. doi: 10.1000000021005000000 URL http:::////:oo.... 0r0/10..000000000000..0..001...0
8002110050002 6
[33] Tan Van Vu and Keiji Saito. Thermodynamic unification of optimal transport: Thermodynamic uncertainty relation, minimum dissipation,, and thermodynamic speed limits. Phys. Rev. X, 13: 011013, Feb 2023. doi: 10. 101.100hss.....11.... 13.01 http::/t/://// aps. .pw.org.//...................... org/doi/ 10. N0.1103//hhysev2 13. 011013 6
[34] Kotaro Ikeda, Tomoya Uda, Daisuke Okanohara, and Sosuke Ito. Speed-accuracy trade-off for the diffusion models: Wisdom from nonequilibrium thermodynamics and optimal transport, 2024. httphttss:/// //t::////xxv horg/abs/2407 00499,
[35] Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinícius Flo- Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner. Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew M. Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, 0as//8806.02611
2018. URL htRLhttp::// //axivv../iii//////////i/ org/ abs 1806 01261
[36] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(11):2278-25 1998. doi: 10.1109/5.72679.1
[37] Alex Krizhevsky. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009. https://www. cs. torontoroooo...... edu/"kriz/
Learning-feature 2009 TR. .df.
[38] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models.
CoRR, abs/2202.00512 2022. URL httpst//:///rtt www.:org/aaa... ha0ts//2200000 00512 g
13

---

[39] Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neu- ral networks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2018, Sals Lake City, UT, USA, June 18-22, 2018, pages 7794-7803 Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1109/CVPR,200 20.200018000000 8.00813 13. URL http::ttcc//ooeaccesssssss
www.secccsssss.s..e..eeeeee thecvf. com/ content_ cttpttt_cp____111111ee Beurrl_leetwork__CPPR. 2018_ _apperrppppp.._ html 9
Non-
[40] Chin-Wei Huang, Jae Hyun Lim, and Aaron C Courville. A variational perspective on diffusion-base based generative models and score matching. M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan, editors, Advances in Neu- ral Information Processing Systems, volume 34, pages 22863-2222 22876. Curran Associates, Inc., 2021. wtt.:R//tttttt.pteeeeeeeeeiiiiie c1 c1a5t2224484441556611111666688
neurips. cc. cc/paper.... salee//paper/22///iieee
pdf 9 22
31
[41] David McAllester. On the mathematics of diffusion models, 2023. URL https:// /arxiv
org/ /
[42] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-firsi International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL. https:// openreview wwtp://wwwtotodtdtiom...ommm
[43] Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. Energy-based diffusion language models for text generation. CoRR, abs/2410.21357, 2024. doi: 10.44550/ARXXI22240..13377 URL htttps://di..rr:::
48550/ arrvv 2410. 21357
[44] Ziguang Li, Chao Huang, Xuliang Wang, Haibo Hu, Cole Wyeth, Dongbo Bu, Quan Yu, Wen Gao, Xingwu Liu, and Ming Li. Lossless data compression by large models. Nature Machine Intelligence, May 2025. ISSN 2522-5839. doi: 0000:/444226005--030333 URL httpss://oi.. org/ /000////22556.. 025 01033-77
[45] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pages 4195-4205, October 2023. 9
[46] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, 000/2000108361, 2020. URI https:/t:://arrtt wwwpaawwww..../////////0/0/ 2001. 08361
[47] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically-damped langevin diffusion. In International Conference on Learning Representations (ICLR), 2022. 9
[48] Alexandra Lamtyugina, Agnish Kumar Behera, Aditya Nandy, Carlos Floyd, and Suriya- narayanan Vaikuntanathan. Score-based generative diffusion with "active" correlated noise sources. CoRR, abs/2411.072233 2024. doi: 1044550ARXXI URL https
//doi org.iorr11...0 400.0.8855////// ww1.:.awwww...1......... 07233
9
[49] C. J.rayyyssii Equilibrium free-energy differences from nonequilibrium measurements: A master- equation approach. Phys. Rev. E, 56:5018 5035, Nov 1997. doi: 10.1 105/PhyyeevE..56.0000
URL httth:ttt//:kkk otg. wtarg//ooi/00 http://..1111//hhhhoeeee 5011
9 26
[50] Gavin E. Crooks. Nonequilibrium measurements of free energy differences for microscopi- cally reversible markovian systems. Journal of Statistical Physics, 9065):1481-1487, 1998. ISSN 1572-9613, doi: 10. 10.1023/A:10223082119999 URL 1023208217925
g
[51] Julius Berner, Lorenz Richter, and Karen Ullrich. An optimal control perspective on diffusion- based generative modeling. Transactions on Machine Learning Research, 2024. ISSN 2835- 8856. httttp:://pppmreeeee.eeee//e//ee
14

---

[52] Yuyang Shi, Valentin De Bortoli, Andrew Campbell, and Arnaud Doucet. Diffusion schrödinger bridge matching. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages Curran Associates, Inc., 2023. URL
newr cc/ wwwpppppppppppppppppeccp saaaerrflle________ files/ Conference .d
[53] Ludwig Winkler, Cesar Ojeda, and Manfred Opper. A score-based approach for training schrödinger bridges for data modelling. Entropy, 25(2):316, 2023. ISSN 1099-4300. doi:
10.3390/e25020316 BtLhhttps://www mmpi. 2ttp://099944002///////////
9
[54] Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information-theoretie diffusion. In
Proceedings of the Eleventh International Conference on Learning Representations (ICLR 2023), 2023. URL /ptp:////epeeeeeeeeee/mem htt.://oooo//mm/i//oooaooaaa
Preprint athttps
7/arxiv. http://orgggbbbb//aaeee abs/ 03792
[55] Giulio Franzese. Mustapha Bounoua, and Pictro Michiardi. MINDE: Mutual information neural diffusion estimation. Proceedings of the International Conference on Learning Representa- tions (ICLR), pages 16685- 2024. URL https: httpps://proceeddingg iclr www.er.cccpppppeee cc/paper
files/ #11p::/722201111rr00000000222222
pdf
[56] Benjamin Hoover, Hendrik Strobelt, Dmitry Krotov, Judy Hoffmman, Zsolt Kira, and Duen Horng Chau. Memory in plain sight: A survey of the uncanny resemblances between diffusion models and associative memories. CoRR. abs/2309. 16750. 2023. doi: 1044550/ARXVV2300.6675 org/10. 48540/ URL (RLLhhtp::/daae htt.://...........////////////// arXiv 2309 16750
[57] Kunio Yasue. simple derivation of the onsager-machlup formula for one-dimensional nonlinear diffusion process. Journal of Mathematical Physics, 19(8)1677-1163, 08 1978. ISSN 0022-2488. doi: 10. 10.1063/1,523885 URL httpss:/dooi
1063/1. 523888
[58] Edward Nelson. Derivation of the schrödinger equation from newtonian mechanics. Phys. Rev. 100:1079-108: Oct 1966. doi: 10.1 http/Physee...0.10000 URL httpssttts//1in aps org/
doi/100 11w../hys3/Peeee....mmmm. 150. 1079
21
[59] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):31--326, 1982 ISSN 0304-4149. doi: hutps://doi.or//0..000
03044-414902290005155. URL http://wwwccceeoccccccc.c/cc//cs p:1/03004149829005188
[60] U. G. Haussmann and E. Pardoux. Time Reversal of Diffusions. The Annals of Probability, 14 (4):1188 1205, 1986. doi: 10. 10.1214/aop/1 176992362. http::htts://oo..oom 1214/ aop/1176992362
[61] Hans Föllmer. Random fields and diffusion processes. In Paul-Louis Hennequin, editor, École d' á'éíé de Probabilités de Saint-Flour XV-XVII, 1985-87, pages 101-203, Berlin, Heidelberg, 1988. Springer Berlin Heidelberg. ISBN 777-3-5440-46042-. 21
[62] Sosuke Ito. Geometric thermodynamics for the fokker-planck equation: stochastic ther- modynamic links between information geometry and optimal transport. Information Ge- ometry, (1)):4414883 2024. ISSN 2511 doi: (1007//41884-003-0000 URL 10w000000/411
https:///oi.org:: wtt.://////0..111111188000000000
22
[63] M. Kac. On distributions of certain wiener functionals. Transactions of the American Math- ematical Sociery, 65(1):1-13, 1949. ISSN 00029947, 10886850. URL httttp://www.. jstor
org/stable/19905//
[64] Wendell Fleming and Raymond Rishel. Deterministic and Stochastic Optimal Control. Stochas- tic Modelling and Applied Probability. Springer New York, NY, edition, 1975. ISBN
978-0-387-90155-8. doi: 1-1779978-1-411-6888---- URL, http:htttt//o....oo. org/
978-
25
15

---

[65] M.I. Kamien and N.L. Schwartz. Dynamic Optimization, Second Edition: The Calculus of Variations and Optimal Control Economics and Management. Dover Books on Mathematics. Dover Publications, 2013. ISBN 9780486310282. URL https://books:. googll com/
books?1.e=11LCAg44BBBJ
25
[66] Ludwig Boltzmann. Further Studies on the Thermal Equilibrium of Gas Molecules, pages
262 349. Imperial College Press, 2003. doi: 10. 10.1142/97811441111133 10000000106133001101510 26
[67] Gavin E. Crooks and Christopher Jarzynski. Work distribution for the adiabatic compression of dilute and interacting classical gas. Phys. Rev. E. 75:021116, Feb 2007. doi: 1033 75.021116. https://:://11 wtt.iinn.....rrr.........o.. 1tw.:////...../////eeeee... 75. 021116 26
[68] Christopher Jarzynski. Equalities and Inequalities: Irreversibilityy and the Second Law of Thermodynamics at the Nanoscale, pages 145-177 172. Springer Basel, Basel, 2013. ISBN 978- -0449-5..355.. doi: 10. 1001000077--00000 5 4. URL https://dol. org wo. 1007
978 3- 0348 0359- 5_4
26
[69] E. T. Jaynes. Information theory and statistical mechanics. ii Phys. Rev., 108:171-190, Oct 1957. doi: 10. 10.3PPhhyssev...o 108.171. URL httpp://linn aps www.app..........e.. org/ 1103/
PhysRev. 108. 171
26
[70] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecom- munications and Signal Processing). Wiley-Interscience, USA, 2006. ISBN 0471241954 26
[71] Maxwell Aifer, Samuel Duffield, Kaelan Donatella, Denis Melanson, Phoebe Klett, Zach Belateche, Gavin E. Crooks, Antonio J. Martinez, and Patrick J. Coles. Thermodynamic bayesian inference. CoRR, abs/2410.01193,, 2024. doi: 104850AARXXV:2410000033 URL Intpp:://oot http:///ooo..r//....oo 485.0/ arxiv. 2410. 01793
27
[72] Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the fokker- planck equation. SIAM Journal on Mathematical Analysis, 29(1):1-17, 1998. doi: 10.1137/ $00361410996303399
URL, htt.hhttt://aaee otg/i/..rg111.. 1ww.: 0110//11111111000011110000333333
27
[73] Mehran Kardar. Statistical Physics of Particles. Cambridge University Press, 2007.
28
[74] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Pridovich-Keil. Nithin Ragha- van, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier fea- tures let networks learn high frequency functions in low dimensional domains. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balean, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7537-7547. Curran Associates, Inc., 2020. URL httptttpppppppcceeeiiine nwwrips. cc/ cc/ppppr.... http//ppppreeeeee/e wiles/feee//eeee hwpppppppp//// /220e//020//E1le/ 550353366677977331166688888888
.ddf
30
[75] Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and I1-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning 11201-11228. PMLR, 2022. URL httpp::/
Research, pages 1 v162/kim22i html
mlr press/
32
[76] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23:1661-1674, 2011. https/tts::/app http::::/app..ssetttcccclooo www.nt .owllaar.rrrg//oo www.oog//oooo.ooo Corg/orrpssD
5560643
31
[77]
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 21696-21707 Curran Associates, Inc., 2021. URL 2021/f1le/b578f22 2011/111167772222028888442222222
neweiieeesipppp cc http:/papr_ffeeee_e files//ppe//////////
.df 35 37
[78]
Akhil Premkumar. Diffusion density estimators, 2024. URL https: : / /aaxivivv.......i www.wwwww..aaaaaaaa... www.o/g///bb/1111/
06986
35 37
16

---

[79] Dimitra Maoutsa, Sebastian Reich, and Manfred Opper. Interacting particle solutions of fokker-plance equations through gradient log density estimation. Entropy, 22(8), 2020. ISSN 1099-4300. doi: 10.3390/220800002. URL https: www http:wwwwwpp...pp/o//// http://coo//000030//////////////
35
[80] Sitan Chen, Sinho Chewi, Jemry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is easy learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023, bRtps::/://oope id=zyLVMgs20U
35
17

---

Contents
Introduction
2
Schrödinger's Gedankenexxerimentt
Diffusion models and Maxwell's demon
4
Entropy matching
4
Thermodynamic uncertainty
6
Experiments
Conclusion
References
11
Contents
18
A Random walk on lattice
19
A.1 Reversal
19
A.2 Entropy production
20
B Stochastic control
20
Notation
20
B.
fluctuation relation for diffusion models
B.
The //theorem
23
B.4
Derivation of the bound
24
Non-equilibrium thermodynamics
25
C.1
Dissipation, lag, and the information gap
25
Entropy and free energy
26
D Score matching
28
Details of experiments
30
Diffusion models
30
Density estimation
36
18

---

A Random walk on a lattice
Random walks on a discrete one- mne-iimensional lattice serve as simple toy model to understand many of the results in this paper. Consider M random walkers on a lattice spacing E, each of whom can occupy any of the sites of the walkers. every time step As every walker a jumps either left or right with probability qu (z. s) and qR (z, s) respectively, QL qR 1. Then, the probability of finding walker updates as
né, Z. We shall use the time variable for the forward evolution
p(z.s s As) qR( 6, s))p(( (, + qu(x + E, s))(( L,s),
(26)
since all walkers that were at x at time cleared out and have been replaced by incoming walkers from either the left or right. Taylor expanding Eq. (26) in small As and L,
L As
0,p(2,
(zgn(z, s) - qr s))p(r, 8)) - 2As
Ep(r,s),
(27)
which is the Fokker-Planck equation with drift and diffusion coefficients
by (z. 8) = - As
L
8
(9x(2, s) (z, 8)), o²
(28)
As
Conversely, given a Fokker- Planck equation with a generic drift by we can think as the small limit of lattice model with jump probabilities
2 + As bb (2, 8), (1(2, 8) =
As
(n( 8)
by (x),
(29)
2
If the diffusion coefficient is time and/or position dependent we can map it Eq. time and/or by a change of variables [57].
28 by rescaling
A.1 Reversal
Eq.
26 can be reversed by returning to x and x + respectively a fraction
qR(x= - (,8)p(z - (, (I, + As),
(30a)
p(r, As)
Or (x + s)p(x + e,
=:qn(z,s + As)
(30b)
p As)
of p(x, + As). It is useful to introduce new time variable / for the reverse direction, as shown in Fig. 5 Then, the site x receives OR fraction the contents from time t, and the distribution of walkers evolve
f, and iL of the walkers in x +
p(x t At) = (n( f, t)p(x t,t) + Ā1(x (I + e, f,t)((r + e, t),
(31)
In the small é limit, we can compute the new drift (cf. Eq. (41))
6_ (x, t) = At (āL (x, t) - (x(x,t))
(āL(
1
At (gn q1))
Or log + Og((nn q1)p) C(an -0000g0000 +...
Al
22
2.8
b s) s) 2) log p(z, s) + (
(32)
The diffusion coefficient remains unchanged at the same order,
@ At (in( (x, = (qn(x, q2(x, ()) ((9).
(33)
19

---

8 + As
t + At
T
Figure 5: Time variables in the forward (top) and reverse (bottom) directions.
A.2
Entropy production
Eq. (31) is not the only way to return to the original distribution of walkers, but is the strategy that corresponds to the optimal kernel h, in Eq. (3). In that expression, is the transition kernel for the forward dynamics Eq. (26). now applied the I-direction. Explicitly,
(Te+ (x) (n(#,, ()drl+ + (3t,, 1) g ( x1) q t)0e44a + (xt, t 1)0444m22--
(34a)
(34b)
Here ar. means Fit is the right of o, etc. We can compute Eq. random walker on lattice.
explicitly for the
(xr((0)) g (ro)
Dkl(h_llg) = Σ (xrl2o) log
(35)
Using the log sum inequality,
T-A Ar
pla,, (Tz+ae log h. (x((40(222)) =; Stot-
<
Σ
Σ Zetani
(36)
g(
1=0
Using Eq.
34
T-
Scot = Σ
Σ
qR log p(znt)
IR + qL log
I 2
(37)
QR
(=0
Substituting Eq.
30) and Taylor expanding,
Stot
R-1
Σ
plaL qR) log yr. + 10pplog
qr.
+ (g1 log(gL))) + qR( 1og(9и)))
l
QR
2
1=
Zt
t0, 100 (R)pp + SoP
As
2L
( O
QR
gr.
(38)
Expressing the jump probabilities in terms of b4 and o² (cf. Eq.
29 and using the Fokker-Planck
equation eliminate some terms, we obtain the final expression for total entropy [16]:
Scot = S'' dt 202 2r Ep || - o²d log
(39)
B Stochastic control
B.1 Notation
We use the time variable for the forward diffusion process, which runs from right (s = 0) to left (s T') in Fig. Sometimes indicate functions of f to remove ambiguity when the same function is also expressed in terms of time variable - That is, f(s) (T -1) f(t). For instance, p(x, () p(r, s). B, and B, denote the Brownian motions associated with the forward and reverse/controlled SDEs, respectively. V the gradient with respect the spatial coordinates, and 0g 0 are partial time derivatives. Stot is the total entropy produced during forward diffusion.
20

---

Sc is the non-equine couiliiiiirium Gibbs entropy of the distribution, and SNN is the neural entropy. We make the time-dependence of the entropies explicit later in the paper after we have introduced the variable; Stot and SNx without the time argument should be understood as Stot (s = 7) = Stot (T). Throughout the paper, we set Boltzmann's 's constant to unity, ÂB 1. log the natural logarithm. Pa and Po denote the initial (s = 0) and final (s T) densities for the forward process, and Peq is its equilibrium state. Diffusion takes an infinite time to equilibrate but we always take T to be large compared to the intrinsic time scale of the diffusion process, which is why we ignore the difference between po and Peq in Secs. 2 and 3 (u(:, 0) and (. 7') are the initial (t 0) and final (t = 7) densities of the controlled process. There is a slight abuse of notation here because (u 0) is distribution that does not depend on the control u, it is just the initial state to which the control is applied.
Assumptions We make the same assumptions given in App. A of 20], with the following additions for entropy-matching models:
1. 3C 04 R.,EE [0, T : eeo t)]l C(1 + |,]|2)
2. 3C Vv. y E CRR [0, T lleo (x,1) eoly, f) < Cl y/|2.
v logp co 11²] <00.
3. Novikov's condition: E, exp Jo' de 2/2 v log pea
B.2 A fluctuation relation for diffusson models
Given a set of data vectors, probabilistic models attempt to learn the underlying data distribution from which these vectors could have been sampled. One way to do this is to minimize the KL divergence between the data and the model distributions. Score-based diffusion models are trained by optimizing an objective that upper bounds this KL [20 40]. In this section, extend this bound to a more general parameterization the generative process.
Consider D-dimensional probability density function pa subjected to a diffusive process
dY, = b+ (Y,, s)ds (s)dB,,
(40)
The noise is isotropic and position-indeppendee independent. Under Eq. 440) the distribution pa(y) = p (y, 0)
evolves to some another distribution po(y)= Pwy T) (see Fig. 6 This process can be reversed by an SDE [1866]
dX; =bb -X, T T t)dt + o(T t)dB where t = T - s. and the drift term is
(41)
s) := (X, o² (s) log P ().
(42)
Starting from po(x) = p(x, 0), the reverse evolution back to pa (x) = p(x, T) appears as a playback of the forward process in Eq. (40). so that p(x, 1) ((x, T - 1) an intermediate time Crucially, we need information about the forward process, specifically the score score function V log P, to construct the reverse drift term in Eq. (42). This makes sense: the final distribution po has little to no memory of the initial state Pa. meaning that many different pa could diffuse to roughly the same po, rendering the problem non non-invertible without explicit knowledge of the forward process.
If we replace b_ in Eq. (41) with a different drift term u. called the control, and evolve Po(x) by the stochastic process
dx, = -=--(XX, (X, t)dt + aT - f)dB, the density pa i) of X, will differ from p(a, t), and land on a terminal distribution pa( T) p(z, T). The KI. divergence between these distributions is bounded as
(43)
L. dt Boc.e) [16- u1(] 2 DEL (p(,)))ll.(.TT)))
(44)
More generally, if we start at some pu (z. 0) + po(z),
p dt 202 Ep [10_ u11] + DDL (pol) (1paa) 2 DxL (p(: (.)Pp((.
(45)
21

---

(x p(x, 0)
Pa(x) p(x, 7)
Forward SDE dY,=b_de+odB,
Reverse SDE
dX,= dtttodd
T
0
Figure 6: A schematic of the forward and reverse diffusion processes.
This result can be derived using the theory of stochastic optimal control 15
by application of the Feynman-Kac formula and Girsanov's 's theorem 40 The details are given in App. B.4 See also (62)
As particular example, we can choose u = b+ o²se. where 89 the output neural network, which converts the I.h.s. in Eq. (44) into the score-matching objective from 5 6 This leads to
Theorem from (0 (cf. Eq. (19).
[" 22 dt 2 Rop |100 - V Vlogrl + DEL (p(())lp((::0)) DEL (plo:)))
(46)
If we pick the initial distribution pe (æ( 0) to be close to po(z). and train neural network to minimize the score-matching term, we can tighten the KL divergence between the data distribution. p(x. T) pa),,, and the generated one, (2. T). This is how score-matching diffusion models work. Similarly, the parameterization Eq. (T).
b+ o²ea gives the entropy- objective (cf.
2 S dt Ep
26
V logpp e0
+ DKL (po())lle((:0)) DEL (p((,))lloo(, (47)
Lastly, if we set u = -b and choose Pz(z, 0) = po( for simplicity, we obtain lower bound on the total entropy (cf. Eq. (16))
Stot (7) = / di 2 Ep
26+ v log
Dxz. (p(, (o(:T)))pn (-T))
(48)
Next, we look the conditions for which this bound is saturated.
22

---

B.3 The H H-theorem
We derive Eq. (15) here, following [5]. states that p relaxes toward pr with the rate of approach slowing down as it nears that state. To prove it we start with the time derivative
d dt
DeL (llp
d
- -log p(x;()
Bp
dt
Pe (x)
d
(-f dr p(a, p(a,
d dt (-
t) log (og
(49)
dt
The first term the r.h.s. the Gibbs entropy production rate S(1) 9
5(t) - / dx dz O.plog
/ /d d.p
/ dæ (-V- (b-p)
0² V'p) logp+ + / dev((...)
2
(0²
dx p(., 1)
log + b_ V Vlogp)
(50)
The second term in Eq. 49 can be simplified by using the Fokker-Planck equations for and pº9,
a dt
(f d p(x, p( logp, (
= =fax dxx (Oap(z,,) () (o)g), (x) + p(E, ()
Oep (x)
(x)
Pt (x)(
=f dx (v- + p2p log Pt ea (x) + dx p(x, (-v - (b+1)) Pt (x)
+ 2 V2ppon)
Here, used the fact that 0,,, 0 when / const., and replaced that zero with Eq. 14p. We will consider each new term separately for clarity. Integrating by parts,
f dx ( (b-p) (v.
o =²v²p)
o² V'p log (x)
2
ʃ do dxp(xx)) 1)
(-b- V Vlogpt" 09
0² v logp- V Vlogr(
2
and
f dx p(x,t) t)
- - (b+p: 0q) + 0²
V2pp21 eq
Pt (Ir)
2
/ dx (6+ -p-b+p
Vp eg
0²
0² + p V log 11)
v
logp. V Flogp;" on
Pt
eq
2
=f dr t(z,t) (14. (V
0²
Vlogp. Vlogp, 0² V log 10gp;99??
v logp;")
Adding up everything, we obtain
DL (p(z, tx))
dt
= Ep
V logp(r, t) V log ". (x)1 (6_ +
(V (Vlog((z,)) bb)))((Vlgggleeeeeee V 2 pi (z))
2
-EE,
Ep
2/2
o V'Vlogg(r, 1)
(51)
20
where in the last step used Eq. 14) to replace V p. (z). Integrating Eq.
51 yields Eq. 15
23

---

B.4 Derivation of the bound
We present derivation of the bound in Eq. (44) drawing from the proofs in 15. We only outline the steps here, and refer the reader to those papers for more formal details. Consider the process specified by the SDE
dx2-
(52)
with the initial condition Xo po(-: The evolution of pz(x, t) under Eq. (52) is given by the Fokker-Planck equation
0²
E.p. + V. (rpb)
V2p p.. 0.
(53)
Switching the time variable equation for P
T (see Fig.
converts this into backward Kolmogorov
o² v P. 0,
a.p. - -V. vV. Vp,
(54)
with the terminal condition Po(:T)= =p)();))) P). The solution for Eq.
54 is given by the Feynman-
Kac formula 63
(o(x,))= =E Po(Yr. 1 exp(( [" ds V vYy,,T 8)),,
(55)
where Y3 is diffusion process that solves
dY, vYY,,T s)ds + (s)dB',
(56)
That is, Eq. (55) is path integral over all paths that start from time and evolve under Eq. (56).
Setting = O in Eq. (55) gives the likelihood Po(-. T) Po(:,)) 0). Next, [0) bounds the log likelihood by a change of measure and Jensen's ' inequality,
dPP aQ
L' d Yo
log T) 2 Eq
log Po (Yr, -
(57)
dB is the Radon-Nikodym derivative. For our purposes is enough to understand the expectation Here 4o
value of this object as
dP aQ
Yo fdyr dyr Q (:) log QUT (x)
E
-
(58)
P(yr(a) [x)
where P is the transition probability corresponding Eq. (6) and the transition probability for a new process
dY, w wY.,, s)ds + (s)d,,
(59)
Then, Eq. (58) can be simplified [14] allowing the r.h.s. in Eq. (57) be written the negative of cost functional (at =0)
J(z, w) := Ew
1 20
||u + w/l2 + V v. - log (Yr, = (60)
We have tinkered with the notation little, using En to indicate that the averages are taken over Eq. (59). Eqs. (57) and (60) will be used in two different ways below. We will set p.(:, 0) (-(-) in both cases for simplicity.
Case 1:
u. w b.. Then, Eq. (22) becomes the controlled process Eq. (43 and Eq. 559 is the forward diffusion from Eq. 40). and Eq. (60) is
log (x, T) > - (x, -u, b+).
(61)
By is reparameterization of B.. See Sec. 4 of 44

---

Case 2: == bb.. Under this choice Eq. 52 is the reverse diffusion process from Eq. 41 which takes Po + Pa via p. Then,
logp(2, T) -J (x,0; -
(62)
This inequality is saturated if we set b+ [5] To see this, we define the value function ((r, s): == min J (I, 8; -b, satisfies the Dynamic Programming equation 64 651.
w), which is the minimum cost over all admissible values of w.
o² 2 - O,W
min (- V.
20 bb_ w| VW
(63)
Pointwise minimization of the r.h.s. gives w, = b_ 0² VW. Substituting this back into Eq. 63 we that W solves
0, + VW VW o² |W
σ²
V o P VW V b..
(64)
with terminal value W (a. T = log (z). But notice that, for -b_, Eq.
54 can be written
following equation for $ log P.
0² |VS/|1 -
o² V b..
0.5 + b+ VS+
5
(65)
2
2
also with terminal value ((x T) = - log Po()). Comparing Eqs. (64 and 65
we see that
(x, - -log P (vr, 8), and Eq. (62 becomes
logp(ar T) = -- -w 0)
= -Ii,, {-
d8
V
log P 112 V V.b.).
log -10g)))og00(F) x
(66)
20
The bound the KI. divergence between p and yv. be obtained by integrating Eqs. (61 and (66)
over pd.
|b
1-
lib u11
d8 R.
(b_ u) 2 Dxz (P))
(67)
20
The last term in the average can be integrated by parts,
E. V (b_ w)] = R [0- u) V log
ES [(b 1) (b_ b+]],
(68)
to rewrite Eq. (67) in its final form
J ds 202 E [lb_ --l]] 2 DRL (p(; ()lppa(.
(69)
Note that (a) since p(,,) ((. 8) we can replace the average E + E, and change the time
integral to run over f, which gives Eq. (44), and (b) we would have an additional KI. term we had Po(+) # (.. in Eq. 667. the one from Eq. 45
C Non-equilibrium thermodynamies
C.1 Dissipation, lag, and the information gap
Eq. 15 is Jarzynski equality, applied Langevin dynamics. Assuming Po peq that ignore the KL between them, Eq. (15) is
(0)
Sio DKL Pa (elllz?7)
(70)
25

---

The I.h.s. is the total entropy produced as the distribution Pa is diffused to pu by Eq. (40) [9] As diffusion progresses in the s-direction, our knowledge of the system diminishes over time. Stot is a
measure of this information loss. Seen from the l-direction, Eq. 441 restores the information worn away in the forward process. This is the perspective put forward in the maikutannthaanJJaryysa (VJ) relation for irreversible processes [27].
W (iisa()) (1) - 3 ' (Drr.Pellle")) (
(71)
This equation can be understood by considering a system, initially at a temperature 8 driven away from equilibrium by varying external parameter X from A to B, over time interval & E [0. 7]. Let (W) be the average mechanical work needed to effect this transformation which, according to the Second Law, is at least equal to the free energy difference AF between A and B. Then, Waiss= = (W)- AF is the average work dissipated over the whole process. Pt. is the phase space density as the system evolves from A to B. and Pi is the equilibrium density corresponding to the value of the parameter at that instant, X That is, if we adjust the parameter to X and wait a long time, the system will evolve P. , its entropy increasing monotonically during the process. on This is Boltzmann's ' H H-thhoremm [66] [5) In other words, Pt is the maximum entropy (or minimum information) distribution consistent with X 3
On the other hand, under finite time non-equilibrimm evolution the system is rushed along to the state Ps and is not afforded the time to relax to the maximum entropy configuration. As result, lag develops between Pt and Pt as measured by the KL divergence in Eq. 711 Lag indicates the extent to which the system is out of equilibrium. The VJ relation, Eq. (7). says that the dissipated work dictates the maximum extent to which the equilibrium can be broken at given instant.
We can also interpret the lag as the information gap between Pr and Pe The entropy of a system is measure of missing information, with larger entropy associated with a greater degree of ignorance about the system's true state. Pt has higher entropy than Pt since much of the information in the latter is lost when Pt. equilibrates to Pt Intuitively, it is clear that the gap is precisely the amount of eq
information we need to exhume Pt from Pt
In the context of Eq. (7).. the non- equiliiiiuiim process the reverse diffusion from Eq. (41). The VJ relation forces shift in perspective. nudging us to think of Eq. (13b) as the 'native' dynamics of the system, with the process in Eq. (13)) driving away from its 'preferred state" peq The gap measures the information deficit that keeps the native dynamics from reaching pa on its own. This is also the lesson from Schrödinger's ss thought experiment (cf. Sec. ( We can close the gap by modifying the native dynamics to enhance the probability of the outcome Pa (cf. Eq. (9) The maximum additional information needed to do this Scot-
(t)
These arguments also serve to illustrate a specific point about the Second Law: it is a statement about the irreveriibiiiyy of a non-equilibrium transformation, even if that process is simulated on a computer. In the real world irreversibility is an observed property of almost all physical processes we take this for granted since we evolve with SDEs that are not time-symmetric by construction (cf. Eq. (40
Fl 688 69 but
C.2 Entropy and free energy
Shannon entropy can be understood as measure of ignorance. The analogous quantity we use for diffusive processes is the non-equilibrium Gibbs entropy [9]
f
Sa[2] : s dr (xx, s) log P (x
(72)
Sg can be understood as a continuous version of the Shannon entropy up to a multiplicative factor kn which we set to I (see footnote 15 in (3)). Gibbs entropy is measure of our uncertainty in the state of the system, which in this case are the locations of the diffusing particles. But we can choose the drift and diffusion coefficients such that the final distribution is narrower than the initial one
2 p p'" depends on time only through the parameter À
We are specifically referring systems with large number of degrees of freedom, N: if NV small enough it is possible to obtain 'second law violating' behavior, see for instance [49 67].
'Eq. (22) is also called the differential entropy of continuous random variable, which has some important differences from the discrete version. Refer to chapter of (70) for details.
26

---

Stot
4
6
time
Figure 7: Diffusion is a non von-equilibrium process that generates entropy over time. On the left, we see snapshots diffusive process Gemstein-Ullenbee the distribution evolves from 1 + 6 (i.e. Pa + po). and entropy produced till that point in time is indicated the right. Note that Scot the total entropy produced, which different from the change in Gibbs entropy of the distribution, which is negative in this experiment.
In the forward direction (increasing s)
(see Fig. 7). Then, our ignorance of the particle positions would be reduced, so the change in Gibbs entropy is negative. However, the total entropy increases, expected; Stot is not just the change in Gibbs entropy.
We can see this explicitly by looking at the expressions for both. Combining Eqs. 66 and 72 and integrating by parts one obtains
Sc ScIPol - Sclpal = ds E
V log P + V 6+
(73)
2
which is different from the total entropy,
Sior / ds -
R.
2b+
V log p
70
DEL (pallE(7)
Dx (pollee2)
(74)
These expressions differ when by * To understand how they are related look the free energy
F(p]
Elp) --
(75)
where the temperature 02/2 and the energy is
= [U(x))
U(x) = -(x)=--a - d
(76)
For simplicity we will assume that b_ and a are time-indeen timeiindpenndet (cf. Eq. (5). so we have a static equilibrium state Peq - exp(-BU((x)) (z)), and that po Peq- The generalization to the time-dependent case is straightforward. Then,
Stot DKL (pallPea) --Sclpal + 3E'pal log Z 3(F[pa) F(peg]),
(77)
where in the last step we have used 3Fipegl = log Z, which follows from evaluating Eq. 75 on Peg. Since Stot is positive, F[pa] > F F(Peq] irrespective of whether Se [pa] is larger smaller than (peae In Fig. 7 frame has a higher Gibbs entropy but also a higher energy, so the particles coalesce into the distribution in frame 6, giving up some of their Gibbs entropy move to lower energy configuration. Thus, Langevin dynamics moves Pa towards the lower free energy state Peq [7I. The minimization of free energy is also closely related to the interpretation of the Lakker-Plance equation as a Wasserstein gradient flow [72].
Relating Stot to free energy also helps us connect the discussion in Sec. 2 to statistical mechanics. In systems with large number of particles, the equilibrium distribution of microstates sharply peaked, with the most probable microstates piled up around the free energy minima (see (3] Sec. 4.6 in
27

---

731). The distributions are the microstates in the Schrödinger picture, and P(p] x exp(-8NFFp)) peaked Peq. Then,
PPNP
Stot
Plpal.
(78)
Modifying the drift term in Eq. 9 changes the free energy landscape such that pa becomes its new minima. In other words, the diffusion model applies external force to do work on the system, and
the range of possible outcomes of the combined arrangement constitutes non-equilibrium analog of the Gibbs ensemble.
D Score matching
In Sec. we touched on the difficulty in defining neural entropy for the score-matching model. This point warrants further elaboration. We start with
dX;; (bb (x,, 7 - dX; bb (X,, . T t)dt o(T t)dB,.
(ogg))llog((p))) ()d + a(T - L)dB,,
(79a)
(79b)
Notice that the drift term in Eq. (79b) has the opposite sign to the one in Eq. (13b). This seems like a natural choice, since modifying -b+ + -b+ + 0²8g sets up the model to learn the score V log p Eq. (19). But note that 6+ is a confining drift term which means -b+ is not. Therefore, Eq. (79b) does not have quasi-invariant distribution, and the intuition from Sec. no longer holds. We can Identify the inconsistencies arising from this conceptual breakdown through straightforward calculation.
Starting with Eq. 11 let us tentatively define the ideal score matching neural entropy
SAN (T) /' ds Ep | logpl?
(80)
The quantity on the right can be related to the non-equilibrium Gibbs entropy of the diffusing distribution, which will write in terms of the time variable (cf. Eq. (72). The change in Gibbs
entropy under the forward process is given in Eq. 73 We may therefore rewrite Eq. (80) as
Sam (T) = SG Pol - Sc [pa] L. ds Ep [V- b4].
(81)
To simplify further we need choose the drift term b4. Let us consider the Variance Preserving (VP) process [ for which by (y, ( 8) = -3())//2 and (s)
Bss) in Eq. (40), where ((s)
positive. Noting that V b+ - (3)) = 3(s), Eq. 81 reduces
S (T) Sc [pu] Salpal D
/ ds B(s).
(82)
Upon closer inspection, Eq. reveals a problem with the score matching neural entropy. Consider the trivial case where Po and Pa identical. The Gibbs entropies at the initial and final times equal, therefore the first two terms cancel. But we are still left with a positive integral on the r.h.s. (see Fig. [b). This also apparent from Eq. (80)the score function static but is not zero, the neural entropy is some positive number, and "information' is delivered to the network. Next, imagine changing pa some other distribution pa'. with Po and the 3-schedule kept fixed. The only change in Eq. 111 is the -Sa [Pa] term. As a result, if pa' has a larger entropy than Pa the neural entropy decreases. would seem that the network needs to remember less information transform Po - Pa' than it does to convert Pa to itself!
The issue arises from the - by term in Eq. 779). For the VP process that SDE
3 x,dt + BdB,
dx
(83)
But this process has repulsive drift term which, given enough time, dilutes the distribution away to infinity. Therefore, the network 8g has to work against -b, to keep the distribution po intact. In the above examples the growth in neural entropy due to the (ss3(s) term in Eq. 82 is indicative the
effort needed to hold the diffusing particles in place the drift 8x/22 tries to drive them apart (sec
28

---

8
Score Matching
Entropy Matching
d8 (()
55
2
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.6
0.8
1.0
(a)
Entropy production rate.
(b) Total entropy.
Figure 8: Ideal neural entropy curves for matching and entropy matching models with VP process. For matthing models, the neural entropy is the same as Stot (cf. Eq. [11 which is why entropy production trails off as forward diffusion approaches its final state.
Fig. 8
For this reason, the matching neural entropy from Eq. (80) is not accurate gauge of the non-trivial information the network must store to reverse diffusion. In practice, Eq. (80) can be approximated by
NNN (T) = l
0² 2 Ep
|
(84)
just like we did in Eq. (18). Experimental comparison the neural entropy in entropy-matching models and the Sum for score-matching models are shown in Fig.
Entropy-matching
Score-matching
D=4
D=5
0.4
0.6
D=
D=7
KI (pa pa)
D=8
0.3
0.4
0.2
0.2
0.1
5.0
7.5
10.0
12.5
15.0
Sem (T)
S
Figure 9: KL versus neural entropy for the entropy-matching and score-matching models. Both models trained series of progressively larger Gaussian mixtures, just like the ones used for Fig. 11 The experiments are repeated in different dimensions, D. Sem is the neural entropy defined in Eq. (T). Sem is analogous, but different, quantity defined in App. D in attempt to define neural entropy for score-matching model. Note how network performance decreases at lower values of Swm in contrast to the trend in entropy-matching. This behavior is explained in App. D The VP process was used for both sets of experiments, with the same 8-schedule.
29

---

VPx
SL
VP
10
10
6
8
SNN (T)
4
2
50
100
150
200
50
100
150
200
50
100
150
200
1e2
le2
8
D=3
0.6
Díi (Pa lpp))
6
9
0.4
0.2
0.0
50
100
150
200
0
50
100
150
200
50
100
150
200
Thup
Thpp
Tep
Figure 10: The evolution of neural entropy (top) and the KL between Pa and the reconstructed distribution pe (") (bottom) over training epochs, Nep- The entropies and KL are measured at The Pa 's are Gaussian mixtures in D 3, 6, dimensions. The dashed lines are the actual value of Stot generated by the respective diffusion processes. The VPx and SL processes (n dg = 0.1) produce two orders of magnitude larger Stot than VP, which is why SNN is slow to catch up in these models-the the network takes longer to absorb more information. This increase in retention is tracked by a decrease in the KL. Here again, the VP model outshines VPx and SL: the diffusion model can reconstitute pa more faithfully when it has to remember less information to do so.
E Details of experiments
All models in this paper, except for the ones in Fig. 9 were trained with 4 random seeds varying both weights initialization and order of training data, and the results were averaged over. The faint bands in the plots are within one standard deviation from the mean measurements. All computations were done on A100 GPUs with 80 GB memory. The CIFAR-I 10 models were trained on 4 GPUs in parallel while the Gaussian mixture and MIP experiments were trained on just one. Training on CIFAR- with the full dataset (n, = 5000 Fig. (6) takes 4.5 hours. Experiments for MNIST that stop between training epochs to compute log densities (see Figs. 3 13). and repeat for different numbers training samples, take around 4 hours for each training seed. For low-l D models transport experiments we used Fourier features on the variable to help the the MLPs learn better 74.. These were inserted before the input stage of an MLP with architecture (512, 256 D). We use T = 1 in all experiments. More details about specific experiments are given in the respective figure captions.
E.1 Diffusion models
We use three kinds of diffusion processes to experiment with different entropy production profiles in diffusion models. These are the VP, VPx, and Straight Line (SL) SDEs from Eq. (24) and Eq. (00) respectively. These are Ornstein-Uhlenbed processes, which have the general form 22
aY, f()) (s)dB,
(855
30

---

1e2
VPx
SL
VP
6
D Kt.(Pal llpp))
2
4
2
2
0
1000
2000
1000
2000
0
10
20
1e2
1e2
1.25
Residual loss
1.00
0.6
0.75
0000000-000
0.4
0.50
0.2
0.25
0.00
1000
2000
1000
2000
10
20
SNN(T) (T)
(7)
(7)
Figure 11: KL and residual loss at Rep = 200 epochs vs. the neural entropy in the network. The plots but we vary the total entropy produced by
are generated with the same experimental setup as Fig. 10 using increasingly broader pa s.
The perturbation kernel of this SDE
p(y.l00) N (ysim(s)yo ((s))))) 11))
(86)
where
m(s) exp (L, d3f(8)
(87a)
E(s) uls)2 /' ds (3)
(87b)
m(s)2'
Starting at 0, a sample ya ~ is propagated to ys p(s)ya + Es)) at an intermediate time E (0, TJ. where NN 1D). The object
(88)
v Vlogp(y. = Σ(s)
(89)
used in the denoising entropy-matching objective
ass)² .p,.bla)
T DDEM := T Egarapa (...T)))) A(s)
V Vlog + Vlogp(y.lu (aa)
(90)
is straightforward to show that LDMM is equivalent the upper bound Eq. when A(s)
76). For instance, plugging se V log p!!) eœ into the derivation in App. A of 40 would suffice. Empirically, it has been found that the choice A(s) 2Σ(s) (s) produces better results in image models 5 although (0) reports that A(s) = gives better log densities when used in conjunction with importance sampling. We have used the prescription from 5 with sampled from the uniform distribution (0, T) with no importance sampling- the experiments shown Figs. and 10 each Ya evolved to 10 random values of from this interval, which improved KL estimates whilst also
31

---

Entropy production rate
1e3
Total entropy
1.0
2.5
0.8
2.0
NN (s)
0.6
SNN (s)
1.5
0.4
1.0
0.2
0.5
0.0
0.00
0.25
0.50
0.75
1.00
0.2
0.4
0.6
0.8
1.0
1.0
2.5
Zoomed
Zoomed
0.8
2.0
(x)
0.6
1.5
0.4
1.0
0.2
0.5
0.0
0.02
0.04
0.06
0.08
0.10
0.02
0.04
0.06
0.08
0.10
5
10
Rep =250
nep 550
#1p=1
=8p=80
Rig =350
Rep -600
Rep
The 150
Rep 400
Entropy production curves for an image diffusion model trained the entire MNIST dataset (ng = 6000 from Fig.3) with the VP process. The different lines correspond to entropy measurements different epochs of training (see Fig. 2p. The lower panels zoom in on time interval close to the start the forward diffusion process. Notice how the entropy production rate is sharply peaked
Figure 12:
0. This is due the dimensionality of the data manifold Ma. Since Ma is much lower dimensional than the ambient pixel space the model must supply large amount of information I' (or 0) place the samples on Ma. The same behavior also appears in score- motchin models, but it is often conflated with a numerical divergence at = 0 due to the vanishing of (s)) Eq. (89) 77 The latter is addressed by truncating the diffusion process
10 [6]. These
entropy production rates are computed by splitting the interval (10-5, 77] into 500 time steps and computing the average Eq. (3 over 1000 samples from the test dataset propagated to each step.
32

---

le2
le2
Rg
10
Rg-
40
Cross-entropy
Re
SNN (T)
Loss
Rg
Re 6000
20
-10
200
600
200
600
200
600
Thpp
Dep
Tee
Figure 13: The evolution of neural entropy, cross-entropy, and loss over training epochs for unconditional image diffusion model (SL) trained on the MNIST dataset. This is the analog of Fig. for the Straight Line diffusion model (ef. Eq. (20)). Notice that far greater amount of neural entropy is produced here compared to the VP process, for reasons explained Sec. However, the cross entropy settles similar values for both VP and SL. The scaling of (T') with the number of samples, at Pep = 600, still exhibits a nearly logarithmic trend, just like with the VP process (see Fig. T6..
le5
Entropy production
lc3
Total entropy
1.2
0.8
SNN
0.6
0.4
0.2
0.0
0.00
0.25
0.50
0.75
1.00
0.2
0.4
0.6
0.8
1.0
1e5
le3
1.2
Zoomed in
Zoomed
0.8
SNY
0.6
0.4
0.2
0.0
0.02
0.04
0.06
0.08
0.10
0.02
0.04
0.06
0.08
0.10
5
The #40=40
Rup 250
Rep 500
=1p=10
nap =80
Rap =350
Pap =600
220=2
nog 150
Ri 4400
Figure 14: Entropy production curves for an image diffusion model trained on the entire MNIST
dataset (n.- 6000 from Fig. [13 with the SL process.
33

---

2
2.5
(ac(Palll (pp)
SAN
Loss
1.5
0
200
300
100
200
300
100
200
Tee
Top
N N 500
N
N- 100
N 1000
N' 10000
Figure 15: The evolution of neural entropy, cross-entropy, and loss over training epochs for diffusion model (VP) with MLP core trained on mixture of five Gaussians in D 6 dimensions. N is the number of samples used for training. The scaling of SNx (7) with the number of samples does not show neat trend like the ones for the image models (see Fig. [6]. Due the unstructured nature of the data and the relatively weak prior constraints imposed by the MLP, the model learns different distributions at different N. This most emphatic for N 10 where the data is so sparse that the model increasingly concentrates probability mass around the given samples as training progresses. This is why the KL rises and loss continues drop for N 10.
le4 CIFAR- IOUU-net+SA
MNIST/U-net
Gaussian Mixture/MI.P
3.6
2.50
1.2
3.4
2.55
5NN (2)
1.0
2.00
3.2
0.8
1.75
3.0
0.6
1.50
2.8
10a
103
10*
10a
10a
10a
10'
10a
10a
10
N
N
N
Figure 16: Neural entropy versus number samples for CIFAR- 10 trained on U-net with self-
attention layers (left), MNIST trained on simple U-net (center) (cf. Fig. 3). and mixture of Gaussians in trained on an MI.P-based diffusion model (right) (cf. Fig. 15). These are the values of
SNN (T) at the end of training. The first two plots are the same ones from Fig. [ Note the absence of the logarithmic trend in the Gaussian mixture/MLPP case. All models shown here use the VP process.
34

---

le5
Entropy production
1e4
Total entropy
1.25
1.00
NN
0.75
SNN
0.50
0.25
0.00
0.00
0.25
0.50
0.75
1.00
0.2
0,4
0.6
0.8
1.0
8
1c5
lc4
Zoomed
1.25
Zoomed in
1.00
0.75
SNN
0.50
0.25
0.00
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
n==10
12=100
n,=1000
n1=30000
5000
Figure 17: Entropy production curves for CIFAR- 10. Note that the different colors correspond to the different number of samples per class used for training, ne. rather than the number of epochs. All CIFAR-10 10 experiments were trained to 200 epochs. Computing the neural entropy in this case requires special care since the peak near the lower-dimensional data manifold with the CIFAR-10 images lives in much higher-dimensional pixel space compared to MNIST.
is even sharper than ones for MNIST (see Fig 12
reducing loss fluctuations 77 per epoch.
78 But in the image models we sampled at just one random 8 per
The functions u(s) and (ss) for the VPx and SL processes can be read off from Eq. 25 and Eq. (2 respectively. In both cases (s)) vanishes at 0. so Eq. (89) diverges at that instant. Therefore we do not venture below s 10 10-5 when training with Eq. (90). The SL SDE has an additional singularity
T 1, we also cut off samples at 1 10 10-5 in that case. Note that Bi approximates E(s)sL ss do since oo is small, but we retain the full time-dependence in our experiments.
New samples from diffusion model can generated efficiently using the Probability Flow ODE 6 79 80
0²
dx(()
bb T ( Vlogp(a,t) dt,
(91)
where the true score is approximated by V log & V log + eg in entropy-matchinn models.
35

---

CIFAR- 10 samples (seed 7)
72=10
12=100
n,=1000
n,=3000
n,=5000
CIFAR- 10 samples (seed 23)
nd=10
n,=100
1₂=1000
n,=30000
nt=5000
few samples generated from the conditional diffusion model trained on CIFAR-10 10 images. Each row in grid contains one image from each class, and the rows correspond to models trained with different numbers of samples n. per class. We used the probability flow ODE to produce these images (cf. Eq. (I)), with the same ten initial noise vectors for each ne- These samples affirm the key takeaway from the neural entropy vs. N trends Figs. and [6] the rate of additional
Figure 18:
information absorbed by the model decreases with each new sample. The two grids differ by the seed value used to initialize model weights and fix the order in which the training samples are applied.
E.2
Density estimation
In our experiments with Gaussian mixtures and MNIST, we compute the KL divergence to the true distribution and the cross- cross-entropy respectively gauge mode performance (cf. Figs. 3 13 and [5]. But diffusion models do not give exact log densities on the learned distribution, despite claims in the literature. However, lower bound the log density log po (x. T) be established from Eq. (61 That latter is, explicitly,
2 E ['
T
1
u11 - V v.u) - log Pa (Yr 0)
logPu (z, T)
ds
202 1b4
(92)
The expectation is computed over trajectories generated by Eq. 440) that start at v at = 0. The bound is saturated b_, in which case (ar, T) = (ar) (cf. App. B.4), but u is approximated by neural network in diffusion models. We can integration by parts to avoid taking the gradient
36

---

of the neural network in the V u term. For a vector-valee valued function h(ys. s),
Ey, V h( s) = =x] 1- dy. h(us,)) s) Vp(us, s a 0)
(93)
E-E,, s,) Vlogp( s/ 0) x]
where p(us; s|2r, 0) is the same kernel from Eq. (89) with the time-dependence indicated explicitly.
and we have assumed that the product hp vanishes at the g-boundaries. Using Eq. (93) in Eq. 92 we obtain
log Pu (xr, T)
N
(94)
T
E
ds
16 u12
V Vlogp(X..s(Xo.0) log
log pm (YT 0)
20
By transferring the gradient operator from u we avoid the need to take derivatives of the neural network; since the transition probability a Gaussian the gradients of their log are easy calculate. The r.h.s. is path integral, which can estimate efficiently as Monte Carlo average
u|l + u v logplus, 8/2, 0)] Sa [po]
log (r, T) 2 -T -TEgee
11b
2
(95)
Here, we have replaced Byr=p(pr.T(aa T(z,0) [log P., (Ur)) with the negative Gibbs entropy [pol since yr would be distributed as Po irrespective of the x at which started, very good approximation (ef. App. C.2). Finally, for entropy- entropy-matching models, -b_ p&ca, and therefore
log Pg em (x, T) 2 Salpo]
022
|| - (b+ 0²ea) V Vlogpuus 8 (r, 0)
-T s0 Pls.
V Plog Peg + eo
2
(96)
We use this lower bound in lieu of the true neural log densities in our calculations. The MC average must be computed with fairly large number of samples [77] 78). For the Gaussian mixture experiments, this an excellent substitution. The results are noisy but still insightful the image experiments. A version of Eq. (96) for score-matching is derived in (20), which is obtained by setting eg = -V log plt) + 8g in this one.
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the goal of the paper and the main contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of work are discussed inline in Sec. 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
38

---

Justification:
The assumptions and proofs are given in detail in App.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details of the experiments are discussed in Sec. 6 and App.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
39

---

reproducible, they Answer: [Yes]
Justification:
The code for all experiments is available here
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Serriides/CodSuummiis
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nip. http:c/cpplllgggiuieee http//ddede//oeubmmi
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training details are given in Sec. 6 and under the figures in App. detail will be provided with the code.
E
More
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experiments in this paper, except for the ones in Fig. 9 are computed over four different initializations of the respective model weights. The plots indicate the region of one standard deviation from the mean results, when relevant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
40

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details are given in App.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We conform.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no direct societal impact that we can see.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
41

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines: Not applicable.
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the creators of MNIST and CIFAR-10.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
42

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We do introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
43

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
44


Reviews:
[{"review_id": "rDwu1azsyi", "reviewer": "Reviewer_VsRX", "summary": "This paper aims to measure the information that a trained diffusion model stores about its training data. The main result of the paper is that this information is proportional to $\\log n$ where $n$ is the training dataset size, rather than $n$, as would be the case for a memorizing network sampling from the empirical training data distribution.", "questions": "What is $S_{\\mathrm{NN}}$ measuring? Why would it be the correct measure of information stored in the network? Is it still valid for an untrained network? It seems it can be made arbitrary large for arbitrary networks. What do the equations look like in the variance exploding case ($b_+ = 0$), do the authors still think of them as valid in that setting? Why are the equations not invariant to time reparameterization? Simply rescaling time should not fundamentally change the information needed to store the training distribution.\n\nIn my current state of understanding of the paper, I cannot recommend acceptance, but I am willing to improve my score if the authors convincingly demonstrate the well-foundedness of their entropy measure.", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 4, "originality": 4, "strengths_and_weaknesses": "This paper tackles a fundamental question in diffusion generative modeling, and its results are very intriguing. However, in its current state, I find it difficult to understand, especially as a non-physicist. \n\nSome of the physics jargon is difficult to understand for non-physicists: e.g., top of page 2, \"the entropy of the phase space density over the neural network’s internal microstates\" does not make any sense to me. Referencing \"the total entropy produced by a diffusion\" without motivating its expression or the Maxwell demon without briefly introducing it is also not appropriate given the audience of NeurIPS. I also don't understand what it means for the marginal distribution of a particle evolving under an SDE to itself fluctuates: by definition the distribution is deterministic, and evolves under an ODE (Fokker-Planck). It seems that what is meant is the empirical distribution of the $M$ walkers, but in that case it cannot be equal to $p_{\\mathrm{d}}$ nor $p_{\\mathrm{eq}}$.\n\nI end up quite lost in the notation, especially given that some objects are never explicited (e.g., the kernel $h_\\star$: does it admit a closed-form equation?) or redefined multiple times. For instance, during the first reading I got confused between $S_{\\mathrm{tot}}$, $\\hat S_{\\mathrm{NN}}$, and $S_{\\mathrm{NN}}$, or $s_\\theta$ and $e_\\theta$, or $p_0$, $p_{\\mathrm{eq}}$, and $p_{b_+}$. These quantities are often introduced early in the text, not used for a while, then mentioned later without reminding the reader what they stand for.\n\nThe main issue is that after reading the main text and the appendix twice, and spending some time thinking about it, I am not confident at all about what is measured in Figure 1, the central result of the paper. More specifically, I don't see how or why the proposed definition of $S_{\\mathrm{tot}}$, and even less so $S_{\\mathrm{NN}}$, should coincide with the \"information load\" on the network. First, in the so-called \"variance-exploding\" SDE (where $b_+ = 0$), $S_{\\mathrm{tot}}$ is simply equal to $H(p_{\\mathrm{eq}})$ - $H(p_{\\mathrm{d}})$. This does not seem to me to be a reasonable estimate of information \"needed to reverse the diffusion process\" in that case. Second, after several pages of careful motivation and explanation of $S_{\\mathrm{tot}}$, $S_{\\mathrm{NN}}$ is introduced instead without explanation. As a result, I have no intuition what it measures.\n\nFinally, the following reference might be relevant to the authors, as it derives several information-theoretic relations in diffusions that are very related to those in this paper:\n> Kong, X., Brekelmans, R., & Steeg, G. V. (2023). Information-theoretic diffusion. arXiv preprint arXiv:2302.03792.\n\nTo summarize:\n- Strengths: the paper studies an important topic, and the claimed results are very significant (though I cannot evaluate them fully due to my limited understanding of the paper)\n- Weaknesses: the style of writing is hard to follow, and the final definition of $S_{\\mathrm{NN}}$, central to the paper, is not motivated enough (why should it measure the amount of information about the training set stored by the diffusion model?)", "was_revised": true, "final_justification": "This paper presents intriguing results, and even if the quantities it measures are somewhat flawed in my opinion, they perhaps could be refined in future work.", "rebuttal": "Thank you for the review, and for engaging with the paper despite the fact that it was outside your area(s) of expertise. We value your feedback as non-physicist, and appreciate your transparency.\n\nWe would like to make the paper accessible to a broad audience while remaining within the constraints of the page limit. For this reason, we focused the main text on presenting our novel contributions, rather than review established concepts in physics and information theory. In the appendix we discuss these ideas more gradually and in greater detail, with extensive references that fill in the pre-requisites. Even when we allude to specific physics concepts like phase space densities and Maxwell's demon, we have written it in such a way that skimming over those portions of the paper does not diminish our core message. We would also like to gently push back against the characterization of some of the wording as jargon: this is standard terminology in statistical physics, and is intended to communicate the ideas efficiently rather than to feign sophistication.\n\nWith respect to the distribution itself fluctuating, an equation like $\\partial_t p(x,t) = \\frac{\\sigma^2}{2} \\nabla^2 p(x,t)$ gives the *dominant* evolution of the distribution. What we hoped to illustrate with the example of the random walkers on the lattice is that finite samples deviate from this average. When we say that the probability of a random walker to jump left (or right) is  $q_L$ or $q_R$, it does not mean that exactly $q_L$ fraction of all the walkers at a site will jump to the left in every single time step. Here is another example: if you take a perfectly fair coin and toss in 1000 times you don't get exactly 500 heads/tails; there will be some fluctuations. In fact there is a small but non-zero chance of getting all heads, $p(\\text{all H})=2^{-1000}$. In the same way, the random walkers have a small probability of deviating away from the evolution governed by $q_L$ and $q_R$ at each time, culminating in a vastly different outcome that what one might expect from the dominant evolution. The probability of this happening is very small of course, which is what leads to the form $\\mathcal{P}[p_{\\rm d}] \\approx e^{-M S_{\\rm tot}}$. This is an example of the *Large Deviation Principle* [1].\n\nRegarding the notation, we are sorry to hear that the symbols were difficult to follow. The meaning of each of these symbols is explained as they are introduced. For example, we state explicitly under (2) that \"$h(x_T|x_0)$ is a kernel that transports $p_{\\rm eq}(x_0)$ to $p_{\\rm d}(x_T)$\" and that $h_\\star$ is the optimal kernel than minimizes the KL to $g$. Furthermore, we give a more elaborate discussion of these objects in App. A.2 where the lattice random walker problem is explored in greater detail. The same is true for the other symbols in the list. There is also a paragraph summarizing these symbols at the beginning on App. B, page 18.\n\nThe concept of neural entropy defined here requires a confining drift term in the forward diffusion process, because those are the only processes which have an equilibrium state, $p_{\\rm eq}$, that results from a trade-off between the confining effect of the drift and the dispersive effect of the noise. As explained above, $S_{\\rm tot}$, and therefore the neural entropy, is related to the probability that the system fluctuates away from the equilibrium state. The variance preserving term does not have an equilibrium state, and therefore we cannot define neural entropy in that case. This is also related to the discussion in App. D. The connection between $S_{\\rm tot}$ and $S_{\\rm NN}$ is explained in the response to the review by nu8R. Briefly, the network must absorb $S_{\\rm tot}$ amount of information to reverse diffuse perfectly.  However, due to imperfections in training and finiteness of the training data it can only asymptote to this ideal. Figs. 2 and 10 in the paper show how $S_{\\rm NN}$ approaches $S_{\\rm tot}$ over the course of training.\n\nQuestions:\nWe believe we have addressed several of your questions in our comments above. We answer the remaining ones here.\n\n1. Is neural entropy valid for an untrained network? Answer: No. Neural entropy measures the amount of information stored in the network rather than total network capacity. An analogy would be a hard drive: if you store a 1GB file, the drive contains 1GB of information, regardless of whether its capacity is 10GB or 100GB.\n\n2. To understand the effect of time rescaling, consider the VP process ${\\rm d} Y = -\\frac{\\beta(s)}{2} Y {\\rm d} s + \\sqrt{\\beta(s)} {\\rm d} B$, which is applied for a time $T$. We can rescale this by introducing a new time variable ${\\rm d} \\tau = \\beta(s) {\\rm d} s$, which converts the SDE to ${\\rm d} Y = -\\frac{1}{2} Y {\\rm d} \\tau + {\\rm d} B$. However, the latter has to be applied for a time $T' = \\int_{0}^{T} \\beta(s) {\\rm d} s$ to reach the same state. It is then straightforward to see, from the expression for neural entropy and (25) in the paper, that both SDEs produce the *same* $S_{\\rm tot}$. So entropy production is indeed invariant to time rescaling, as you correctly point out.\n\nWe hope this response clarifies the key concepts and addresses your concerns. Thank you for acknowledging the significance of our work, and being forthright with your self-assessment.\n\n[1] H. Touchette, \"The large deviation approach to statistical mechanics\", arXiv:0804.0327v2 (2009)"}, {"review_id": "eeYhZHwnmA", "reviewer": "Reviewer_KtyK", "summary": "This paper explores the interplay between deep learning and information theory through the lens of diffusion models. It proposes a novel metric, neural entropy, to quantify the amount of information retained within the neural network during training. This measure captures aspects of both the underlying data distribution and the dynamics of the diffusion process. Empirical evaluations on image diffusion tasks demonstrate that these models achieve remarkable efficiency in compressing structured data.", "questions": "I would appreciate a detailed and thoughtful response to the concerns raised in the Weaknesses section. If these concerns are adequately addressed, I would be willing to reconsider my evaluation and potentially adjust the score in a more positive direction.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper is well-motivated and makes a meaningful contribution to the theoretical understanding of diffusion model training.\n2. It is well-written and presents extensive theoretical analysis, supported by empirical results that validate the proposed theory.\n\nWeaknesses\n\n1. The relationship between network size and neural entropy requires further consideration. The paper would benefit from additional theoretical analysis and experiments addressing this relationship. Referring to related studies, such as [A], could strengthen this aspect.\n2. Including a discussion on generalization could strengthen the impact of this study. In particular, it would be valuable to investigate whether the model tends to overfit or generalize when training proceeds with new data after entropy saturation. Such an analysis would provide deeper insight into the relationship between entropy dynamics and generalization behavior.\n\n[A] Morris, John X., et al. \"How much do language models memorize?.\" arXiv preprint arXiv:2505.24832 (2025).", "was_revised": true, "final_justification": "The authors have provided a clear and satisfactory response to several of the concerns I raised in my initial review. I appreciate their effort in addressing these issues, and as a result, I am updating my score accordingly.", "rebuttal": "Thank you for the review and suggestions. We address your concerns below.\n\nWe preface our comments by noting that we are actively working to understand the relation between neural entropy, network size, and generalization. Therefore, the fact that you and another reviewer have independently raised these questions is encouraging. Our goal with the present paper was to introduce the concept of neural entropy and substantiate it with theory and experiments. To that end we have only included findings that are clear and unambiguous. While we are eager to share results related to network size and generalization, this research is still a work in progress, but we share some preliminary insights below.\n\n1. Network size: We conducted some early experiments with the Gaussian mixture + MLP setup where we added/removed layers to the MLP to study the effect on neural entropy., keeping $p_{\\rm d}$ and the forward process fixed. That is, the diffusion process produced the same $S_{\\rm tot}$ in all these experiments, which is also the information the network needs to store to reverse the process. While it is true that a very small MLP fails in absorbing the entirety of $S_{\\rm tot}$, we find that beyond a certain point adding more layers has little benefit. That is, there is some network size at which the NN captures most of $S_{\\rm tot}$, and making the network bigger than this has diminishing returns in the quality of reconstruction, as measured by ${\\rm KL}(p_{\\rm d} || p_\\theta)$.\n\n    In image models, this question becomes more nuanced due to the interplay between network size and inductive bias. For example a larger U-net where the inductive biases are weakened, say by using a larger stride in its convolutional layers, may perform worse than a smaller U-net where the stride is set to 1. We are currently investigating this tradeoff in greater depth.\n\n2. Generalization: Diffusion models tend to overfit/memorize when training on sparse datasets and/or over long training runs [1]. Apparently the strong inductive biases in image diffusion models safeguard against memorization to a certain degree, and much more so than in the Gaussian mixture/MLP case (see paragraph to the left of Fig. 4 in *Neural Entropy*). This is another way to understand Fig. 1 and 16 from our paper. Let us explain.\n\n    Memorization happens when the model learns a reverse drift that drives the initial distribution back to set of Dirac delta functions at the training data. But concentrating the probability mass into such small regions requires that the network supply a far greater amount of information, since information negates entropy and 'thinner' distributions have lower entropy. Or, in terms of Schrodinger's Gedankenexperiment, it is much more improbable for a homogenous distribution to automatically fluctuate into a configuration with very dense slivers of probability, than to a distribution where the probability is more evenly distributed. This is to say that the neural entropy would have to be much higher for the model to memorize. But, as we see in Fig 3. for example, the growth in neural entropy plateaus at later epochs of training, which means the model has to train for much longer to reach a memorized state. So memorization is harder for diffusion models because they resist overfitting due to the information cost. While this does not fully explain generalization, for example why the generated images are sensible, it gives us a promising new tool to approach the problem. We plan to explore this direction in a follow-up work.\n\nWe appreciate your suggestion to cite [A] (Morris et al., 2025). While their study focuses on language models, their insights on memorization behavior are indeed relevant.\n\n[1] Gu et. al. \"On Memorization in Diffusion Models\" arXiv:2310.02664v2 (2025)"}, {"review_id": "1n1yK1P9NI", "reviewer": "Reviewer_nu8R", "summary": "The paper defines and analyses a measure called neural entropy. The definition of neural entropy is justified via concepts of information theory and thermodynamics applied to random walk processes and related to diffusion models in machine learning. The analysis of this measure is performed empirically in both a toy controlled setup, where the ground truth entropy is known, and an uncontrolled setup, using two common datasets in the machine learning community.", "questions": "## Questions\n\n1. Section 6, paragraph “Transport experiments”: Good point to limit the inductive biases here, but what about the other effects on the neural entropy such as the size of the network, the choice of the optimisation algorithm, the batch size, the number of samples per epoch (or did you resample your Gaussians at each epoch as if an infinite amount of data was available?), etc.? Did you notice any interesting behaviour or was the neural entropy stable with respect to such hyperparameters? Sorry if I missed this in the text.\n\n1. Did you try your method on a larger and much more complicated dataset such as ImageNet? As I acknowledge that one could always try more, it is not a weakness strictly speaking. Nevertheless, it would be interesting to check whether, at a certain point, there is a limit to the inductive biases of the network which would manifest via the neural entropy.\n\n1. Section 6, paragraph “Storage experiments”: Do you have any quantitative argument to justify that the model is indeed not memorising?\n\n## Suggestions\n\n- Section 6, paragraph “Transport experiments”, line 4: There may be a typo with the reference to Eq. (15), which I think should be Eq. (17).", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The paper clearly states, at multiple places, the limitations of the neural entropy measure and the experiments performed.\n\n- The analysis performed gives several important insights on neural networks, namely on their capability to store information and the rate at which they learn it.\n\n- The analysis of neural entropy on a controllable multivariate Gaussian case gives clear and interpretable results.\n\n- The conclusion on the success of the variance preserving scheme is interesting.\n\n## Weaknesses\n\n- I find the definition of neural entropy a bit confusing. Equation (11) defines the ideal neural entropy as the total entropy, while equation (18) defines the “actual” neural entropy as the subpart of the total entropy that is due to the diffusion model formulation. This shift in definitions creates a confusion on the interpretation of the measure.", "was_revised": true, "final_justification": "The response given by the authors, complemented by the rebuttal to reviewer KtyK, are convincing. The rating will remain the same.", "rebuttal": "Thank you for the review and many insightful questions. We address your points below.\n\nWeaknesses:\n\n$S_{\\rm tot}$ is the total entropy produced by the forward process. This is also the information that must be supplied to go back to the initial distribution $p_{\\rm d}$. In principle $S_{\\rm tot}$ can be computed if we knew $\\nabla \\log p(x,t)$ exactly, using (16) in the paper (we tried to copy the expressions here to make it easier to access, but it does not render). A perfect entropy matching model, with access to infinite data and infinite training time, would make the LHS in (17) zero, in which case $e_\\theta = \\nabla \\log p - \\nabla \\log p_{\\rm eq}^{(t)}$ exactly and the information stored in the NN is the ideal neural entropy $S_{\\rm tot}$. But this never happens in practice. Instead $e_\\theta$ only approximates this ideal (see Fig. 10). The actual neural entropy $S_{\\rm NN}$ is calculated with $e_\\theta$. We only introduce the ideal $\\hat{S}_{\\rm NN}$ for completeness.\n\nQuestions:\n\n1. Reviewer KtyK also raised the question about network size; please refer to our response there for a detailed discussion.  Re: batch size and number of epochs, yes we did experiment with these hyper parameters in the Gaussian mixture + MLP setting. The results are discussed in detail in another paper of ours, which has been cited in *Neural Entropy*, but we refrain from citing it here to stay in compliance with the double-blind reviewing policy. The short summary is that if the number of samples $N$ is too sparse relative to the ambient dimensions, the model overfits and we see the neural entropy rising (see response to KtyK). This also happens if the model trains for a long time. The results were stable over a range of batch sizes. We never experimented with the optimization algorithms. As for the question about resampling Gaussians: we used the same samples in every epoch, with shuffling. This was done to stay as close as possible to the image model experiments.\n\n2. We did not run experiments on ImageNet due to limited computational resources, but we agree it would be an interesting direction. We hope to run experiments on bigger datasets soon.\n\n3. See second point in the rebuttal to KtyK. In short, the inductive biases of the neural network impart a certain 'stiffness' to the increase in neural entropy, as shown in Fig. 1 and 3 of the paper. Therefore (image) diffusion models naturally resist memorization, since it would take a very long time to absorb enough neural entropy to reverse diffuse back to Dirac delta functions at the training samples.\n\nSuggestions:\nThanks for catching that, it will be corrected in future revisions."}, {"review_id": "rwKzu8aXpm", "reviewer": "Reviewer_mGKE", "summary": "This work introduces the concept of neural entropy, to quantify the amount of information stored in a neural network. The focus of this work is on diffusion-based generative models, and thus the “score network” or “denoising network” that is typically used to learn how to reverse the diffusion dynamics. Throughout a very didactical derivation, the authors argue that diffusion models are very effective in storing information, especially when dealing with large amounts of structured data: this claim is supported by mathematical derivations as well as simple experiments both on synthetic data and on simple image datasets.\n\nUsing neural entropy, the authors show that the growth of neural entropy with the number of training samples in diffusion models obeys to logarithmic scaling, suggesting that the marginal information gained per sample decreases as approximately $1/N$. Moreover, they also show that there is some sort of “speed limit” in the sense that diffusion models that employ “tricks” to make them faster, also need to store more information for an accurate reversal process.\n\nFurthermore, this work introduces the entropy-matching objective, as a way to tighten the KL divergence between the data distribution and its parametric approximation achieved with a “denoising network”. This is in contrast to a score-matching objective, for which a neural entropy expression is conceivable, but hard to interpret, especially in the case of a variance preserving process. Instead, the entropy-matching objective has a clearer interpretation in terms of neural entropy.\n\nFinally, this article presents a series of simple experiments to validate numerically the main theory of the paper: empirical lines match the predictions of the ideal ones, supporting the claims of the authors.", "questions": "* What are the main practical implications of this work? Can you illustrate how your ideas can help designing new diffusion-based generative models? This kind of discussion could help broaden the scope and audience of your work\n\n* I think the problems with score-based diffusion models, having a drift term that is “repulsive”, is important. The discussion in the appendix and in particular Figure 9, right-hand side, points at a possible mismatch of the neural entropy metric for score-based models. Can you help me clarify wether this implies that the proposed method to measure information stored in a score-matching based model is limited, or if there are any important limitations to score-matching models due to this drift term?\n\n* Recently, there has been a significant increase of work that use diffusion models, as well as other generative models to compute information theoretic quantities such as Entropy and Mutual Information. In those cases, the key objective is to compute the entropy of a given data distribution, using score networks for example, or to compute the mutual information between two data distributions, again using score networks for example. Do you think these works are relevant to some extent and could be discussed as related work?\n\n[1] Butakov et. al., “Mutual Information Estimation via Normalizing Flows”, https://arxiv.org/abs/2403.02187 \n\n[2] Franzese et. al, “Mutual Information Neural Diffusion Estimation”, https://arxiv.org/abs/2310.09031\n\n[3] Kholkin et. al, “InfoBridge: Mutual Information estimation via Bridge Matching”, https://arxiv.org/abs/2502.01383", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "* Strengths:\n  * This is a very easy to read article, very pedagogic and clear. It has many examples, and begins with simple processes inspired by established results in physics, but continue with more elaborate and realistic stochastic processes that are used in modern generative models. \n  * This work presents a very intuitive explanation for the amount of information stored into a network used to reverse the dynamics of diffusion processes. The relation between information storage and the structure of the data, the number of training samples, and the particular form of the stochastic process used is important and could be used to better engineer diffusion-based generative models\n  * The authors introduce the entropy-matching objective, which is easy to implement and optimize, similarly to flow-matching objectives, although care needs to be taken to select appropriate measures for numerical stability\n  * I liked the discussion and perspectives in the conclusion, aiming at answering the question about the application of the proposed neural entropy measure to other generative models, such as language models. The connections to diffusion-based LLMs is pertinent and interesting\n\n* Weaknesses:\n  * I’m lucky enough to have sufficient background on the literature required to understand this paper, but for a generic reader, this work might be somehow difficult to digest, as it is not self-contained. The (long) appendix does a good job in providing background material and proofs, but it requires a particular effort to “put all pieces together”\n  * The practical application of neural entropy to image diffusion models is hindered by some instabilities and calls for more sophisticated diffusion-based methods to avoid divergence\n  * The authors did not put a particular effort in translating the methodological results presented in this work into actionable information for practitioners who might use entropy-matching, and neural entropy to design novel (and better) image diffusion models", "was_revised": true, "final_justification": "I am happy with the rebuttal and the answers to my questions.\n\nI really liked this work and believe the narrative rooted in physics is a nice contribution to the community, which will spark interest and discussions.", "rebuttal": "Thank you for the thoughtful review and the many constructive comments. We will try to address your concerns below.\n\nWeaknesses:\n\n1. We acknowledge that not all readers in the ML community may have the necessary background in statistical physics. We have references several review articles and pedagogic papers throughout this work to direct readers toward the necessary preliminaries. Rather than repeating these materials with reduced clarity or completeness, we opted to focus on presenting the novel contributions of this work, which already makes for a lengthy paper, as you noted. We have assumed that basics of information theory, at the level of earlier chapters in McKay or Cover and Thomas, is familiar to the reader. If there are resources that present this material in a more accessible way to NeurIPS readers we are happy to cite them in the paper.\n\n2. If the divergence at initial time, $s=0$ is what is referred to as the instability in entropy matching, we should point out that the same phenomenon appears in score-matching models. There is some discussion of this in the caption for Figure 12. We point out that, for image models, there are two divergences sitting on top of one another at this instant. The first is due to the vanishing of $\\Sigma(s)$ in $\\nabla \\log p(y_s|y_0) = - \\epsilon/\\Sigma(s)$. But the second is due to the fact that the image manifold is much lower dimensional than the ambient pixel space. Score-matching suffers from both divergences as well.\n\n3. We have follow up papers in preparation that address precisely this. This paper is meant to be the first of a series. Our goal here was to introduce the concept of neural entropy and justify that it does indeed measure the information stored in the network. That said, there is a comparison between the SLDM and VP processes on MNIST images in Fig. 12 and 14.\n\nQuestions:\n\n1. The entropy picture allows us to gauge and compare the information load on the network under different choices of diffusion processes. For example, we mention that these results can be generalized to discrete diffusion, and therefore diffusion LLMs. In that case the thermodynamic speed limit can help us choose more optimal 'noising' processes to be applied on the training data. We plan to do some of these experiments ourselves. For image models, a relevant question is the neural entropy in latent space models. We suspect that the neural entropy for a model trained on the latents will be much smaller that one trained directly on the images.\n\n2. Great question! The fundamental problem with defining neural entropy in score-matching models is that the repulsive drift term does not allow for a quasi-invariant state. In practice, this means that a score-matching network has to put additional effort into 'fighting' the repulsive drift term, to keep the random walkers from moving to infinity, while also reshaping the distribution back to $p_{\\rm d}$. Although not shown in App. D, we found that the entropy matching models train faster (that is, the loss converges quicker) than score matching models in experiments involving Gaussian models. This makes sense, since entropy-matching puts less information 'strain' on the neural network than score-matching. These experiments can be found in another paper of ours, which is cited in the present work. We refrain from citing it here to preserve anonymity.\n\n3. Thank you for the references. Indeed, there is a close relationship between neural entropy and mutual information. We are writing a paper at the moment which discusses this in greater detail, and we will make sure to discuss your references as related work. In addition, Sec. 3.2. of Franzese et. al is closely related to the entropy matching model in our paper, so we will cite this work in future revisions.\n\nOnce again, we sincerely appreciate your comprehensive review, and your time."}]

Author rebuttal:

Meta-review: The reviewers agreed that the paper tackles a fundamental question in diffusion generative modeling, offering intriguing results. This is also an original contribution that connects diffusion models with information theory, presenting extensive theoretical analyses which are complemented by empirical results, giving several important insights on neural networks, namely on their capability to store information and the rate at which they learn it, whereas the analysis of neural entropy on a controllable multivariate Gaussian case gives clear and interpretable results. The conclusion on the success of the variance preserving scheme was found interesting. There were favourable comments on this manuscript, including clarity and readability, intuitive explanations for the amount of information stored into a network used to reverse the dynamics of diffusion processes, relation between information storage and the structure of the data, the number of training samples, and the particular form of the stochastic process used is important and could be used to better engineer diffusion-based generative models, and introducing the entropy-matching objective, which is easy to implement and optimize, similarly to flow-matching objectives, although care needs to be taken to select appropriate measures for numerical stability.

There was clear support for the recommendation to accept this work. In the rebuttals the authors diligently committed to implement some clarifications and improvements on the manuscript, to address concerns on the specialised jargon and the need to improve clarity on some definitions and motivations, as well as practical implications. I strongly encourage the authors to integrate the feedback received into the revised version of their paper.
Decision: Accept (spotlight)