On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion models
Authors: Christian Horvat
Jean-Pascal Pfister
Venue: ICLR 2024 poster (2024)
Keywords: gauge freedom
conservativitym intrinsic dimensionality estimation
diffusion models
explainable AI
theory
Primary area: generative models

Abstract: Diffusion models are generative models that have recently demonstrated impressive performances in terms of sampling quality and density estimation in high dimensions. They rely on a forward continuous diffusion process and a backward continuous denoising process, which can be described by a time-dependent vector field and is used as a generative model. In the original formulation of the diffusion model, this vector field is assumed to be the score function (i.e. it is the gradient of the log-probability at a given time in the diffusion process). Curiously, on the practical side, most studies on diffusion models implement this vector field as a neural network function and do not constrain it be the gradient of some energy function (that is, most studies do not constrain the vector field to be conservative). Even though some studies investigated empirically whether such a constraint will lead to a performance gain, they lead to  contradicting results and failed to provide analytical results. Here, we provide three analytical results regarding the extent of the modeling freedom of this vector field. {Firstly, we propose a novel decomposition of vector fields into a conservative component and an orthogonal component which satisfies a given (gauge) freedom. Secondly, from this orthogonal decomposition, we show that exact density estimation and exact sampling is achieved when the conservative component is exactly equals to the true score and therefore conservativity is neither necessary nor sufficient to obtain exact density estimation and exact sampling. Finally, we show that when it comes to inferring local information of the data manifold, constraining the vector field to be conservative is desirable.

Published as a conference paper at ICLR 2024
ON GAUGE FREEDOM, CONSERVATIVITY AND IN- TRINSIC DIMENSIONALITY ESTIMATION IN DIFFUSION MODELS
Christian Horvat
Jean-Pascal Pfister
Department of Physiology University of Bern christian www..i.....raauuneeee...
Department of Physiology Bern, Switzerland
jeanpascal wttpcccal.fsssterrrete .ww.sttrrunibee.c..
ABSTRACT
Diffusion models are generative models that have recently demonstrated impres- sive performances in terms of sampling quality and density estimation in high dimensions. They rely on forward continuous diffusion process and a backward continuous denoising process, which can be described by time-dependent vector field and is used as generative model. In the original formulation of the diffusion model, this vector field is assumed to be the score function (i.e. the gradient of the log- proo-abbbiiliyy at given time in the diffusion process). Curiously, on the practical side, most studies on diffusion models implement this vector field as a neural network function and do not constrain it be the gradient of some energy function (that is, most studies do not constrain the vector field to be conservative). Even though some studies investigated empirically whether such constraint will lead to a performance gain, they lead to contradicting results and failed to provide analytical results. Here, we provide three analytical results regarding the extent of the modeling freedom of this vector field. Firstly, we propose novel decomposi- tion of vector fields into conservative component and an orthogonal component which satisfies a given (gauge) freedom. Secondly, from this orthogonal decom- position, we show that exact density estimation and exact sampling is achieved when the conservative component is exactly equals to the true score and therefore conservativity is neither necessary nor sufficient to obtain exact density estima- tion and exact sampling. Finally, we show that when it comes to inferring local information of the data manifold, constraining the vector field be conservative is desirable.
INTRODUCTION
Generative models generate data from noise. To do so, most generative models learn a mapping from the noisy latent space the structured data space. Different mappings and different learning procedures lead to different models. For instance, for Normalizing Flows (Kobyzev et al.) 2021). this mapping is bijective and trained on maximum likelihood. In contrast, for Generative Adversarial Networks (GANs) (Goodfellow et al. (2020) this mapping is not bijective, and the training objective the Sensen-Shanon divergence between the model and data distribution. Recently, new class of generative models diffusion models have shown tremendous success in various domains, even outperforming GANs regarding the visual fidelity of resolutooo images (Yang al., [022). Unlike classical generative model that learns single mapping from latent to data space, diffusion models incrementally add structure by an infinite series of denoising steps.
Mathematically, this can be described using stochastic differential equations such that starting from structured data at time / = 0, the data is unstructured at time = 1. Crucially, this process can be reversed using the gradient of the true score function of the underlying stochastic process, that is, by using t) := V log p(x, t) for E 10 and x E R where p(x, t) describes the density at time t. Here, call diffusion model neural network sæ with parameters A which approximates s(x, t). Even though s(x, t) is the gradient of the scalar function log p(x, t), diffusion models are typically unrestricted in the sense that there is guarantee that 8q is the gradient of some scalar

---

Published as a conference paper at ICLR 2024
function, that is there is no guarantee that sø is conservative vector field. Several authors thus discussed the question whether So should be conservative or not by construction (Salimans & Ho
2021 Chao et all 2023 Lai et al., 2023 Wenliang & Moran 2022; Du et al. 2023 Cui et al. However, we argue that this is the wrong question to ask for the density estimation- and sampling ability of diffusion models. The right question to ask is if there is functional freedom in diffusion models such that instead of learning the true score s(x, f), it is sufficient to learn a broader class of vector fields s(x, f) + r(x,t). f). To do so, we will derive in section D non-trivial necessary and sufficient conditions for r(x, t) such that the corresponding generative model generates exact samples and learns the density exactly as well. We call this functional freedom gauge freedom in diffusion models! !
2022).
As a direct consequence of this gauge freedom, we show that conservativity is neither necessary nor sufficient for exact density estimation and sampling. To the best of our knowledge, this is the first theoretical answer to the question of whether a diffusion model should be conservative by construction not. Indeed, all previous work exclusively argued based on empirical evidence with contradicting results, as we will discuss shortly. Surprisingly, we also show that conservativity is sufficient when investigating local features of the data- mata-manifold, such as local variability. In section [S] we present a method for estimating the intrinsic dimensionality of the data manifold by analyzing the local variability when approaching the data manifold. Overall, our findings can be summarized in terms of two takeaway messages to practitioners using diffusion models:
1. For density estimation or sampling, there is no need to constrain the diffusion model to be conservative. However, for exactness, the gauge freedom condition needs to be fulfilled.
2. For analyzing local features of the data manifold, such as the intrinsic dimensionality, a conservative diffusion is guaranteed make the right conclusions.
Different authors studied the question of whether a diffusion model should be conservative or not. Salimans & Ho (202T) observed that, in terms of image generation, constraining sæ to be conserva- live does lead to a similar performance as having no constraints on So. To ensure that is conser- vative, (Salimans& Ho (202T) proposed to calculate the gradient of scalar function, which requires an additional backward pass and is thus computationally more demanding than directly modeling a vector-valued 8g. As a result of this observation, it is widely accepted that 89 can be unconstrained
without losing much of generality (Song et al. 2021: Salimans & Ho 2021 Yang et al. 2022:) Liu
jet al. 2022 Zeng 2023 Wenliang & Moran 2022). However, in some application domains, using
consistent score function (Arts et al. 2023 Neklyudov et al. 2023) may be more suited.
Despite the findings of (Salimans & Ho (021)). it is mathematically unsatisfactory to construct se, knowing that it is generally not consistent. This mathematical ghost is haunting the difffusion com- munity, which is reflected in a surge of recent papers addressing this conflict (Chao et al. 2023]
Lai et al. 2023 Wenliang & Moran 2022 Du et al., (023) Cui et al., (022) The corresponding conclusions are contradicting. For instance, while Wenliang & Moran (2022) report that non- conservative so learns vector field that constrains the samples to be within the data-manifold and thus only little sampling improvements can be expected by enforcing se to be conservative, Thus, only little sampling improvements can be expected by enforcing sy to be conservative, [Caa et al. (2023) observe that a non-conservative so may lead to degraded sampling performance. However, Chao et al. (2023) also observed that an unconstrained so may enhance the density estimation ability. Therefore, Chao et al.] (2023) and Cui et al. (2022) suggest implicitly enforcing conservativity by adding a penalty term to the usual objective function instead of explicitly modeling sy as a gradient of a scalar. This penalty term is V/Vs Vsg"'ll where Vsg is the Jacobian of so and || I|e is the Frobenius norm of a matrix. As a vector field parametrized by a neural network is conser- vative if and only if the Jacobian is symmetric under some mild conditions (Im et al.) 20166), this penalty indeed offers an incentive for so to be conservative without losing the architectural freedom of unconstrained 8p-
On the practical side, Du et al. (2023) and /aremi (2019) give additional reasons in favor of con- servative vector fields. Du al. (2023) compose several likelihood models into a new one through multiplication, division, or summation. The latter refers to mixture of distributions. However,
In electromagnetics. the electric scalar potential, and the magnetic vector potential are not uniquely defined but enjoy some freedom, called gauge freedom, see chapter 10.1 in Griffiths (2055) or Abedi & Surace (2099) for study on the gauge freedom in the context of non-linear filtering.
2

---

Published as a conference paper at ICLR 2024
discussed in Du al. (2023). can not mixture composition without explicit likelihood functions, that 18, without conservative vector fields in the case of diffusion models. In addition, conservative vector fields enable the use of more accurate numerical samplers such as Hamiltonian Monte Carlo (Duane et al. 1987 Neal 2011). Surprisingly, (aremi (2019) showed that for an un- constrained 8o to be conservative (thus being able learn exactly), the weights of the first hidden layer must be parallel to the weights of the output layer. The author concludes that "the neural net- work is required represent only one feature in its first hidden layer." providing a strong argument for explicitly constraining So to be conservative.
2 NOTATIONS AND BACKGROUND
The high-level principle diffusion models is to remove structure by adding noise incrementally in way that can be reversed (Sohl-Dickstein et al. 2015: Song & Ermon 2020). Recently, [Song al. (202]) unified different mathematical formulations of such models under the umbrella of stochastic differential equations (SDEs). For comprehensive overview of the young history of diffusion models and alternative formulations, we refer to Yang et al. (0022). For our purposes, we adapt the notations and concepts offSong al. (202I). which we will repeat in the following for convenience.
Let Xo E RD be random sample from the data distribution (x).. Let us further assume that this random sample serves as an initialization for the following stochastic differential equation:
dx:- = fx,, t)dt + g(t)dw;
where f R x [0, - R is a vector field, also known as drift, and g: |0, 1] R is the diffusion coefficient determining the magnitude of noise added at time as w, is D dimensional Brownian motion. The stochastic process (x+ (E,,1) is referred to forward process.
In principle, the drift f and diffusion coefficient can be chosen almost arbitrarily. However, as one data, f and g need to be chosen such that
wants to ultimately reverse the process and generate the limiting distribution pi is known and can be easily sampled from.
Sampling: Surprinsingly, (Song al. 2021) showed, based results from Anderson 1982), that one can write down the reverse process [x+)rejo.1] with starting distribution and limiting distribution po explicitly as a backward ODE using the gradient of log p. s(x, t) == V (log((x,)) t),
dx, = f(xx, t)dt with x1 ~ p(x. 1),
where f(x,,t) t) f(x,,)) f) (1)s(( 1). This allows one sample new data from p(, 0) by sampling from p(- 1) and solving the ODE backwards (that is from = 1 to 0).
Density estimation: Moreover, equation 2 allows for estimating the density exactly by making use of the instantaneous change of variables formula (Chen et al. 2018).
L. -(x(xx t)dt
log p(xo, 0) logp(x;;11)
(3)
where V. denotes the divergence operator. The latter applied on f(x,, t) is the trace of the Jacobian of f fxx,,)) t) which can be efficiently estimated through automatic differentiation (Paszke et al. 2017)
using the
trace estimator (Skilling) 1989; Hutchinson, 1990).
f(x,, 1) Tr (vî(x,, t)) Eewp(e) [ ('VV(xx, t)E
where, typically, p(e) (N, /). and V Vi(xx,, () the Jacobian of f(xe, 1). Note that equation (3) depends on the whole trajectory (x, ) drawn from
Therefore, once we can learn s(x, t) = V log p(x, t), we can sample new data through equation 2
and calculate the density explicitly through equation (3) (efficiently even in high-dimension thanks
to equation (4). see Han et al. (2055). Unexpectedly, to estimate s(x, t) using a neural network
with parameters 0, se, it is sufficient to estimate the conditional score log por (Xe (x)) a procedure
known as score matching (Hyvärinen & Dayan
2005 Song & Ermon 2019). With sufficient data
and model flexibility, have that 88- (x, logp(x, () for almost all and where
0" = argminEE.. { (A(t) Ex, [x]s)(Xe,)) t) V log Po: (x, xx)
(5)
8
3

---

Published as a conference paper at ICLR 2024
1] + R>0 is positive weighting function. The expectation Ex1x0 is the expectation over Por (X, Xo) such that given the drift and diffusion coefficient and g. respectively, efficiently at the one hand, and calculate V log Po: (x/|Xo) explicitly on the other hand. However, data is typically limited, and so is not arbitrarily flexible such that so will not match the true score s(x, t) after training. Hence, constraining Sg to be conservative or not can impact performance.
The time t is uniformly distributed on [0, t ~ 4(0. 1), and A(t) [0.
sample from this conditional distribution
3 GAUGE FREEDOM FOR EXACT SAMPLING AND DENSITY ESTIMATION
To sample data point with a diffusion model 8g, the initial value problem (IVP) needs to be solved. Let sy be a vector field of the form
5x t) = log,(x, t) + rx( t)
(6)
where re: RR X + RD summarizing the discrepancy between the learned vector field sq (x, ()
and the true score s(x, t) V log p(x, t). What is the gauge freedom of such that sampling with $8 (x, t) is equivalent to sampling with the true score s(x,)) t)?
An equivalent description of the underlying ODE in (2) in terms of the corresponding density p(x, t) can be derived using the Fokker-Plance equation or also known as Kolmogorov forward equation. This equation, without diffusion term, is given by
0p(x,t) = - (f(x))))(xxt)) Ot
This equation holds for every given point The change in density along a path {x, frejo.11 is given by the instantaneous change of variables formula from equation B] see appendix A.2 in Chen al. (2018) for details on how derive the instantaneous change of variables formula from the Fokker-Planck equation. We will discuss the difference be- tween equation
see appendix D.1 in Song al. 2021
( and equation
more detail in section D
Let the vector field corresponding to IVP (2) when using instead of s be denoted by
(t)ss(x,))) fx = f(x,t) (t)) t).
(8)
From that perspective, the above question can be reformulated as follows: What is the gauge freedom of re such that the evolution of p(x, 1) does not change when replacing / with fa? Suppressing the arguments to avoid clutter, standard calculus yields
0p -- (fr) f- - -v f
(9)
Öt
where we have used Vp = pV log the last step (we hence assume that p + 0). Now, replacing
by fo in equation 9 we have that the density will not change whenever
(((0))(()))) (5000))(xx)))) Vlogp(x,t) =
(10)
which is equivalent
V ror ro(x, t) logp(xxt)= V
(11)
as 2² is typically independent of and strictly positive. Therefore, whenever re fulfills equation 11 for all x E D and E [0, 1]. we have that and will lead to the same samples and densities since the evolutions of the corresponding marginal probability distributions are the same.
The gauge freedom condition (TT) yields unique decomposition of any square-integrable (with respect to the measure induced by t)) diffusion model 59 into a sum of conservative vector field and remainder term satisfying equation (1I.. see theorem [ This unique decomposition has some direct consequences, which we summarize in corollary T First, it shows that whenever the diffusion model s9 is conservative, the remainder must vanish, ro (x,t) = O. Second, it follows that exact sampling and density estimation is provided if and only if the conservative part is the true score, that so is given by equation (6)
4

---

Published as a conference paper at ICLR 2024
Theorem (Orthogonal decomposition) Let t 1]. For any vector field (p), there exists a unique conservative vector field Vo E (p), and a unique vector field r E (2))) fulfilling the gauge freedom condition A7 such that
v(x, V + r(x.1).
(12)
Corollary Let u E (p) with unique decompositions Vo and r such that u(x, t) = Vo(x, t) + r(x,t), with satisfying the gaufe freedom condition (1
(a) If((x,)) 1) is conservative, then must hold that r(x, =0
(b) v(x, t) provides exact density estimation and samples for the IVP 2 (replacing s by v) if and only if Vo(xxt) -V llo p(x,t).
Theorem 1 shows that the space of conservative vector fields in (p) is orthogonal to the space
of vector fields fulfilling the gauge freedom condition in (p). see figute This orthogonality provides a new intuition on the score matching loss in diffusion models. Let sp (x(x
xx () = Vos (x, ()
ra (x(x1) (p))). where ry satisfies the gauge freedom condition (1). then
[[|=(x,
!) so (x, 11
[[]s(x, () (x D E [Irr(xx
(13)
where the expectation is over x ~ p(x, t). Thus score matheing minimizes two terms. The first the right hand side of equation (13) is relevant (since when it is zero, correct sampling and density estimation can be obtained). The second term is irrelevant since it does not affect sampling and density estimation. However, for unconstrained 8o this second term will be generally different from O, see Saremi (2019). showing that unconstrained Sg cannot match s exactly.
Remark divergence-free remainder term is gauge freedom for the instantaneous change of variable formula derived in Chen et al. (2018), see section D in the appendix. Note, however, that this not sufficient for exact sampling and density estimation as opposed to equation 477). The instantaneous change of variable formula derived in Chen et al. (2018) describes an ordinary differential equation for p(xe; 1), that is, how p(x,; changes totally as a function of time. The Fokker-Plune equation (9), on the other hand, describes how p(x:: L) changes partially of time treating X2 as constant. The latter is a much stronger requirement ensuring that all vector fields with remainder term satisfying condition (11) correspond the same marginals and thus stochastically equivalent sample paths.
function
CONSERVATIVITY IS NEITHER NECESSARY NOR SUFFICIENT FOR EXACT DATA GENERATION AND LIKELIHOOD ESTIMATION
A direct consequence of the gauge freedom for diffusion models derived in section is that con- servativity is neither necessary nor sufficient for exact likelihood estimation or generating samples from the true data distribution, see figure To substantiate the theoretical framework with an empir- ical illustration, in this section we construct simple counter-example of vector field sp which not conservative but still satisfies the sufficient condition for exact density estimation and sampling, equation [T].
Let the target distribution be Gaussian with a diagonal covariance matrix. Then, by the additive closure of Gaussian distributions, the true score transforming a standard Gaussian to the target Gaussian must take the following form.
o (1) 0
s(x,, t) V llgg((x,, - X2, with DT¹
(14)
T (1))
where o1)) 0. Defining the remainder term as
=f(1)
tx(x Rxx, with R, (1)
(15)
it is easy verify the gauge freedom condition from equation (II: the one hand, ro is divergence-free as the trace of the Jacobian R2 is O. On the other hand, we have that
Mog,((x,0) = - x3 RT D X4 = -x (9) 1 x =
(16)
5

---

Published as a conference paper at ICLR 2024
Finally, note that re cannot be conservative since the Jacobian is not symmetric (Schwarz theorem). Therefore, we have constructed a simple counter- counter-example example proving that the gauge freedom condition, equation ( be satisfied without the necessity of šg to be conservative. This example can be straightforwardly generalized for higher dimensions.
Conservativity is also sufficient as, for example, (x(, - s(x,, t) would reduce the back- ward ODE from equation (2) to be defined solely by the drift term f. For f = O, such an ODE would only generate samples from the limiting distribution pi (x) dynamics are involved.
exact sampling and
exact sampling and density estimation
density estimation
gange freed. satisfying
gange freed. satisfying
v
r
V log p
U VΦ
conservative
conservative
Figure I: Every vector field v E L2(p) can be orthogonally decomposed into a conservative vector field Vá and remainder term that satisfies the gauge freedom condition given by equation (II). (A) Exact sampling and density estimation is obtained when the conservative component Vó of the vector field equal the true score (i.e. V v log which is the case for all the points the green dashed line. So v does not need to be conservative. (B) Even u is conservative, is not sufficient guarantee exact sampling and density estimation since may be different than the true score.
A CONSERVATIVE VECTOR FIELD IS DESIRED FOR EXACT LOCAL INFORMATION
In this section, we show how to estimate the intrinsic dimensionality of the data manifold when- ever 8g matches the true score Additionally, we provide empirical evidence that with non- conservative vector field, the ID is not estimated correctly while using the derived method we can estimate the ID correctly if sa is constrained be conservative (not necessarily matching ex- actly). This suggests that constraining the diffusion model be conservative should be preferred inferring local information.
A sample from diffusion model is the solution to initial value problem, see equation 2 We denote the solution of this IVP as (x1) where X1 ~ p(x, 1). Note that this solution is unique whenever f and globally Lipschitz in both state and time Øksendal (2003). How does this solution depend on the initial value x₁? Let s > O and e ~ N'0 7), then we have that
(x(x1 + SE) = =x((x)) + 0x1
+ O(s3)
and hence for -
(x + SE) - (x(xx) 4. Y (x;,;,;)));;))
(17)
where we have defined tx),
(x)) and 4 denotes convergence in distribution. Equation قх
(77 means that a diffusion model maps locally Gaussian distribution into a Gaussian distribution. Therefore, relate local information the manifold, such directions and strengths of

---

Published as a conference paper at ICLR 2024
forward SDE
dxs f(x,, g(C)iw:
X
X1
barkward ODE.
de.
[f(x,,
t) b9 (0)
P1
laareet distribann Pomiiiio
dutt--maniisdd
pertorised
Y;
Y2 Adl1
A6te X1
Au(d)
(mmanifold direesion;
(manifokl direction))
Au(d) - ne < (olf-maniiolld dirrctionn)
Ax(t)
manifold dircctionn
Figure 2: Intuition of how the singular values of tx (x1) evolve over time for low- dimensional Rata-manifold. The singular value in the manifold direction will saturate, while the singular values in the off-manifold direction will tend to 0 (bottom left).
dx;
variability, the singular vectors and singular values of Y (x1, 0), respectively, figure A similar result to equation (T7) was also observed for normalizing flows (Cunningham et al. 2022 Horvat & Pfister 2022), and exploited by Horvat & Piister (0022
normalizing flows for estimating the intrinsic dimensionality of low-dimensional manifolds. In the following, we want to estimate the ID similarly using diffusion models, For the sake of brevity, from now on, we drop the dependence of Y (x),t) on X and set Y == Y t))
Figure serves as illustration of the main idea. Starting from low-dimensional manifold, an embedded in R?, with density Pu, we gradually transform the data-density to standard Gaussian P1. To sample new data point, we first sample X1, and analyze how a vicinity of X1 evolves through the backward diffusion, we can study how the singular values of Y, change as function of time (bottom row). Crucially, since the data lives on low-dimensional manifold, the singular value of Y, associated with the off-manifold directions must approach zero when -> O. At the same time, the other singular value will converge to some fixed value. Indeed, the sensitivity to the initial condition of the backward denoising process is much larger along the "on-manifold" direction than on the "off- manifold"" direction. Therefore, we can study how the singular values Y, evolve function of time, the number of saturating singular values will correspond to the true intrinsic dimensionality.
Unfortunately, cannot access Y,. However, standard result from the study of ODE is that Y; is the solution of
dY, Vi(o,(xi),,)) Yo=1
(18)
where (f(d((x)) is the Jacobian of f(x,t) t) evaluated (dx (x1), t), Teschl (002)). This description of Y, allows us to express the singular values of Y, in terms of the eigenvalues of fx)x which we can calculate in practice when we approximate the gradient of the true score logp(x p(x, using diffusion model so sx(xtt). t). In the supplementary materials, prove the following theorem:
Theorem Let the data distribution p(-, be supported on low-dimensional manifold M of dimension d embedded in IR.". Let sy(x,t) E [p) be diffusion model trained on data from pl, 0) providing exact samples and density estimation. Let Px(x1) = (xx,,)Y (x),, 1)J have smooth eigenvalues in t for all X1 E D where Y (x1,,) 0x.(x.). Suppose that there exists an
8xj
0 for all E [0, that is. P((xi) and V (x( commute for all t E E). Then, we have that
0 such that [P((x)), V fe(cr
89 is conservative = rank eepp (vi(xo.0)) rank (exp
(19)

---

Published as a conference paper at ICLR 2024
where Xo (-((x1)
Theorem shows that the intrinsic dimensionality can be estimated using the rank of the Jacobian of fo. In the following, we will empirically confirm this.
Remark If both P. and fog, are diagonalizable on the one hand, and Pt and fa,, have the same eigemvectors on the other hand, we have that indeed [Pt, fo.z = 0. For sufficiently small the eigenvectors of V (xx will align with the normal and tangent space of the manifold, see discussion in Wenliang & Moran (2022). Also, (Permenter & Yuan 2023) support this hy- pothesis they show that denoising is approximately projecting close to the data manifold. As also the singular vectors of Pt will align with the normal and tangent space, the assumption that [P)())), Vfo(or(xi);)) f)] = 0, Vt E [0, for a sufficiently small s is reasonable.
Intrinsic dimensionality estimation using diffusion models: We consider a -i-dieensoonal Gaus- sian embedded in R as a toy example and a proof of concept. Thus, the intrinsic dimensional- ity of the data-manifold 2. We train conservative and non-conservative diffusion model using tx, 0 and g(t) 25' drift and diffusion coefficient, respectively. The non-conservative dif- fusion model is simply an unconstrained neural network sy (x, t) = t/e where the I2 x R>00 - RR The conservative version is sx(x, = V/l (x, t)]B as suggested by Du et al. (202)). In figure A, we see the evolution of the singular values as stated in theorem ass a function of time in log-log scale. Each color stands for 1 of a total of 5 singular values. If we use a conservative vector field (left plot), the singular values evolve as predicted; that is, 2 of them saturate, whereas the remain- ing diverge. All singular values saturate for the non-conservative vector field, and the intrinsic dimensionality cannot be estimated using the singular values of Y1. Although we only show the trajectories for one representative sample, we observe the same behavior across different samples
In figure 3 B, we show that our method scales with increasing embedding and intrinsic dimension. Our method perfectly matches the true intrinsic dimension for a sphere with dimension D/2 embedded in D for different values of D. We conduct more experiments in the supplementary on different manifolds (spheres, tori, swiss rolls) with different embedding dimensions and observe that the results do not change: conservative 8y can estimate the intrinsic dimension exactly whereas a non-conservative does not even if we increase the number of parameters used for S8 or add a penalty term enforcing conservativity by symmetrizing the Jacobian of sø as suggested by Chao et al. 2023) and Cui et al. (022).
Note that also Wenliang & Moran 2022 and Batzolis et al. (2022) estimate the intrinsic dimension using diffusion models. However, Wenliang & Moran (2022) does not come with any theoretical guarantee for estimating d correctly, and Batzolis al. (2022) does not estimate the ID correctly for spheres. Besides, our main motivation is to discuss the gauge freedom and conservativity question and their importance for correctly inferring local information. We did not focus on developing state-of-the-ate ID estimator, which is why we leave a thorough comparison of recent ID estimators based on neural networks (Horvat Plister [2022] Batzzoli et al. [022]] Wenliang & Moran 2022
Tempezyk et al. 2022 Mohan et al. 2019) for the future.
6 DISCUSSION
In this paper, we have argued that instead of asking whether a diffusion model should be a con- servative vector field (as required by the original theory) or not (as usually done in practice), better question to ask is if there exists a greater class of diffusion models without sacrificing exact- ness in both density estimation and sampling ability. Indeed, we have demonstrated theoretically that diffusion models enjoy gauge freedom for data synthesis and density estimation. As direct consequence of this gauge freedom, we have shown that conservativity neither necessary nor suf- ficient for exact density estimation or perfect sampling. To the best of our knowledge, this is the first theoretical answer to the conservity question, which was previously only addressed empirically with contradicting and unsatisfying results. Our theory also provides new intuition on the ccore-matching objective and confirms previous results that an unconstrained diffusion model will likely not learn the true score exactly.
We added some slack the curves for better display some overlap.
8

---

Published as a conference paper at ICLR 2024
Ag conservative Intrinsic Dim== 2
8g not couservative
ID estimation as a fimction of
Intrinsic Dim-= 2
10
curs
Singllar values (,( of Y
101
true ID
10-
ostimated dimension
10m
on-manifold
107
singular values
6
10"
3 oB. ua--nanaaa comeennnnn
aargular caloes
1077
10"1
10"4
10-7
107
3
6
10
20
10017
10 1011
10th
2013
201h
intrinsic dimension
time
time
Figure 3: A: Singular values of Y as predicted by lemma iin the appendix for se conservative (left) and con-conservatiee (right). Each color represents one singular value in total as the embedding dimension is 5). B: Intrinsic dimensionality estimation of sphere with dimension d = 1 embedded in D for different values of d.
In practice, enforcing the gauge freedom conditions may be challenging since, to do so, would need access to the true score function. However, since time-continuous diffusion models are trained to learn the transitional probability function Pos (x, Xo) for all times and data points Xg, equa- tion (5), one could add penalty terms to enforce the gauge freedom conditions accordingly. We leave this exploration for the future.
Finally, we derived the appendix lemma which relates the singular values of Y(1, xi) (which are unknown) to the singular values of fo (which can be calculated). for conservative vector fields only. As the singular values of Y, describe how small neighborhood of the initial value x1 evolves when applying the sample generating ODE, we have used this information to estimate the intrinsic dimensionality of the data-manifold. We have seen empirically that only if 89 indeed conservative, we obtain the right behavior as predicted by theorem Though one key assumption of theorem strong, namely that the commutator of Y,Y," and fe vanishes sufficiently close to the manifold, we demonstrated on different manifolds that, nevertheless, the singular values behave as predicted, and the true ID can be estimated. Therefore, we hypothesize that, indeed, the eigenvectors V fe and Y,Y, align close to the data manifold. Finally, the intrinsic dimensionality should be also estimated correctly if the remainder term ra of 80 + ro fulfills the gauge freedom condition. However, as discussed, this is difficult ensure in practice. Relaxing the conditions of theorem to accommodate for the general case is an interesting direction to pursue and might provide new insights on the gauge freedom condition.
As takeaway message, when using diffusion models for data synthesis or density estimation, con- servativity neither necessary nor sufficient, but the gauge freedom condition from equation (II) is necessary for the remainder term when the diffusion model is expressed sg = s However, when one is interested in inferring local information the data-manifold using diffusion models, we recommend working with conservative vector field such that the right conclusion can be made.
ACKNOWLEDGMENTS
We express our sincere gratitude to the anonymous reviewers for their meticulous examination of our manuscript and their valuable recommendations. We would like to extend special acknowledgment to Reviewer h3ie, whose insightful observations regarding gauge freedom significantly contributed to the formulation of theorem T and corollary [. see htttp:://opezeeview net net/forum? forum? 1d= F Io wweedddwwwwweMggree wwbeNMgr 9r for details.

---

Published as a conference paper at ICLR 2024
REFERENCES
Ehsan Abedi and Simone Carlo Surace. Gauge freedom within the class of linear feedback particle filters. In 20/9 IEEK 58th Conference on Decision and Control (CDC), pp. 666-671. IEEE, 2019.
Brian D. O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-33
May 1982. ISSN 0304-4149. doi: 10.
101600304-411400677000111
URL https: /wwp: sci cncieeneeiieet. cenceedirett.cooms........ www.:cceecccceeeeoo....mm.c
Marloes Arts, Victor Gareia Satorras, Chin-Wei Huang, Daniel Zuegner, Marco Federici, Ce- cilia Clementi, Frank Noé, Robert Pinsler, and Rianne van den Berg. Two for One: Diffu- sion Models and Force Fields for Grained Molecular Dynamics, February 2023. URL
http:// http///aiivv..../.iiv/..../...ia org/ abs 2302 2302000600 arXiv:2302.0000 [cs].
Georgios Batzolis, Jan Stanczuk, and knows the dimension of the data manifold. arXiv preprint arXiv:2212./26/// 2022.
Schönlicb. Your diffusion model secretly
Chen-Hao Chao, Wei-Fang Sun, Bo-Wun Cheng, and Chun- Yi Lee. On Investigating the Conserva- tive Property of Score-Based Generative Models, January 2023. URL http://axxvv.org abs/ 12753 arXiv:2209. 12753 [cs].
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems, volume 31. Cur-
ran Associates, Inc., 2018. URL / www.neerricc.o.ocoo http://s............paaaee///e/a
hash/ h22277/77766611...fe666422666633
Chenwei Cui, Zehao Yan, Guangshen Liu, and Liangfu Lu. Generalizing and Improving Jaco- bian and Hessian Regularization, December 2022. URL http://axxiv.oroo abs/2212.
00311 arXiv:2212.0m
[cs, math].
Edmond Cunningham, Adam D. Cobb. and Susmit Jha. Principal Component Flows. In Pro- ceedings of the 39th International Conference on Machine Learning, pp. 4492 4519. PMLR, June 2022. htt.itttts://ppoeeeeeeee ISSN: 2640-3498.
ml pwa.s.rees/vvl2 ngham222 html
Lokenath Debnath and Piotr Mikusinski. Introduction to Hilbert spaces with applications. Academic press, 2005.
Yilun Du, Conor Durkan, Robin Strudel, Joshua B. Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Grathwohl. Reduce, Reuse, Recycle: Com- positional Generation with Energy- Based Diffusion Models and MCMC, February 2023. URL
http://arxiv org/ abs/2302 11552 arXiv:2302. 11552 [cs, stat].
Simon Duane, A. D. Kennedy, Brian J. Pendleton, and Duncan Roweth. Hybrid Monte Carlo. Physies Letters B, 195(2)211-222. September 1987. ISSN 0370-2693, doi: 10. 1016/0377--2933377911911x URL https: /ttp://wwwwww.. www. www..ssiencediiett.coomm wttp://cicccttccocoocccccccc////
arricile/p1//33705338799933
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commun, ACM, 63(11);
139-144, October 2020. ISSN 0001-0782, 1557-7317. doi: 10.1145/3422622. URL https:
//d1 a.m. www. wcc..orrg/caaa//00.11147777222
David Griffiths. Introduction to electrodynamics, 2005.
Insu Han, Dmitry Malioutov, and Jinwoo Shin. Large-scale log-determinant computation through stochastic chebyshev expansions. In International Conference on Machine Learning, pp. 908- 917. PMLR, 2015.
Christian Horvat and Jean-Pascal Pfister. Intrinsic dimensionality estimation using normalizing flows. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems. volume 35, pp. 12225- 12236. Curran Associates, Inc., 2022. URL ht o httpaaaawwreeeeeeeeeaeeeeeeeeeee
file E11249955535200233444c3333333333
10

---

Published as a conference paper at ICLR 2024
M.F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian Communications in Statistics Simulation and Computation, 19(2):433-
smoothing splines.
450. January 1990.
ISSN 0361-0918.
doi: (0.18000336009100083 12866. URL https:
//doi org/ / 1080 00800/331100 091 9008812866)
Publisher: Taylor & Francis eprint:
https///di..org/10 108000361091900881
Aapo Hyvärinen and Peter Dayan. Estimation of non-nonnnee non-normalized statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005.
Daniel Im, Mohamed Belghazi, and Roland Memisevic. Conservativeness of untied auto-encoders.
In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016.
Ivan Kobyzev, Simon J.D. Prince, and Mareus A. Brubaker. Normalizing Flows: An Introduc- tion and Review of Current Methods. IEEE Transactions Patlern Analysis and Machine In- telligence, 43911):3944-3979 November 2021. ISSN 1939-3539. doi: 10.1 10.1109/PAM...000. 2992934. Conference Name: IEEE Transactions Pattern Analysis and Machine Intelligence.
Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Ste- fano Ermon. Improving Score-based Diffusion Models by Enforcing the Underlying Score Fokker-Planne Equation, January 2023. URL http: //axxiv/rrxvv..m wtg/ 04296 DiXiv:2210.04296 [cs].
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional Visual Generation with Composable Diffusion Models. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision - ECCV 2022, Lecture Notes in Computer Science, pp. 423-439, Cham, 2022. Springer Nature Switzerland. ISBN 978-
3-031-19790-1. doi: 10.1007/978-3-033 00117788---01-9999900 1.26.
Sreyas Mohan, Zahra Kadkhodaie, Eero P Simoncelli, and Carlos Fernandez-Granda. Robust and interpretable blind image denoising via bias-free convolutional neural networks. arXiv preprint
arXiv:1906.054730 2019.
Radford M. Neal. MCMC using Hamiltonian dynamics. May 2011. doi: 10. 10.1//11990 URL http://arxiv. www.oogg///b/////. /206 1901 arXiv:1206.1001 [physics, stat].
Kirill Neklyudov, Rob Brekelmans, Daniel Severo, and Alireza Makhzani. Action Matching: Learn ing Stochastic Dynamics from Samples, February 2023. URL http::/arxiv. org.vv.rrrga.... abs/
2210.06662 arXiv:2210.0665 [es].
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, E. Yang, Zach DeVito, Zeming Lin, Alban Desmaison, L. Antiga, and Adam Lerer. Automatic differen- tiation in PyTorch. October 2017. URL https: wwww: www.stisscoolall.a...moomm frrrpppperr///rttttttieraaaaarrr ot..ooos///i.....tttttoooooooo
b36a 222201100b0222023333311111111222 -ra
Frank Permenter and Chenyang Yuan. Interpreting and improving diffusion models using the eu- clidean distance function. arXiv preprint arXiv:2306.04848, 2023.
Tim Salimans and Jonathan Ho. Should EBMs model the energy or the score? April 2021. URL htttps://ppenev//// net
Saeed Saremi. On approximating / with neural networks. arXiv preprint arXiv: 2019.
John Skilling. The Eigenvalues of Mega-dimensional Matrices. In Skilling (ed.), Maximum Entropy and Bayesian Methods: Cambridge, England, 1988, Fundamental Theories of Physics, pp. 466.466.666666666 Springer Netherlands, Dordrecht, 1989 ISBN 978-94-015-78600 8. doi: 10.1007/ 466.
978-94--15-7860-5 48. URL http::://do...ogg/.1.....
01 7
48
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015.
11

---

Published as a conference paper at ICLR 2024
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribu- tion, October 2020. URL
org /bt.///bb////.00000
[cs,
stat].
Yang Song, Jascha Sohl-Dickster Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- Based Generative Modeling through Stochastic Differential Equations, February 2021. htRthhttp///xxxiie
13456 arXiv:201 1.13556 [cs, stat].
Piotr Tempezyk, Rafat Michaluk, Lukasz Garncarek, Przemystaw Spurek, Jacek Tabor, and Adam Golinski. Lidl: Local intrinsic dimension estimation using approximate likelihood. Interna- tional Conference on Machine Learning, pp. 11205-21233 PMLR, 2022.
Gerald Teschl. Ordinary Differential Equations and Dynamical Systems. American Mathematical Soc., August 2012. ISBN 978-0-8218-8328-J Google-Books-ie FZICAQAAQBAJ.
Li Kevin Wenliang and Ben Moran. Score-based generative model learn manifold-lis structures with constrained mixing. In NeurlPS 2022 Workshop on soore-Based Methods, 2022.
Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion Models: A Comprehensive Survey of Methods and Applications, October 2022. /ttL:httt///aiiii http:p///aaxv...or////////0...0 00796 arXiv:2209.00790 00796 [cs].
Weili Zeng. How Construct Energy for Images? Denoising Autoencoder Can Be Energy Based Model, March 2023. URL http:://rriw.orgg//eeaea abtrg//bb//3333 03887 arXiv:2303.03887 [cs].
Bernt Øksendal. Stochastic Differential Equations. Equationn Universitext. Springer, Berlin, Heidelberg, 2003.
ISBN 978-3-540-04758-2 978-
doi: 10.1007/998---6442- 14394-6. URL http:
7/link. waink.ssringer.coom http:///..eeeeccomooo./0///////0
A
PROOF OF THEOREM
Let u E (p), that is
|| == Exxp(x.) [x²(x)] = f v² (x))((( t)dx < oc.
(20)
Note that (p) is a Banach space with a scalar product inducing above norm. This scalar product allows to define the orthogonal complement of the subspace of conservative vector fields in (2(p)) As we will see below, this complement is exactly the space of vector fields fulfilling the gauge freedom condition. Finally, this complement also closed subsef and thus, Banachs projection theorem guarantees the desired unique decomposition, see theorem 3.6.6 Debnath & Mikusinski 2005).
What is left to show is the aformentioned orthogonality. Let V E (p),), and E (²(p) fulfilling the gauge freedom condition (IT). We have that
(V
Ex-p(x,) Vo(x t))
Vo(x, t)(x (xx, t)dx
RD
Φ(x, () r(x, 1) + r(x, ()T log logp(x, l))xx
(21)
where in the second to last equation we have use the integration by parts formula. Thus, V is orthogonal in the Hilbert space L (p). O
proof of this standard result from the study of Hilbert spaces can be found in Dehnath & Mikusinski theorem 3.6.2.
2005
12

---

Published as a conference paper at ICLR 2024
B PROOF OF COROLLARY I
Corollary (a) is a direct consequnce of the uniqueness the decomposition. One direction of corol- lary (b) is shown in the beginning of section (3). There, we have shown that if the conservative part of e indeed matches the true score, then e provides exact samples for the IVP
Now, we assume that v provides exact samples for the IVP
2 Thus, the difference between v and the true score needs to fulfill the Gauge freedom condition
that is
Vo(x, t) - V log t) + r(x,t)
(22)
fulfills equation [T]. However, /' already fulfillls equation (II), and conservative vector fields are orthogonal to vector fields fulfilling equation (IT). Therefore, the conservative part Vo(x, t) v log p(x, 1) needs to vanish. With other words, must hold that Vo(x, log p(x, 1) which was left to show. M
C PROOF OF THEOREM 2
As we assume that is conservative and yields exact sampling and density estimation, corollary implies that $g = s. Thus, we have that fo = f. What is left to show is that the rank of the matrix exponential exp (x)xx converges to d. To do so, we will relate the singular values of
Pt with the cigenvalues of fx(x, through lemma 1 Note that the rank of lim Y,Y," where (x1x)) (x1) must
di (x1)) is the solution to the IVP from equation (2) and Po = YoY"" defines the local variability on the manifold (which d-dimensional) see equation (17). the rank of Po is given by the number of non-zero singular values of Yo. we will see how the aformentioned relation allows us to estimate d by the number of non-exploding eigenvalues of lim,- V f t), or
0x,
cquivalently: the rank of exp ( fv(xx00) 0)).
Lemma 1 With the same assumptions as in theorem let V ((())(x)).) have eigenvalues μu (t) S (D))), then the eigenvalues 0 S (t) S (( of Pi Pi(xi) are given by
(1) Ax(e) exp (2/ /2// mi(s)ds)
(23)
for all t [0,e]
Proof of lemmal 1 The singular values of Y((x)) are given by the eigenvalues of Y (x₂ (x1)) We simply write Yú instead of (xxx) in the following. Note that P. Y,Y, has the same eigen- values as Y, (but not necessarily the same eigenvectors). Let p; be an eigenvector of P. with eigenvalue X; * 0 (see lemma 2). that is P.P: A.p. Then, taking the time derivative on both
sides of the eigenvector equation, we get
P.p. + P.p. = A.p + Aap;
(24)
Note that every symmetric matrix has an cigenvector decomposition consisting of orthonormal
cigenvectors. In this context, P. is either orthogonal to p. (hence eigenvector of P) P. 0.
In both cases we have that (P.. p.) = 0. Therefore, we multiply both sites of equation 24 with
p." from the left, we have that
(p.1Pp.) A
(25)
Note that
P Yx + Y. = VjP.+
(26)
since Y, f.YY where V == (x), t), and thus for the transpose holds Y;7 = -
YTVV.. (note that f. is symmetric as sa is conservative by assumption). Introducing the com-
mutator [P,, f == V.P.P P.V fc. which is 0 for all E [0, by assumption, we can further simplify above equation for E [0
P = [P, V 2P.V 2P.
(27)
13

---

Published as a conference paper at ICLR 2024
Note that [Ps, fil 0 implies that Vfps is eigenvector of Pt P.V/iP, VfiP.p: X, f.P. If X # in equation (25 then have that ftp: μaP; for some 14 BR) {R) oth- erwise VfPP jaPj for some o and hence (p. P.P. 244 (P: which is a contradiction to A, # O.
A; 0, however, then we must have for all that ViP: is an element in the space spanned by all eigenvectors except p.. In other words, V change-of-basis with a permutation matrix as change-of-basis matrix. However, such transformation cannot be symmetric which we have assumed for f..
Therefore, we have that X # 0 and VfP. p4Pi
Then,
P.p:
2P. 2A:1.P:.
(28)
Finally, inserting this into equation
25 we have that
À (pilPaps)
À (p:12\.u.p.)
A 2p1A;
2/13
A;
In In(Aa) 2p
dt
1
( Ax(5) exp (2 (2/, (s(s)ds)
(29)
Note that we for the third step, we need that x + 0 which we proof below in lemma the proof. D
This ends
Lemma The eigenvalues (t) are non-zero for all 2 0.
Proof: Liouvilles formula for the determinant of the matrix solution, see lemma 3.11 2012), to the ODE
Teschl
dY, ())
Yo =
(30)
states that
det Y, det Yu exp ( (V((O((())))))))
(31)
The trace of f(xx(x)) is given by Ex (t), and the determinant of Y. by (t) The right- hand side always non-zero. Therefore, each factor the left-hand side non-zero. This is what we wanted to show. D
Finally, we finish the proof of theorem 2 The rank of is d by the characterisation of P. through equation 17 On the other hand, the rank is given by the number of non-zero eigenvalues of P.. These eigenvalues can be calculated using the eigenvalues of Vi.. see lemma 1 Thus, (D d) eigenvalues of V f must converge -co for / 0 which corresponds to the rank of exp V f0.o which is what we wanted to show. D.
D IF EXACT SAMPLING IS PROVIDED, HELMHOLTZ DECOMPOSABILITY IS SUFFICIENT FOR EXACT DENSITY ESTIMATION
In the previous section, have derived gauge freedom for diffusion models expressed in equation (TO). By initially considering the ODE formulation of the sampling procedure, we have exploited the equivalent description of sample trajectories in terms of the underlying marginal probability
14

---

Published as a conference paper at ICLR 2024
densities given by the Fokker-Plande equation. The close relation between sampling and density estimation is not surprising as evaluating the density, see equation (3). requires knowledge of the en- tire sample trajectory. In this section, show that if the model generates exact samples, Helmholtz decomposibility is sufficient for exact density estimation.
Let so be given by equation (6) with rx t) being rotation field (that is vector field with V ra (x,,t)) = 0). Replacing the true score by 8a (x, t) for evaluating the model likelihood in equation
Bh, will lead to the same likelihood because
Tr [Vsp(x,, t)) Tr (v2 log t)) Tr(Vre(X,, t)) Tr (V² (xx,, t)) V re (x),,)
( (x,, 1)) as the trace of the Jacobian of ro is equal to the divergence of re which is 0 for all rotation fields. Therefore, for given path (x.))ej the diffusion model se
which results
as defined in equation (6) and the true score yield the same density when using equation (3) to
estimate (x)))), no matter how close 89 is to the true score.
E INTRINSIC DIMENSIONALITY ESTIMATION
As mentioned at the end of Section 5.1 of the main text, we perform more experiments for estimating the intrinsic dimensionality.
For the non-conservative diffusion model, we simply use a standard feed-forward neural network where we first embed the data into 100 dimensions and linearly transform it followed by non- linearity (first step). Further, we embed the resulting features into 200 dimensions, again linearly transform it followed by non-linearity, and finally project back into the data dimensions (second step). We embed the time into 100 dimensions using Gaussian-Fourier projection and add these embeddings to the features after the first step. The conservative version additionally takes the gradi- ent of the corresponding L.2 nonoorm with respect to the inputs.
In figure and show the evolution of the singular values (in log- scale) function of time for the Swiss Roll, Sphere, and Torus embedded D on the left and for embedding dimension D 5 on the right, respectively. On each side, we show the evolution for both conservative and not conservative diffusion model sg. The number of lines corresponds to the embedding dimensions D this is the number of singular values of Y. We can see that for sæ conservative, always 2 of in total D singular values saturate when approaching the manifold (that is when - 0). However, the remaining singular values do not saturate and tend to -o,, that the singular values tend (confirming the intuition from the main text). For sy not conservative, however, all singular values saturate showing that 80 does not behave as predicted close to the manifold. Even we add penalty term the Jacobian enforcing symmetry and thus conservativity, suggested in Chao et al. (0223). we observe the same scaling behavior.
15

---

Published as a conference paper at ICLR 2024
Torus
D
D = 5
Sp conservative
86 not conservative
Sy conservative
Sy not conservative
Intrinsic Dim=
Intriasic Dim=
Intrinsic Diu==
Intrinsic Din==
10
Singular values (t) of (t)
100
10a
> 30
Singular values
10
10
10
30
10:3
10
time
Figure 4: Singular values trajectories of as torus for different embedding dimensions (D = 3 and
D 5). We show the evolution of both conservative and not conservative diffusion model.
16

---

Published as a conference paper at ICLR 2024
Sphere
D D=3
=
#e conservalive
8o not conservative
80 conservative
80 not conservative
Intrinsic Dim=
Intrinsic Dim= 2
Intrinsic Dim= 2
Intrinsic Dim=
Singular values (t) of Y (tt)
Singular values (f) of BYY
101
107
10a4
10
time
10
time
10
10
10a
10-2
10"2
10
10
10-2
10
Swiss Roll
D
D
Se conservative
Sp not conservative
Sg conservative
Sy not conservative
Intrinsic Dim
Intrinsic Dim
Intrinsic Dim
Intrinsic Din=
10
100
Singular values (t of (t)
Singular valuis A:(t) of Y
10
100
10:1
100
10-2
10
107
10-1
10
10-1
10-5
10-1
10
10
10
10
10
10
10
hime
time
time
Figure 5: Singular values trajectories of the Swiss Roll and sphere for different embedding dimen- sions (D = 3 and D 5). We show the evolution of both conservative and not conservative diffusion model.
17


Reviews:
[{"review_id": "bCiUcEzNNZ", "reviewer": "Reviewer_KRsj", "summary": "This paper explores the conservativeness of the score vector field in the context of diffusion models. The author show that density estimation and sampling can be exact even if the learned score vector field is non-conservative, as long as the gauge freedom condition holds. This condition enforces that the remainder vector field (i.e. the gap between the true and learned vector fields) must be both divergence-free and orthogonal to the true score everywhere. Two dimensional empirical examples are presented to support the derivations. Moreover, the authors show that for exact estimation of of intrinsic dimensionality of the underlying manifold, a conservative vector field is sufficient.", "questions": "In section 4.1 a two dimensional example is discussed to show that conservativeness of score is neither necessary nor sufficient. The true score in this example has a divergence of zero which make it quite trivial. Is this an inevitable consequence of conditions 12a and 12b in 2 dimensions? In other words, doesn't satisfaction of gauge freedom conditions imply that the true score must have zero divergence in 2D? Since the authors use this example to make a more general conclusion in higher dimensions, it might be more convincing if 1) it is shown whether an example exists where the true score divergence is not zero yet the conditions are satisfied 2) if such example exists, that could replace the current example so that the generalization to higher dimensions would be more realistic.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is written fairly clearly with a nice flow. The topic of conservativeness of the learned vector field in the context of diffusion models or denoising is still an open-ended question, with important empirical and theoretical implications. Proposing the gauge freedom condition as a valid relaxation of conservativeness requirement is an interesting and fairly significant theoretical contribution.", "weaknesses": "In my opinion, the main weakness of the paper is that it cannot yield to real world practical results. More specifically, in order to test or enforce the gauge freedom condition (eq 11 or conditions 12a and 12b), one needs to have access to the true score, which is not generally possible. In fact, diffusion models are mostly used for cases where the true score is not known. So it is not clear how knowing this condition would solve the question of conservativeness. The empirical results given in the paper are all toy examples for this very reason. \n\nThe result of section 4 states that for exact density estimation only condition 12a needs to be satisfied. In other words, if the discrepancy between the true score and the learned score can be written as a rotation then the density can be estimated precisely from eq 3. However, this result also assume that the model generates exact samples, which is not clear how can be tested in real cases such as image densities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jwbeNMqn9r", "reviewer": "Reviewer_h3ie", "summary": "This paper investigates whether learning a conservative vector field is a necessary and/or sufficient condition to perform various generative modeling tasks, namely, sampling, density estimation, and dimensionality estimation. For the first task (and maybe the second one, see below), the authors show that it is slightly different condition which is necessary and sufficient. For the third task, the authors show that being conservative is sufficient.", "questions": "I also have been thinking about score conservativeness for a while. I thus make several suggestions which I hope can help the authors improve the paper (which is already interesting and quite clear!).\n\n### On Helmholtz decompositions\n\nI did not find equations (12a) and (12b) to be helpful, but rather confusing. Rather than the Helmholtz decomposition, the right decomposition is that any vector field $s \\in L^2(p_t)$ can be decomposed uniquely as $s = \\nabla \\phi + r$ where $\\phi$ is a scalar field and $r$ satisfies the gauge freedom equation (10) (this decomposition then depends on $p_t$). Furthermore, this decomposition is orthogonal, in the sense that $\\mathbb E[||s(x_t)||^2] = \\mathbb E[||\\nabla \\phi(x_t)||^2 + ||r(x_t)||^2]$. This is readily checked with the integration by parts identity $\\mathbb E[\\nabla \\phi(x_t) \\cdot r(x_t)] = -\\mathbb E[\\phi(x_t) \\left( \\nabla \\cdot r(x_t) + \\nabla \\log p_t(x_t) \\cdot r(x_t) \\right)]$ for any vector field $r$ and scalar field $\\phi$, which shows that conservative fields $\\nabla\\phi$ are orthogonal in $L^2(p_t)$ to vector fields $r$ which satisfy (10). More abstractly, the adjoint of the gradient operator $\\nabla$ is the \"gauge freedom\" operator $-(\\nabla + \\nabla \\log p_t) \\cdot$, showing that the image of the former is the orthogonal complement of the kernel of the latter.\n\nI believe that stating this decomposition helps making the paper clearer. In particular, it has several direct implications for the paper:\n- Because of the orthogonality property, the score-matching loss can be decomposed $\\mathbb E[|| \\nabla \\log p_t(x_t) - s_\\theta(x_t) ||^2] = \\mathbb E[|| \\nabla \\log p_t(x_t) - \\nabla \\phi(x_t) ||^2 + ||r(x_t)||^2]$. This shows that (denoising) score matching training naturally regularizes the $r$ part of the score to have a small norm (but it cannot be zero, see above). We also see that $s_\\theta$ is conservative if and only if $r=0$.\n- It simplifies the proof of Theorem 1 and removes an unnecessary assumption. Indeed, the assumption that $s_\\theta$ is conservative is equivalent to $r = 0$ (at all times!). There is thus no need to assume that the divergence of $r$ is finite.\n\n### Typos\n\n- \"scaler\" bottom of page 2\n- \"satisfie\" middle of page 6\n- \"boundery\", top of page 8\n- \"symmyrizing\", bottom of page 8\n- \"conservity\", middle of page 9\n- \"freemdom\", bottom of page 9", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper tackles an important issue on which there is both interest and confusion in the community, as there have been several conflicting studies on the topic of conservativeness of learned score vector fields. Section 3 is particularly well-written and makes an important point that had been overlooked in the literature. The numerical experiments in Section 5 are also very convincing.", "weaknesses": "I have several comments:\n\n- Though the $r$ part of the learned score does not influence the marginals along the ODE trajectory in _continuous time_, it might do so when the ODE is _discretized_. In that case, it seems reasonable that \"straighter\" trajectories would be easier to discretize, and that these would correspond to $r = 0$, thus giving another practical motivation for regularizing the network to be conservative.\n\n- I am slightly confused by section 4. It seems to me that a subtle point when evaluating densities is that the score $s$ appears both explicitly in the integral in (3), but also implicitly through the trajectory (3). Therefore, it seems hard to evaluate exactly the effects on the modeled density. Besides, it seems that the paper make conflicting statements about the result: is the condition that $r$ satisfies (10), or that it satisfies both (12a) and (12b) (which are stronger than (10))? Different parts of the paper state different results (e.g., abstract and Figure 1 vs section 4). It seems to me that $r$ satisfying (10) is sufficient, since the samples have the correct distribution, and the change of variable formula still holds for $s$, so (3) must compute the correct density.\n\n- Theorem 1 should state that the assumptions imply that $r$ vanishes when $t \\to 0$ (as shown in the proof), which makes it easier to understand the theorem. Also, the assumption that the divergence is finite may be questionable (why would $r$ remain stable when $t \\to 0$ when the score explodes?), and should be discussed. In the proof, I don't see why the finite divergence of $r$ should imply that $r_\\theta^T\\nabla \\log p_t \\to 0$. Fortunately, these problems can be solved, the assumption removed, and the proof simplified. In fact, as it stands, the theorem is misleading, as assuming that $s_\\theta$ is conservative implies that $s_\\theta = s$ and $r =0$, and it is thus a statement about the true score rather than an approximation of it.\n\n- Let me suggest two additional references. [1] studies the approximation power of unconstrained conservative neural networks (not parameterized as the gradient of a scalar function). Essentially, exact conservativeness implies that the network only depends on a one-dimensional projection its input [1, Theorem 6]. It explains why $r$ will never be exactly zero in practice, unless one explicitly parameterizes the score as the gradient of a scalar function. This should be mentioned in the introduction together with the literature review on the topic. [2] studies the intrinsic dimensionality of image manifolds using the singular values of the network Jacobian and is thus very much related to Section 5. An observation that might be of interest to the authors is that for natural images the dimensionality _increases_ when the noise level vanishes, as opposed to what one expects for a low-dimensional manifold.\n\n[1] Saremi, Saeed. “On Approximating $\\nabla F$ with Neural Networks.” arXiv, November 6, 2019. http://arxiv.org/abs/1910.12744.\n\n[2] Mohan, Sreyas, Zahra Kadkhodaie, Eero P. Simoncelli, and Carlos Fernandez-Granda. “Robust and Interpretable Blind Image Denoising via Bias-Free Convolutional Neural Networks.” arXiv, February 8, 2020. http://arxiv.org/abs/1906.05478.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sPc0fOIOQV", "reviewer": "Reviewer_ovjk", "summary": "This paper investigates an additional degree of freedom that appears in the approximate learnt score function in a diffusion model when one relaxes the constraint that the score function must be a conservative vector field.\n\nThey do this by analysing the modified score function $s(x, t) = \\nabla \\log p_t(x) + r(x, t)$.\n\nThey show that under the constraints of 1) $\\nabla \\cdot r(t, \\cdot) = 0$, $r$, is conservative and 2) $\\nabla \\log p_t(x) \\cdot r(x, t)=0$, $r$, is orthogonal spatially to the score, that when initialised with the same initial distribution, the evolution of the distribution of the SDE evolved through time with and without the additional $r$ term remains the same.\n\nThey then apply reasoning from this conclusion ideas around density estimation, data generation, and local information estimation.", "questions": "- Do the authors have an intuitive interpretation of the conditions in 12? \n- How can one practically realise the constraints of equation 12? Imposing the orthogonality constraint in combination with the divergence free part seems as difficult as just estimating the exact score.\n- If one takes no drift (i.e. the ode does not move the distribution), then it becomes clear that the constraints force the dynamics to be such that 1) particles are constrained move along the same level-set of the distribution (i.e. through time a particle remains at $p_t(x)=c$ for all $t$), the orthogonal constraint, as long as 2) the dynamics do not alter the density of those level sets, the divergence free constraint. Is this perspective interesting when combined with a non-zero drift?\n- I do not understand the point of the first part of section 4. I understand the argument that given a path $x_t$, if one adds a divergence free term to the score preserved density estimation. However, why would one assume they have a path? Without the exact score, or one satisfying the additional orthogonality constraint, how would one be able to generate a correct path from the model?\n- Section 4.1, it would be highly instructive of the authors to plot examples of the trajectories with and without the additional $r$ term I think. From my own plots, one ends up with a series of trajectories that at each instance are \"spinning\" around the mean, following the shape of the covariance ellipses. One can multiply the $r$ term by a constant to speed up said spin. This related to my previous point also. Even if one initialises the ODE with the reference measure, the particles can still move as they can follow the covariance ellipses at constant speed on each ellipse.\n- The experiments of section 5 do not seem to be related to theorem 1. Theorem 1 says that you can correctly estimate the data dimensionality if the extra $r$ field satisfies the gauge freedom constraints. However the experiments only use a fully unconstrained score, and a score constrained to be conservative. There is not experiment showing that one can estimate the dimension correctly if the vector field is non-conservative, but does satisfy the gauge constraints.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The topic discussed is an interesting one, and would have impact on how people train score based models in practise for the specific task of dimension estimation.\n\nThe presentation of diffusion models in section 2 is clean.\n\nThe main result of section 3 is interesting.", "weaknesses": "- While the analysis in section 3 is interesting, there is no sense of how one could exploit these conditions in practise.\n- The first half of section 4 to me does not make sense from a practical perspective (see questions)\n- The experimentation to back up section 5 is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lyW3ryFsem", "reviewer": "Reviewer_pZCv", "summary": "Score-based generative models achieve state-of-the-art performance in image generation tasks, and are based on fitting a model to the gradient of the log likelihood of blurred data. As such, it would be natural to fit models which are themselves conservative (the gradient of a function), but in practice this constraint is not enforced at all, and instead general vector fields, parametrized by a deep model, are used. This discrepancy leads to a natural question that is addressed in this work - how does learning a possibly non-conservative vector field affect the performance of score-based generative models? The authors derive an equation that precisely addresses this question (they refer to this as a gauge invariance), and show that it can be obeyed by both conservative and non-conservative vector fields. They then consider the problem of intrinsic dimension estimation and show that a conservative vector field -- under some technical assumptions -- is sufficient to estimate the intrinsic dimension of the data. Finally, they provide some empirical results on synthetic data demonstrating their ability to recover the intrinsic dimension with a conservative vector field, and an inability to recover intrinsic dimension with a non-conservative vector field.", "questions": "- I suggest you formalize the gauge invariance principle\n- As mentioned above, the relationship of the empirics, Theorem 1, and the gauge invariance principle are confusing. It would be good to study and discuss more how these results complement (or don't) one another.\n- For Figure 3: I don't see why the true singular values must be 1. They should depend on the local geometric properties of the manifold, the true data density, and the map.\n- For Figure 3: the colors are very difficult to make out. Please use fewer lines (e.g. throw out 2 of the 3 non-material singular values) and more visual differentiation (e.g. different line types).\n- For Figure 3: as it is constructed (and from what I can tell due to the difficult of making out the details, see above), it seems there are more than 2 singular values (green, black, yellow) converging to 1, which seems to contradict Theorem 1 and the discussion in the first paragraph after \"Intrinsic dimensionality estimation using diffusion models\" where it is said \"2 of them converge to 1, whereas the remaining 3 diverge\"\n- There are many typos.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper studies an important and relevant problem and derives a simple, clear, condition to precisely characterize when a vector field leads to the same marginal distributions as the true score. Using this equation, they cleanly show that conservativity is neither sufficient nor necessary, theoretically answering a question which has been studied empirically in the literature.", "weaknesses": "The derivation that leads to their main finding, the gauge invariance in equation (11), is not especially novel, since it follows immediately from the definition of the reverse time process. Also, their main finding - if I understand correctly, that gauge invariance as in (11) is necessary and sufficient for the time marginals of the learned process to be identical to those of the true process - is not formally stated as a mathematical result. I think the paper would be helped by a clear and precise mathematical statement of this main finding.\n\nAlso, Theorem 1 on the intrinsic dimension estimation problem states that conservativity implies their intrinsic dimension estimation procedure is consistent (under their assumptions). But their gauge invariance finding (equation (11)) states the law of the reverse time process is unaffected under this gauge invariance, so it would be more natural to expect that the intrinsic dimension estimation procedure is consistent under the gauge invariance (so that conservativity is not necessary). And then in the discussion about Figure 3, the authors claim that their empirics demonstrate that conservativity leads to accuracy of the intrinsic dimension estimation procedure, while models which aren't constrained to be conservative have inaccurate intrinsic dimension estimation. But this is an odd finding give that their main observation is this gauge invariance, so that in particular, we would expect that enforcing conservativity is not important. There is no formal contradiction here, since the trained models which aren't enforced to be conservative may not satisfy the gauge invariance, but it does seems to somewhat contradict the essence of the paper. Finally, the empirics in Figure 3 (they are difficult to read so it's possible I am mis-interpreting them) seem to be inconsistent with their discussion of them and their Theorem 1 (see below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)