Hessian Geometry of Latent Space in Generative Models
Authors: Alexander Lobashev
Dmitry Guskov
Maria Larchenko
Mikhail Tamm
Venue: ICML 2025 poster (2025)
Keywords: fisher metric
hessian metric
diffusion models
generative models
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: This paper presents a novel method for analyzing the latent space geometry of generative models, including statistical physics models and diffusion models, by reconstructing the Fisher information metric. The method approximates the posterior distribution of latent variables given generated samples and uses this to learn the log-partition function, which defines the Fisher metric for exponential families. Theoretical convergence guarantees are provided, and the method is validated on the Ising and TASEP models, outperforming existing baselines in reconstructing thermodynamic quantities. Applied to diffusion models, the method reveals a fractal structure of phase transitions in the latent space, characterized by abrupt changes in the Fisher metric. We demonstrate that while geodesic interpolations are approximately linear within individual phases, this linearity breaks down at phase boundaries, where the diffusion model exhibits a divergent Lipschitz constant with respect to the latent space. These findings provide new insights into the complex structure of diffusion model latent spaces and their connection to phenomena like phase transitions.
Our source code is available at \url{https://github.com/alobashev/hessian-geometry-of-diffusion-models}.

Hessian Geometry of Latent Space in Generative Models
Alexander Lobashev
Dmitry Guskov Maria Larchenko" Mikhail Tamm
Abstract
Latent Space
Image Space
This paper presents a novel method for analyzing the latent space geometry of generative models, including statistical physics models and diffusion models, by reconstructing the Fisher information metric. The method approximates the posterior distribution of latent variables given generated samples and uses this to learn the log-partition function, which defines the Fisher metric for exponential families. Theoretical convergence guarantees provided, and the method is validated the Ising and TASEP models, out- performing existing baselines in reconstructing thermodynamic quantities. Applied to diffusion models, the method reveals a fractal structure phase transitions in the latent space, characterized by abrupt changes in the Fisher metric. We demonstrate that while geodesic interpolations approximately linear within individual phases, this linearity breaks down at phase boundaries, where the diffusion model exhibits divergent Lipschitz constant with respect to the latent space. These findings provide new insights into the complex structure of diffusion model latent spaces and their connection phenomena like phase transitions. Our source code is available at https://gite com hessian-geometry--ee wfw.rroooooo--ffoo----o-oooooooo
p(xlt))
sampling from generative model
p((|x)
p(xit)
x
estimating posterior distribution from a single sample
p( Xx)
p(x(t)
posterior estimation approach
p[t|x,, ... Xx) depends only on
log Z(i
haaeemmalobar alobashev/
Figure 1: Visualization of Theorem 3.1. Having ples from p(x[(') we can approximate posterior distribution p[t|x1,... .N The partition function log Z(t) defines Hessian metric on the latent space. Our Theorem 3. guar- antees that log Z(t) limit depends only on the posterior distributions p(t|xr,,... ,N) and Theorem 3.2 offers a way to learn log Z(t) from p(N)..
1. Introduction
ffate-of-the-are image generation models often exhibit abrupt changes in image appearance during interpolation, indicating non-smooth latent space (Liu et al., 2021; Guo et al., 2024). These abrupt transitions have been studied in
community from two different perspectives.
Riemannian geometry of latent space Recent work by Park et al. (2023) constructs latent basis by considering singular vectors of Jacobian in the feature space, deriving metric on a latent space via pullback from the euclidean met- ric in the feature space. A similar pullback-bard based approach from euclidean metric in image space was used by Shao et al. (2018). It was demonstrated that linear interpolation in the latent space closely approximates geodesic interpolation.
GGam AI, San Francisco, USA 2 Artificial Neural Computing Corp.. Weston, FL, USA "Magicly AI, Dubai, UAE "School of Dig- ital Technologies, Tallinn University, Tallinn, Estonia. Correspon- dence to: Alexander Lobashev clobashovalexandere waaiiremaaaal....... Dmitry Guskov <guskov01 @ gmail.com>.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Hessian Geometry of Latent Space in Generative Models
Arvanitidis et al. (2017) studied the Riemannian geometry of GAN latent spaces, deriving metrics under the assump- tion that the stochastic generator's mean and variance func- tions are twice differentiable. Other notable study (Li et al., 2024) proposes an approach to automatically discover inter- pretable directions in a latent space. In (Brown et al., 2022) authors verify the union of manifold hypothesis for various image datasets. Additionally. Tosi et al. (2014) treats latent variables coordinates on Riemannian manifold, using the Jacobian-derived metric to construct geodesics in latent spaces obtained through nonlinear dimensionality reduction.
exhibits a divergent Lipschitz constant with respect the latent space at phase boundaries.
2. Background
Let X' be a high-dimensional data space and S a lower- dimensional latent space. We assume the existence of a stochastic generative mapping from S to X, defined by the conditional probability distribution p(z[t) on X' for each latent vector t E S.
A generative mapping could be a statistical physics model, such as the Ising model, where t = (T, H)) represents the temperature T and external magnetic field H, and x rep- resents a spin configuration on two-dimensional lattice. Alternttivly, generative mapping could be a trained diffu- sion model using stochastic sampler. Here, x generated image and is the corresponding latent noise tensor.
Learning phase transitions in statistical physics Ma- chine learning methods have been widely applied to the study of classical and quantum statistical physics models (Van Nieuwenburg et al., 2017; Carrasquilla & Melko, 2017: Rem al., 2019; Wang et al., 2021; Canabarro et al., 2019). These works primarily focus on determining phase transi- tion boundaries and extracting learned order parameters, which serve to distinguish one phase from another.
2.1. Ising Model
In (Walker et al., 2020), it was observed that the principal components of the mean and standard deviation of latent variables in VAE trained 2D Ising model configurations were highly correlated with known physical quantities, sug- gesting that the VAE implicitly extracts sufficient statistics from the data.
We consider the 2D Ising model (Ising, 1925). A microstate of this model is set of spin variables 8; #1 defined on square lattice of size I. x L. At equilibrium the probability distribution over the space of microstates is
p( Z(H + Eass) 4185 H
(1)
We could unify these two approaches by considering a gen- crative model as a statistical physics system and examine it using information geometry methods.
where H and T are external parameters called magnetic field and temperature. This model is exactly solvable for If O (Onsager, 1944; Kac & Ward, 1952; Baxter & Enting, 1978), i.e. Z(H, T) can be analytically found. The model demonstrates a phase transition at Tor R 2.27 between the high-temperaturre disordered state, where the distribution is concentrated on microstates where spin variables are on average equal zero and the low-temperature ordered state, where the distribution is concentrated microstates with non-zero average spin.
Our contributions This work provides novel method to analyse latent space properties in generative models. It's main contributions are
For two-parametric systems or two-dimensional sec- tions of the latent space, we propose a method to re- construct the Fisher metric. We provide a theoretical proof of the method's convergence. The efficiency of the method was tested on exactly solvable statistical physics models: Ising model and TASEP, demonstrat- ing that the reconstructed free energy coincides with the exact solution.
2.2. Totally Asymmetric Simple Exclusion Process
Totally asymmetric simple exclusion process (TASEP) is simple model of 1-dimensional transport phenomena (Der- rida et al., 1993; Blythe & Evans, 2007; Krapivsky et al., 2010). A microscopic configuration is a set of particles on a Id lattice. Each particle can move to the site to the right of it with probability pdt per time dt provided that it is empty (we put p= without loss of generality). A particular case is open boundary conditions, when particle is added with probability adl per time di to the leftmost site provided that it is empty and removed with probability 3dt per time dt from the rightmost site provided that it is occupied. For this boundary condition the probability distribution known exactly:
Using the proposed approach, we analyze the latent spaces of generative models. We introduce the notion of distinct phases within the diffusion latent space and identify boundaries where the recovered Fisher metric exhibits abrupt changes.
We validate the findings of approximately linear geodesics for interpolation, as discussed in (Shao et al., 2018). However, our work reveals that this result holds only within a single phase. We extend the analysis to phase transitions, showing that the diffusion model
plzla, B) f(x 3)
(2)
Z(a,
2

---

Hessian Geometry of Latent Space in Generative Models
where microstate is concrete sequence of filled and empty cells. Importantly, the function f. which is known exactly does not take the form of the exponential family, (q.( TASEP with free boundaries exhibits a rich phase behavior: For large system sizes three distinct phases the low-density phase, the high-density phase and the maximal current phase are possible depending on the values of a, S, and the asymptotic "free energy" equals:
Bryant-Amari-AAmsttnn theorem (Bryant, 2013; Amari & Armstrong, 2014; Bryant, 2024) states that this is always locally true for 2D analytic manifolds, later extended to smooth cases (Bryant, 2024):
Theorem 2.1. Aenyant-Amar-Arntttoon Any analytic Riemannian metric on 2-dimensional manifold locally admits a Hessian representation.
While Theorems 3.1 and 3.2 presented in the next section apply to arbitrary dimensions of data X' and latent space S, they require a special (exponential) form of the data distribution. Theorem 2.1 allows us to analyze 2D sub- space of latent spaces in GANs, diffusion models, and other non-exponential generative models. Notably, the approach proposed below is theoretically justified for any generative model.
a>>00 2. 3
FrASEP (a, = a(1 a), 8,
(3)
B(1 B), 3 < a, 3 << 5
2.3. Information Geometry
Fisher metric Fisher metric for a distribution p(z](( is defined as
gF(1)= Jx ht00:///u//oooooot/////oooooooo (oop dx (4)
3. Method
log Z(t)
The Fisher metric Riemannian metric. It equips the space S of parameters / with the structure of Riemannian manifold 98.
Da(t.))
V log Z(r)
Hessian metric A Riemannian metric g is called Hessian metric if it can be expressed as the Hessian of a convex potential Φ in local coordinates:
e Dzl1.
O (K(x) Oxt W==
g
da' dx'
(5)
Figure 2: Visualizing Bregman divergence. (Top) A convex function log Z(t) (black curve) and its tangent line (blue) at a point 1'. The Bregman divergence (1 (') (vertical green segment) measures the difference between log (tt) and the linear approximation at L'. (Bottom) The expo- nential distribution e (t(tt") (blue surface) approximates the posterior distribution on parameters p(t|xr,.... .NN, ) where 21...., ZN ~ p(z[('), illustrating Theorem 3.1.
Exponential Family The exponential family consists of distributions of the form
p(x(() () _0 2().
(6)
where the partition function Z(1 is given by
Z(t)= / c(l
The function f(x) called unnormalized density in machine learning and Hamiltonian in statistical physics. is gener- ally unknown, making the direct integration of Eq.7 impos- sible.
This section describes a method for approximating the Fisher metric on the latent space of a generative model, p(z[t), enabling the computation of geodesics for smoother interpolation and the identification of phase transitions.
A key property of exponential families is that their Fisher metric is always Hessian, equaling the Hessian of the log- partition function:
Given generative model p(=](() we proceed in steps (I) approximating the posterior distribution p(t]æ), and (2) estimating the log partition function log ((t) by training a network to simulate p(t[z)) Theorems 3.1 and 3.2 justify this approach for exponential families.
N
y log (t) dt' dt 2 log Z(t)
gF (t) =
(8)
Theorem 3.1. Let X be a space of data samples x E X, and S C RD be a compact domain with the continuous prior distribution p(t) supported on S. Suppose the condi- fional distribution of data samples given parameter t an
Hessianizability A natural question arises: When does a Riemannian metric admit Hessian structure? The
3

---

Hessian Geometry of Latent Space in Generative Models
exponential family
3.1. Approximation of the Posterior
p(xlt) (ol.
(9) The first step is illustrated in Fig. 1. During this step we obtain an approximation of the posterior distribution from set of samples p(t)z.....,,xx) ZN).
where
7(1) / elt.f(e)) dx:
(10)
One could deal with this task by directly training a mapping model, that takes as input T1..... sampled from p(z[(t) and returns a normalized probability distribution on a compact domain S. This ap- proach is most suitable when the samples have stochastic nature and no feature extractor be utilized.
Training a Mapping
converges for all t E S. Let Letz1..... 21...... EN ~ p(zltt). Then, as N oo the posterior distribution satisfies:
Jim No(t)o1....,,N INN N
1/N a.s. (t,)))
(11)
This is the case for Ising and TASEP statistical systems, where a sample z, is a microstate, i.e. it is one of the many indistinguishable realizations of the system with the given parameters t. Importantly, such samples can be pixel-wise uncorrelated. Therefore in this case we use U?eNet (Qin et al., 2020) trained via maximizing likelihood of the true parameters f'. For the training details please refer to the Appendix B.3.
where Dioy zo, t') is the Bregman divergence between ex- ponential family distributions
Ding Zor (n,ttt))
log (()) log Z(1)) e log Z(1)),-11) (12)
For exponential families, Diog (1) (t, E') coincides with the Kullback-Leible divergence DKL (p(z)tt)]pp(ztt) Theo- rem 3.1 thus implies that as more data samples are observed the posterior concentrates on parameters minimizing KL divergence with the true f'. For intuition, see Fig.2.
U--Net
Theorem 3.2. Suppose that the following integral converges to zero
1. Drve ((0 - - D 2000 dt dt' - 0,
Using a Feature Extractor In the image domain, one could use pre-trained feature extractor. Note that from Theorem 3.1, it follows that the posterior distribution behaves as Exp(-NDxx.6
(13)
where Diog Z,(e) (t,t') f') is the Bregman divergence. Then
the Hessian log Z1 (t) converges to the Hessian log Zz(t)
P2)). Having a pre-trained feature extractor &, we can construct an approximation of DxL in terms of a distance between feature vectors:
uniformly in t.
| log (t) 2 log Z2(t)] + O,
(14)
DKL(p(a(ti) |l p(x((2)) ~ (((())),
where || denotes the L norm.
I p(z(t)),
(16)
x2 ~ p(rlt))
In other words, Eq. 13 is MSE loss function for training log Zt).. Strictly speaking, it obtains the partition function only up to an affine transformation log (t) ~ log Z(t) + (c,t) - b. However, our goal is to examine a metric which is a Hessian (t) = V2 log Z(t) and therefore it does not depend of affine term. Unfortunately, training with MSE loss suffers from vanishing gradients during initial optimization stages. Loss selection for the experiment is covered in Section 3.2.
Indeed, if the feature extractor & acts approximate sufficient statistic, then
DRi (p(z(tt) I| p(xlt2)) (ra(((E(z))t)) p| ((E()))t2))
(17)
If the distribution of features (xE()))) under tę is approxi- mately normal N(4, 7), then the KL divergence simplifies (up to an additive constant) to the squared difference be- tween the feature means:
For simplicity, we assume uniform parameter distribution p(t) over S:
(xa((((()))) p(E(z))tz) 1 711m p2l22 (18)
t S,
p(t)= = Vol 0.
(15)
Then the posterior could be approximated as p(t| ................................ IN) R 4 ((0()))
otherwise.
This choice avoids bias toward specific regions of S, ensur- ing equitable exploration of the latent space.
(19)
x x~ p(xtt, o' ~ p(x)t').

---

Hessian Geometry of Latent Space in Generative Models
Below in the Experiments section we take CLIP as fea- ture extractor & and produce the approximation of posterior distribution based on distances between CLIP images em- beddings.
3.3. Geodesic Approximation
After obtaining the Fisher metric, gF. we unlock the ability to explore the intrinsic geometry of statistical model space. Geodesies, in this context, represent the shortest paths between two probability distributions within this space. They are analogous to straight lines on a flat surface, but in a potentially curved space dictated by the Fisher metric. To find these geodesics, we aim to minimize the curve length, L (t),, for smooth curve (() parameterized from t = to I and lying within statistical model space. This curve length is calculated using the Fisher metric as:
3.2. Approximation of the Fisher Metric
The loss function in Equation 13, while theoretically ensur- ing Hessian convergence as stated in Theorem 3.2, suffers from vanishing gradients during the initial optimization stages (see lemma A.6 from the Appendix section). To ad- dress this problem, we normalize e e Ding =ce;(())) to obtain a probability distribution. Then we compare with the pos- terior distribution p(t: .1.-.... .N) using -haann-Shannon divergence (JSD), which is proper metric between two probability distributions.
[[x(0]= =1 VA0F
(25)
Lemma 3.3. Let Z : - (0, oo) with S R" compact domain and log Z convex. Then for any fixed t S:
Here, (() represents a path in the parameter space, and (tt) = dr(t) is its tangent vector.
dt
exp( Dior z (t, ())
We split (t) into discrete points (70: 71..... :.. where To and YN are the starting and ending points of our desired geodesic. The continuous integral is then represented by discrete sum of distances between consecutive points, using the Fisher metric to measure these distances, optimizing intermediate points via Adam to minimize L[>]] (see (Shao et al., 2018) for details).
Is exp ( -Dio z (8, (²)) ds
(20)
exp (- (t, V. log Z(t') + log (t))
exepp( (-(8, V log ((')) + log (3)) ds'
where Diog ztt, f') is the Bregman divergence.
Define normalized distribution which depends only on the log-partition function following Lemma 3.3
4. Experiments and Results
To support theoretical reasoning, in this section vali- date our method on the exactly solvable statistical models, namely Ising and TASEP, and compare our estimation of log (t) with a ground truth. Then, we evaluate our method on two-dimensional slices of diffusion models, comparing the path length and curvature of the learned trajectories with those produced by other methods.
exp (- (1, v log Z(t')) + log (t))
Plog (tttt)) IS exp (- (s, r log (t")) + log (s)) ds (21)
Now given the posterior p(t|#),.... IN), which approxi- mation was discussed in the previous section, we could train the log-lartinn partition function log Zt) by minimizing the loss
(0) = / Dis EN)) Plog (41()) di', (22)
dF/dT
dF/cH
Energy
Magnette Rild (H1
Magneskc Rold
where F1,--., EN p(xltt). The Jensen-Shannn diver- gence for distributions P and Q is defined as
P + 2) Dis(P.Q) = (PI P+Q, Q (Q1
1
P+
Temper alure (T)
Terpet (T)
2
Figure 3: 2D Ising model. Partial derivative of the recon- structed free energy with respect to temperature (T.H. Partial derivative of the reconstructed free energy with re- spect to magnetic field @Fme(T,H) and reconstructed free energy. See Appendix B for dataset and training details.
(23)
87
The resulting approximation of the Fisher metric is
(r(() 2 log Zo- (t), 0* argmin L())
(24)
äH
We model log (t) by MLP with 5 hidden layers, hidden size of 512 with ReLU activation. We do not require the MLP to be convex as converges to the convex function during the training.
4.1. Exactly Solvable Statistical Models
We compare against two baselines: (1) posterior--maan-ss statistics (Mean-as- as-aat))) approach (Fearnhead & Prangle,
5

---

Hessian Geometry of Latent Space in Generative Models
Ours
Ground truth
Table I: Performance results (RMSE) for ISING and TASEP models. F stands for Free Energy.
0.3
0.3
ISING
F RRSS
dF
RMSE
dF
RMSE
0.2
JF
JH
Convex (Ours)
0.0883
+
0.1106
0.1237
0.1
0.0006
0.0002
0.0016
0.0
0.4
011
Mean-as-Star
0.0981
+
0.4766
1.0936
1.0
1.0
0.8
0.8
0.0010
0.0023
0.0033
0.6
0.6
404
0.2
0.000
0.2
0.4
a
0.6
0.8
00.4
0.2
0.
0.2
0.4
0.6
0.8
PCA-VAL
0.0018
0.1669
+
0.0025
0.7428
+
0.7988
0.0022
+
TASEP
F RMSE
dF RMSE
dF RMSE
Figure 4: TASEP. Left: reconstructed free energy (red) com- pared to the exact solution (blue). Our method achieves near-exact agreement except near domain boundaries (Ap- pendix B.
Convex (Ours)
0.0112
+
0.1165
+
0.1135
0.00008
0.0025
0.0017
Mcan-as-Stat
0.0529
+
0.3832
+
0.3833
+
0.0005
0.0038
0.0031
ICA-VAE
0.0524
0.3837
+
0.3872
0.0006
0.0038
0.0022
(Song al., 2020). For our generation we use 50 inference steps, classifier free guidance scale set to 5. Prompt is chosen as "High quality picture, 4k, detailed" and negative prompt "blurry, ugly, stock photo".
Sog 391 i8'
To build 2 dimensional latent space section of the diffusion model we generate random initial latents 20, 21, 22. We use interpolation between latent representations:
0.8
1.0
z = 20 + a(21 20) + B(22 - 20),
3.2
0.8
0.0
where and 3 are uniformly sampled from U|0, 1j. In this setup we take three different triplets of initial latents and generate 60000 images for each with a fixed prompt and initial latent uniformly sampled from the interpolation grid. To prevent our interpolation fall off the Gaussian hyper- sphere in all our experiments employed normalization of latent vector z.
Figure 5: Free energy of diffusion model reconstructed with CLIP distance.
2012: Jiang et al., 2017), and (2) PCA-VAE, a dimensionality reduction approach following (Walker al., 2020) (see Appendix B.1 for details of baselines evaluation).
At first we limit ourselves to the case of deterministic gener- ation process, setting DDIM parameter = 0.
Table compares the performance of our method ("Con- vex") against two baselines on 2D Ising and TASEP models. We report the root mean squared eиror (RMSE) for the re- constructed free energy (F) and its partial derivatives with dF respect the model parameters dF dF for Ising; des de ( dir
Using a CLIP Feature Extractor For the given dataset we then compute CLIP distances approximate prior distribu- tion discussed in Sec. 3.1. Following our method we train model to reconstruct log ((t) log Z(a, 33) presented in Fig 5. The retrieved log Ztt) is not smooth and has abrupt changes in derivatives shown on Fig.7(C),, reflecting phase transitions in image space. To evaluate our emergent metric, we analyze geodesic paths. Our results demonstrate that geodesics are approximately linear within a single phase but exhibit nonlinear deviations phase boundaries, Fig. 7(B).
for TASEP). Accurate reconstruction of partial derivatives is critical for identifying phase transitions, where these deriva- tives diverge. Our method achieves lower RMSE across all metrics in reconstructing thermodynamic quantities.
3ig.3 and Fig.4 demonstrate the results of free energy log (t) reconstruction for both systems.
The Fisher metric enables the construction of smoother inter- polation paths (geodesics) compared to linear interpolation, Fig. 7(A)
4.2. Two-dimensional Slices of a Diffusion Model
The experiments with diffusion are based on StableDiffusion 1.5 (Dreamshaper8) (Lykon, 2023) with DDIM scheduler
We then investigate images near the phases boundaries. Un- like classical statistical physies models with continuous
6

---

Hessian Geometry of Latent Space in Generative Models
Figure 6: The fractal structure of phase boundary in the interpolation landscape of diffusion model. The last plot represent the parameter variations 10-5 between neighhoring images. The small changes in latent space cause switching between mountain and lion.
phase boundaries, the latent diffusion model exhibits phase diagrams with fractal boundaries, illustrated by Fig.6.
terior distribution exactly recovers the target one resulting in smooth log (
Crucially, we observe that near these boundaries, samples from neighboring phases blend together, leading to a phe- nomenon where phases permeate one another. Repeated magnification of the boundary shown in Fig.6 reveals self- similar patterns across scales, starting from variation scale 10-5 until reaching float16 accuracy of 10-8 In other words, near the phase boundaries the diffusion output is highly sensitive to small changes in latent space.
Following this result, we evaluate our method on generation with DDIM `j 0.1. We observe that adding the noise simply results in blurring the posterior distribution predic- tion. In the case of U?-Net it doesn't 't change the learned geometry, following the argument above. However, in the case of CLIP this results in smoothed free energy landscape and thus doesn't i show sharp boundaries as in noiseless case.
Baselines and Metries To evaluate our method, we com- pare it against the approaches of Wang al. (2021) and Shao et al. (2018), which represent related work in deterministic generative models. Below, we clarify the key differences and present quantitative results from these comparisons.
This behaviour is commonly characterized by the Lipschitz constant. The work by Yang et al. (Yang al., 2023) dis- cusses the Lipshitz constant for diffusion models with spect to time variable. However, to our best knowledge, the observation on divergence of Lipshitz constant with respect to latent space is new.
Shao et al. (2018) study VAES and compute geodesic curves based on the pullback metric induced by the Euclidean met- in pixel space. Wang et al. (2021) consider GAN models and define metric on the latent space via the pullback of the l distance in the feature space of VGG-19 (LPIPS distance). In both cases, the generative models are deterministic: each latent Z produces only single image X.
The Fisher metric no longer analytic or smooth there, meaning the AaarittAmar--mssstoee theorem does not apply, and the metric cannot be expressed the Hessian of convex function.
Using Unet Mapping We investigate our method on diffusion model with 2/e-Net and observe a distinct out- come. Unet doesn't take into account semantic information, solely relying on pixel space representation. Therefore is able to predict the exact parameters (a, image gener- ation. From this perspective the generation is completely deterministic and every image is distinguishable. Thus, pos-
The key distinction of our work lies in its consideration of a broader class models, specifically, models with stochastic generation, where single latent Z corresponds to distri- bution in the image space (X Diffusion models with stochastic sampling (and all statistical physics models) can- not be addressed within the formalism of Shao et al. (2018)

---

Hessian Geometry of Latent Space in Generative Models
Geodesic
A
Linear
sudden
appearance of cat
Geodesic
linear
alogZ aix
alogZ aB
Endpoints
B
ablogZ da?
aologZ aB²
α
α
Inside a phase the geodesic approximately straight
First and second order derivatives log Z indicate phase boundaries
Figure 7: (A) Geodesic (ours) and linear interpolation between images. Note that our interpolation variant is more perceptually smooth. (B) Illustration of 60 000 generated by StableDiffusion images with geodesic plots (C) First and second derivatives of the log-partition function of diffusion model. Note, that second derivatives are diagonal components of the Fisher metric.
or Wang et al. (2021).
analysis reveals that trajectories constructed using the Wang metric show significantly higher curvature and frequent turn- ing compared to method. We attribute this behavior to the finite differences, which introduce high-frequency noise in the metric components. In the case of diffusion, the Jaco- bian is hard to obtain via backpropagation, as suggested in Shao et al. (2018), due to high computational cost. During our evaluation we observed that the phase boundaries re- mained stable across all tested feature extractors. Therefore, conclude that the algorithm consistently captures the same phase structure.
We compare our algorithm with these baselines the deter- ministic sampling regime. We obtain the pullback metric from the Euclidean metric in the space of CLIP embeddings. The Jacobian is estimated via finite differences. We use three evaluation scores: CLIP, pixel, and Perceptual Path Length (PPL) (Karras et al., 2019), which measure the aver- age path length in CLIP embedding space. pixel space, and the feature space of VGG-19, respectively. These scores computed as the cumulative & distance between con- secutive feature vectors (or images for pixel space), then averaged across multiple trajectories (Table 2).
U Underlying mechanism for phase fransition We believe the observed phase transition behavior is connected to the geometry of the image space. This idea aligns with findings from the Brown et al. (2022), which shows that natural images lie on a union of disjoint low-dimensional manifolds with varying dimensions.
We observe that our interpolation is on par with baselines in case deterministic sampling. We additionally compute the curvature of each trajectory as the mean angular change per unit length between consecutive path segments. Our
8

---

Hessian Geometry of Latent Space in Generative Models
Table 2: Comparison of trajectory lengths and curvature in deterministic sampling
sampling regime.
Metric
Geodesic (Ours)
Linear
Geodesic (Wang/Shao)
CLIIP Length
72.3 + 4.00
73.6 3.54
73.6 + 4.37
Pixel Length
2.77 x 10 4:2.88 10*
2.76 x 10 +.2.7 10*
2.74 x 10 3.53 x 10*
Perceptual Path Length
3.12 +: 0.16
3.17 + 0.23
3.19 + 0.21
Mean Curvature
0.367 + 0.691
0.00 + 0.00
1.33 + 0.53
In diffusion models, generation begins from a high- dimensional Gaussian latent distribution. The reverse ODE process maps this distribution onto disjoint, lower- dimensional manifolds corresponding to distinct image modes. Such a transformation-ffrom
latent
data
unimodal la- tent space to multimodal data space with disjoint sup- marts-may result in a diverging Lyapunov exponent or, equivalently, a diverging Lipschitz constant in the genera- tive mapping, indicating phase transitions.
time steps
This phenomenon can be illustrated by the following propo- sition, which simulates a lower-dimensional data manifold with disjoint supports.
Figure 8: Illustration of Proposition 4.1
Proposition 4.1. Suppose that the (larget) data distribution a bimodal mixture of two Gaussians, each with variance 2²:
L N -1,0²) + 2 Nxxx 1,,²).
operate in more general settings than MCMC-based tech- niques like importance sampling.
-
(26)
The latent (source) distribution is the standard normal Na 00 1). Consider the variance-preserving SDE
theorem allows us to go beyond exponential family distributions and apply our method to study woo-dimensional sections of latent space of arbitrary generative models.
ax. - 3x dt + _x,=-]ax.dt
dWW
(27)
We extend notion of phase transitions known in statistical physics, where small changes in parameters lead to signifi- cant shifts in system output, to describe abrupt changes in generative models. While recent studies (Sclocchi et al., 2025; Biroli et al., 2024) have explored phase transitions in diffusion models with respect to time-step, our work ad- dresses a distinct problem. We treat the latent space as a parameter space and generated images microstates. Our reconstruction of the Fisher metric for two-dimensional section of diffusion model's latent space indicates the pres- ence of distinct phases. We observe that within a phase, geodesic interpolations between images are approximately linear, consistent with the work Shao et al. (Shao et al., 2018). However, our results reveal that this assumption breaks down phase boundaries, where the geometry be- comes highly nonlinear. Specifically, we observe that diffu- sion models exhibit fractal boundaries between phases. We argue that near these boundaries diffusion has divergent Lip- schitz constant, complements results of Yang et al. (Yang et al., 2023) showing that diffusion has divergent Lipschitz constant in time variable.
Then the Lyapunov exponent of the corresponding reverse- time ODE has the following form:
8 ( 1
2
(28)
2
g0
and it diverges to infinity as O. In this case, the point 0 can be interpreted as a phase transition boundary.
5. Discussion
In this work we develop a method to reconstruct the Fisher metric for any exponential family distribution. This typ- ical for statistical physics systems. Our method based on reconstructing the nog-partition function from posterior distributions the parameter space. We provide theoretical grounding and validate it through experiments by compar- ing our reconstruction with exact solutions, outperforming mean-as cann-as--ufffiie aas-sufficient-staaee statisstiss and PCA-VAL baselines. Im- portantly, we do not assume prior knowledge of the unnor- malized density or Hamiltonian, enabling our method to

---

Hessian Geometry of Latent Space in Generative Models
Impact Statement
Derrida, B., Evans, M. R., Hakim, V., and Pasquier, V. Exact solution of a 1d asymmetric exclusion model using matrix formulation. Journal of Physics A: Mathematical and General, 26(7):1493, 1993.
Our work contributes into the fundamental understanding of generative model latent spaces, offering mathematically grounded tools for detecting phase transitions and construct- ing smoother geodesic interpolations. While generative models in general raise concerns about misuse, our theo- retical focus does not directly enable such risks. Nonethe- less, we acknowledge that better exploration of latent space could, in principle, facilitate adversarial prompt crafting or model probing. We expect our contributions to support better understanding of generative models and stimulate cross-disciplinary dialogue between machine learning and information geometry communities.
Fearnhead, P. and Prangle, D. Constructing summary statistics for approximate bayesian computation: semi- automatic approximate bayesian computation. Journal of the Royal Statistical Society Series B: Statistical Method- ology, 74(3):419 474, 2012.
Guo, J., Xu, X., Pu, Y., Ni, 7.,, Wang, C., Vasu, M., Song, S., Huang, G., and Shi, H. Smooth diffusion: Crafting smooth latent spaces in diffusion models. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 7548-7555 2024.
References
Amari, S.-i. and Armstrong, J. Curvature of hessian man- ifolds. Differential Geometry and its Applications, 33: 1-12, 2014.
Ising, E.
Beitrag theorie des ferromagnetismus. Zeitschrift für Physik, 31(1):253-258, 1925.
Jiang, B., Wu, T.-y., Zheng, C., and Wong, W. H. Learning summary statistic for approximate bayesian computation via deep neural network. Statistica Sinica, pp. 1595-1618, 2017.
Arvanitidis, G., Hansen, L. K., and Hauberg. S. Latent space oddity: on the curvature of deep generative models. arXiv preprint 1rXiv:1710.11333
2017.
Baxter, R. J. and Enting, I. G. 399th solution of the ising model. Journal of Physics A: Mathematical and General, 11(12):2463, 1978.
Kac, M. and Ward, J. C. A combinatorial solution of the two-dimensional ising model. Physical Review, 88(6): 1332, 1952.
Biroli, G., Bonnaire, T., De Bortoli, V., and Mézard, M. Dynamical regimes of diffusion models. Nature Commu- nications, 15(1):9957, 2024.
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Pro- ceedings of the IEEE/VV conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Blythe, R. A. and Evans, M. R. Nonequilibrium steady states of matrix-produce form: solver's guide. Journal of Physics A: Mathematical and General, 40(46):R333, 2007.
Krapivsky, P. L., Redner, S., and Ben-Naim, E. A kinetic view of statistical physics. Cambridge University Press, 2010.
Brown, B. C., Caterini, A. L., Ross, B. L., Cresswell, C., and Loaiza-Ganem, G. Verifying the union of manifolds hypothesis for image data. arXiv preprint arXiv:2207.02888 2022.
Li, H., Shen, C., Torr, P., Tresp, V., and Gu, Self- discovering interpretable diffusion latent directions for responsible text-to-image generation. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12006- 2024.
Bryant, R. Answer to mathoverflow question "when riemannian manifold is of hessian type". http:
Liu, Y., Sangineto, E., Chen, Y., Bao, L., Zhang, H., Sebe, N., Lepri, B., Wang, W., and De Nadai, M. Smoothing the disentangled latent style space for unsupervised image- to-image translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10785-10794, 2021.
/ 1/mathoverff ow. net httt//uustioons/1111 122308// 2013. Accessed: Jan 20 2025.
Bryant, R. L. Hessianizability of surface metrics. arXiv preprint arXiv:2405.06999 2024.
Canabarro, A., Fanchini, F. F., Malvezzi, A. L., Pereira, R., and Chaves, R. Unveiling phase transitions with machine learning. Physical Review B, 100(4):045129.0 2019.
Lykon. Dreamshaper-8. htttps://huginneee http://gggggggg.ccocco co/ Lykon aerton/drramshappe---r 2023.
Onsager, L. Crystal statistics. i. a two-dimensional model with an oeder-disorder transition. Physical Review, 65 (3-4):117, 1944.
Carrasquilla, J. and Melko, R. G. Machine learning phases of matter. Nature Physics, 13(5):431 -434, 2017.
10

---

Hessian Geometry of Latent Space in Generative Models
Park, Y.-H., Kwon, M., Choi, J., Jo, J., and Uh, Y. Un- derstanding the latent space of diffusion models through the lens of riemannian geometry. Advances in Neural Information Processing Systems, 2004129-244422 2023.
Qin, X., Zhang, Z., Huang, C., Dehghan, M., Zaiane, O. R., and Jagersand, M. U2-net: Going deeper with nested structure for salient object detection. Pattern Recognition, 106:107404, 2020.
Rem, B. S., Käming, N., Tarnowski, M., Asteria, L., Fläschner, N., Becker, C., Sengstock, K., and Weiten- berg, C. Identifying quantum phase transitions using artificial neural networks on experimental data. Nature Physics, 15(9)917--220, 2019.
Sclocchi, A., Favero, A., and Wyart, M. A phase transition in diffusion models reveals the hierarchical nature of data. Proceedings of the National Academy of Sciences, 122 (1):e2408799121, 2025.
Shao, H., Kumar, A., and Thomas Fletcher, The rieman- nian geometry of deep generative models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, 315-323, 2018.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Tosi, A., Hauberg, S., Vellido, A., and Lawrence, D. Metrics for probabilistic geometries. arXiv preprint 1rXiv:1411.744 2014.
Van Nieuwenburg, E. P., Liu, Y.-H., and Huber, S. D. Learn- ing phase transitions by confusion. Nature Physics, (5):435 439, 2017.
Walker, N., Tam, K.-M., and Jarrell, M. Deep learning the 2-dimensional ising model to extract the crossover region with a variational autoencoder. Scientific reports, 10(1):1-12, 2020.
Wang, J., Zhang, W., Hua, T., and Wei, T.C.. Unsuper- vised learning of topological phase transitions using the delinski-harabas index. Physical Review Research, 3(1): 013074, 2021.
Yang, Z., Feng, R., Zhang, H., Shen, Y., Zhu, K., Huang, L.. Zhang, Y., Liu, Y., Zhao, D., Zhou, J., et al. Lips- chitz singularities diffusion models. In The Twelfth International Conference on Learning Representations, 2023.

---

Hessian Geometry of Latent Space in Generative Models
A. Proof of theoretical results
Theorem A.1. Let X a space of data samples X, and S I be compact domain with the continuous prior distribution p(t) supported S. Suppose the conditional distribution of data samples given parameter is an exponential family
p(alt) = cit- -1og
loR Z(1)
(29)
where
2(1) = /xx ell,((e))
(30)
forverges all E S. Let The p(a)tt) Then, as N Do the posterior distribution satisfies:
lim
e IN)) as -Dagzaoltl))) = -llaaltta)""lll(())))
I/
(31)
N
where Ding 2o, (') is the Bregman divergence between exponential family distributions
Dioe (t, t') = log (t) - log ((t)) (V, log (tt),,
(32)
Proof. We begin by applying Bayes' theorem derive the posterior distribution. Then we use properties of exponential families and the law of large numbers to obtain the desired limit.
By Bayes' theorem:
pl EN (t)
p( IN) Js p(x1 EN s pps)ds
(33)
Since the data samples Tı,... EN are i..dd given i, we have:
p(r xNl II p(a,tt) = II ((r)(()))) log 2(1)
(34)
(N Z(t)
Substituting into the posterior, we find:
eltit Em ((x,)) log 2(1)+ + log p(t)
p(t| IN) I/N
(35)
(Is M Mes EN S(xs)) log 2(z)+ log log ds)
1/N
Now, consider the limit:
lim N-00
N (a.))
(36)
By Kolmogorov's strong law of large numbers, this converges almost surely to the expectation:
/ f(z)p())t)dz = V log Z(t'),
Σ f(x))
as.
(37)
N
where the last equality holds because p(zlt) is exponential family distribution.
In other words:
p ( (WEE: Ω: lim N
1 N
Σ ((x(w) v. log Z(t)) /
1.
(38)
By the continuous mapping theorem: if X1, Xo,... is sequence of random variables and h. - a continuous function, then:
X as X
h(Xx) as, 1(X).
(39)
12

---

Hessian Geometry of Latent Space in Generative Models
Let (8,
Since the function vp is continuous in s, and we have shown that 1/4 Σ ((14) -n
Ve log ((t') N oo, the continuous mapping theorem implies:
+ Ei (oo))) 2()+ + logp(t) _8, et. V log -log Z(t)
(40)
where p(() is a continuous function on compact domain, and therefore bounded, which implies & log - 0 as N + 00.
Now, let's consider the term in the denominator of the posterior:
(s e 2im ((x)) log 2(0)+ + log popp(e)],b
1/N
(41)
We can rewrite the integral part using Lemma A.2. This lemma applies because, according to the strong law of large numbers, the average N Ei= (1;) converges almost surely to finite value. To utilize the lemma, we define the function:
N
N
Σ (()(w))) Ms N.w) = (8, N
1
- log (()) = v/ 8, Σ ((x(ww)
(42)
i=1
We will ignore the term n log p(s) as it converges to 0 as N goes to infinity and wont affect the value of the integral. Then by Lemma A.2, we can state that the following limit holds almost surely:
1
log / (w) ds as. N
1 N
lim
max 18ss, Ve log Z(t)).
(43)
SES
Using this result, we have
ellit 2ith ((x.)) log 2(1)+ + p(1)
28, el. log Z((')) -)og))))-o--- (())
IN)
(44)
(Js elant Mi f(x,)) lon (())+ & (on I/N
emax.es V...V log Z(e)))
Now, observe that we have:
max (s, v log Z(t') = max Kis, V log ((t)) - log Z(s)]
(45)
S
sES
Since the function is concave, we can state that the maximum achieved when s = f,, which allows us obtain
max (ss, V log Z(t')) (t', V log (tt)) - log (t')
(46)
SES
Substituting back into the limit, we arrive at the expression for the Bregman divergence:
lim INN a.s. elt.Ve log z(t)) -oo Z(t) N
(47)
elt'.V log z(1)) Z(r')
To complete the proof, we need show that the KI, divergence DxL (p(=(t)))l((((())) in the theorem statement equivalent the Bregman divergence (t,t, t') in our result. We compute the KL divergence as follows:
f plxlt") log p(xltt) p(x(())
dx
plal())lopp<(11)) log p(x(()dxx
/ (()))),ff(*) log Z(t') - ((t. f(x)) log Z(t))
/ (f. - logddr--og22 Z(t') / p(zlt') (t. f(z)) + log Z(t)
= -(t', Ve log Z(t') log Ztt) - (t, Vr log Z(t)) ) + log Z(t) =(t'- Ve log Z(t') + log Zt) log (tt)
DD((tt')
13

---

Hessian Geometry of Latent Space in Generative Models
where use the property that the expectation of f(x) under the distribution p(a:(() is equal to the gradient of the log-partition function Px=p(=le)) [f(=) V log ((t))
Thus, we have shown that the limit of the posterior is related to the Bregman divergence of t' and t, and is consistent with the theorem statement, which concludes the proof.
Lemma A.2. Let X; : 0 - IR" be measurable functions, & E S, w E St N E N, and assume E|X,] Ma < oo (first moment exists). Define:
1 N Z
o(s,N,w) = (
(48)
where (xx, is a continuous function in both variables and uniformly continuous in x for all s. Then:
1
def
lim No00 N
1
log
(1 (eNole E (.,xx)d
ds)ds
= max
pz) M. .ES
(49)
Proof of Lemma. Let XN = tV Ei Xi(w). By Kolmogorov's strong law of large numbers, we have:
xx(w) - pz: E(WEE:: lim Xx(w) mob) = 1. N
(50)
Since (ss, is continuous in æ, by the continuous mapping theorem:
lim Xx(w) = Pe N
lim v'ss, (x()) - ((..
(51)
No000
Thus, for w e a::
lim V(s, (N(W)) = (ss,He)
(52)
N
In other words, for any >> 0 and fixed E S. there exists No(e, 8) such that for all N >> No(e, 8):
W(s, (N(ww) (s, <e
(53)
and for all á s E S. and e 0, there exists No(w, s, e) such that for all N No(w,
lbbs,, (w)()) (s.HHo)) <e.
(54)
Define:
My max (iss, (w)).
(55)
sES
Since 1b(8, .r) is continuous in and S a compact domain, Mv(w) < cc. Moreover, since Xv(w) 28 Ma and both a and max are continuous functions:
(w) max (bss,, (N #ES
as max ((s = M. aES
(56)
Now consider:
[ Nole do S / N Nmaxes ((),8x(w)) do N maxis (w
(57)
Taking the logarithm and dividing by N:
N log (oMM( s MM(()++ + N log Vol(S).
1
(58)
14

---

Hessian Geometry of Latent Space in Generative Models
Finally, taking the limit as N oo:
1 N log e NN S lim (Mx(w)+ + N000
log vol(s)) My(w) us. M.
lim N-000 N
(59)
Thus, the result holds almost surely:
lim - N log Ne(oXx(eeds ds SM- = max (s,
(60)
sES
Now, almost everywhere, have I S M. Define:
K, = {s
M\ < </22).
(61)
Then:
/ Nolls.N(W)ds ds
(NulssN(WW)dds V..
(62)
K.
Now, consider:
We, [w((w) - MI = -b/s, Xx(w)) - 1b(8 + (bs M2) - M
(63)
$ (vss, XN(w) - (a) + (1(5,, - M
(64)
First Term: By the strong law of large numbers and uniform continuity of th, for any 0, there exists No such that for all N No and K,:
Mz)) e/2.
(65)
This follows because N (w) -4 Mas and (s. is continuous in Hence, for sufficiently large N, vhs (x(w))
converges uniformly to (8 (a)) for E Ke-
Second Term: By the definition of Kg, for all s E Kg:
(b), MI e/2.
(66)
Combining these results, for all > E Kr, and N > No:
(w)) - M <c
(67)
Thus, for all > 0, s. E Ke, and almost all w E Ω:
(x())( > M - c.
(68)
Therefore:
f NVleXxwWDdss ds /
= -
(69)
Taking the logarithm and dividing by N:
1
log / e (N((sxx(=))ds
1
log (N(MM-
(70)
N
N
K.
= = - c
log Vol(K.). N
(71)
Taking the limit as N 00:
1 log
e No(s:& ds M
lim
(72)
N N
K.
15

---

Hessian Geometry of Latent Space in Generative Models
Combining this with the upper bound S M, have:
M M
(73)
Since this holds for all e > 0, it follows that:
I= M.
(74)
Lemma A.3. We will show that the following function is bounded
((s,w, (s, (w()))) log (s),
(75)
where E N, sE S 1R
Proof. First, we use the
inequality
(s,, Xx(W))) || | [
(76)
Furthermore, since the Le norm is a continuous function, and there almost sure convergence XN Mx. then
( -3, ||
(77)
Thus, there exists No such that for all N >> No.
(1))11))122 || <E = [Xx(w))]l2 < Melle +
(78)
Since log 7 is continuous the compact S, there exists Cz such that log 7(8) < Cz for all S.
On the compact set S, there exists C, such that ||s||2 < C, for all E S. Therefore, for all s E S, there exists No such that for all N > No.
Ns, < C= = |l C. + Cz te
(79)
Lemma A.4. We will show that the following function is uniformly continuous ((s,w, N) (s, (N(ww)) log (s)),
(80)
where E SN
Proof. Let 1i and 22 be vectors in IR..
(1) x2))
[s, 21) log Z(s) ((s, 22) log (s))
((, (s, x2))
[s, x2)
|s|| [m J22 cos(
where the angle between s and Ø1 Ø2. Since compact, there exists M such that |s| < M for all s E S. Thus,
(1) (s. 22)) S M [x] 22
Given 0, we choose 8 = e/M. If [r x2l < 8, then
(0( - (8)) <
Therefore, (88, 2) is uniformly continuous in x.
16

---

Hessian Geometry of Latent Space in Generative Models
Theorem A.S. Suppose that the following integral converges zero
10l// Diux (1(1) - e Dix 22(1)11)) dl di' - 0,
(81)
where
p(t') ((t) (Veiptt)), is the Bregman divergence. Then there log (1) converges to log Z2(() uniformly in
| log (t) V log (t) - +0
(82)
where ||-|| denotes the 1,2 norm.
Proof. Since the integrand non-negative and its integral goes to zero, the integrand must converge to zero almost
everywhere. As the functions involved are continuous, we have:
Disg zn (t) a) = e Dro (t)))
(83)
for t, E O. Taking the logarithm of both sides, we get:
- (t,, -Doo (t',tt
(84)
which implies
Diog (.(0) (1,,)) Drog 22(1) (1,
(85)
Substituting the definition of the Bregman divergence, we have:
log Z1((') - log (1) - (V log Zi()), -
= llog Zz((') - log Zz( - (V log Zz(t), t).
Rearranging terms gives:
(log (t') log Zz))") (log (t) log Zz())) (V log (1( V log Z2(t),t' =
(86)
Let (tt) log (t) log Zz( Then the equation becomes:
(('') A() (VA(I), - 0.
(87)
This implies that the first-order Taylor expansion of A (t') around t is exact, which holds if and only if A(t) is affine
function:
A (t) = (c,t) + b,
(88)
for some constant vector e and scalar b. It means that two oog-parttiinn function are related via an affine transformation
log (t) log Ze (c, +
(89)
Therefore the L norm equals to zero
V log (t) - log Zz(t))l
(90)
Lemma A.6 (Vanishing Gradient in Bregman MSE Loss). Consider the loss functional L
2
Js e- Js (1)22)))((t (g(
(1,1))
dt dt'. Assume there exists region C S S where either 2 C uniformly for some C' 1. Then the gradient of L with respect to
Ding 7, (t, 2 C Diog, (t,
parameters governing log Z1(t) (t) satisfies:
|V < -C
(91)
where > O is a constant which depends on the measure of and Lipschitz constant of log Zt(t) which we assume to be bounded. Thus, gradients vanish exponentially as increases.
17

---

Hessian Geometry of Latent Space in Generative Models
Proof. The gradient of the loss functional L with respect to log (1) can be formally written (using functional derivatives) as proportional to:
8 V x 8 log (1)
E Diog (o) - #2cn)(tt) dt dt'.
(92)
Expanding the square and taking the derivative with respect to log Zi (1), we focus on the terms involving Diog Zi (1) (1, The gradient is then proportional to:
VL α / (e-Dun - - eiux -e-Duzzaco((te)) ô log (t)
5
e Diue (1.0) dt dt'.
(93)
The functional derivative of e Ding (t)((t))t) with respect to log (t) involves the derivative of the Bregman divergence Ding 2i (t) (f, t') with respect to log Z1 (t). Let us denote (t, t') = Ding (t))(t, and (t = Diog Za(t) t', Then the gradient can be expressed as:
VL x JS] -e (-
(94)
where VD1 (t, t') represents the gradient of Diog (t, t') with respect to parameters governing log Z1 (t). We consider the magnitude of the integrand within the region R C S x S. In region R. either Diog (t) t') 2 C or Diog (t, 2 C.
Case 1: Diog z, (t, 2 C. In this case, e Dive ((02) Sec The magnitude of the integrand is bounded by:
Ding (t,))) e- Ding ((os))))()))) e- Ding (col) (-VD,,
F Dro 2ioeit + -D-ux escoo(s)))) (e- le Dive (100)) //
- $ e +e Pior )(oezgoo(hi))
(t,
< (1 + e
(200 (p (t,
Assuming that V/ (f, l'Ill is bounded by some constant M' due to finite Lipschitz constant of log Zi (t) and since e Dioe (t,)) < 1 (for nen-negative Bregman divergences), the integrand magnitude is bounded by 2M AM
Case 2: Diog (1, (') 2 C. In this case, e DioR (1.1) S6- ---.Thee The magnitude of the integrand is bounded by:
Diux 2((0)(18)) - e --- ((ox2) E Drox -Piox (-
e Diog 2100011) + le- -Dog (x11(11)))))) le PI02 (1
e DIeE (1rlle)) + to-0)= e- - ((),,,)))
e 20 +e - -G-Dunzzco(ttt) |VD)((t
S (1 + CM = 2
assuming Diog 2i (1) (t, 2 0 and Drng (t) 1.
In both cases, within the region R, the integrand's magnitude is bounded by 2.Me Let m(R) be the measure of region R. Then the norm of the gradient can be bounded by integrating over R:
|V & / 2M' - dt di' = 2 = Ke-C
(95)
where K- (.1'm(1) is a constant dependent on the measure of R and the bound M' which is related to the Lipschitz properties of log Zı (t). This shows that the gradient vanishes exponentially as C' increases.
Lemma A.7. Let Z S (0, with 5 TR. compact domain and log 7 convex. For any fixed & S:
exp ( (- Drog x (1' 1))
exp (-
log Z()) + log Z(("))
(96)
cxp (- (- -Doo (8, t)) ds Js exp (- exp
V log Z(t) + log 2()) ds
18

---

Hessian Geometry of Latent Space in Generative Models
Proof. Expand the Bergman divergence Diog z 2 ()) log ((() log (()) (V log Z(1), l' l). The numerator be rewritten
exp( Hep(-Diog z(t', t)) = exp (- (log ((t) log (t) - (V log 2 - t))) -t))
eep ( - log ((() + log (1) + (V log 2(1),1) - (V log 2()),))
exp(log Z(1) exp(- (V log ((1), exp(-log ((1)) + (V ((1)))))
(97)
The denominator, integrating over 8 E S, becomes:
/ exp(-Dior 2 (s. 1)) ds
exp (- (log (8)) log (t) - (V log Z(t), s f))) ds
exp (- log (8) + log (1) (V log 2(1), 8) (V log 2(1). ()) ds
((t), t)) / exp(- log ((s) + (V log Z(t), s) ds
exp(log ((t) exp - exp(
log log ((),,))) expl (V log x()) ds
= (t) exp( (V
(98)
Z(s)
Forming the ratio of the numerator (97) and the denominator (98):
exp( Ssp(-Diog 2 (t', t))
Z(t) (xt)exp((((
(V (V log 2(t), t))
exp( log Z(t).r))
2(1)
(99)
Ig (xxp(-Drog (s, ()) ds
(t)) exp( (V log tt), Is
exp( log 2(1).e)
ds
Z(a)
Simplifying the ratio:
exp( log Z(t).t))
exp( Exp(-Diog Dog z (t',t))
Z(H)
(100)
Is expl Diog 7 (8, ()) ds
Ss
expl(V 2(1).)) ds
Z(a)
Rewrite the terms in the exponent:
exp((V (V log Z(t), x))
exp( log 2)))x)) - log Z(x)) exp(- log Z(x)
(Vlog x(),
(101)
(xx
Multiplying the exponent by -1 and rearranging:
exp( (V log 7(t), x))
exp(-(log Z(z) - (Vlog x(t), exp(-(-(x, V log Z(t)) + log ((x)))
(102)
Z(x)
Further manipulation:
exp( (V log Z(t), x²) exp(- V log Z(t) + log Z(x))
(103)
Z(x)
Substituting this back into equation (100):
exp(- - 2 (1)
exp(- (1', V log Z(0) + log Z((')
(104)
Js exp( Drog 2 (s, f)) ds Js exp( (s. v log Z(t)) log Z(s) ds
This yields exactly the RHS expression.
Proposition A.8. Suppose that the (larget) data distribution is a bimodal mixture of two Gaussians, each with variance o²:
1 - Nox 11)
1 NN(x 1,0²).
po(x) =
(105)
The latent (source) distribution is the standard normal N(x 0, 1). Consider the variance-preservingg SDE
dx, - - 3xx,
3x, dt dBW.
(106)
19

---

Hessian Geometry of Latent Space in Generative Models
Then the Lyapunov exponent of the corresponding reverse-lime ODE at 0 has the following form:
B 1 - o² 2
==
1
(107)
and it diverges to infinity as a - O. In this case, the point 0 can be interpreted as a phase transition boundary.
Proof. We begin with the reverse probability flow ODE:
dXX,
(t))
- -( t)
V log (X.)),
(108)
ds
where the drift term is:
v(x) = -f(z, (z, + g()) 2
V log (x)).
(109)
Linearizing around 0, the Lyapunov exponent is defined as:
g()² d²
A = -=v(0)=-((0,)
2 dx (o).
(110)
The density (x) is computed via convolution with the Gaussian noising kernel:
(x) / Po (2 1 1 1 e- --d)ddy dy.
(111)
Since convolution of Gaussians is still Gaussian, we obtain:
(=(x)= = 22/2TO1
A(x).
(112)
where
A(z) exp (- (x (x- v((t)2
exp (- (x m(t))²
(113)
(t)
(0
and
(t)
Bt - (1 o-), p(t)= -8t
(114)
Computing the derivative in the definition of the Lyapunov exponent, get:
8 B e- - of(t) B ( (1+ e e of(())
A
(115)
2
(1)
2
of(1)
As time goes to 0, we have 0 - a. and thus:
B
(1 (1+ 02 2
-
(116)
ot
suggesting divergence of nearby reverse ODE trajectories for small σ, and identifying æ 0 as a potential phase transition boundary.
B. Experimental Details
B.I. Numerics
Mean-as-Stat The main idea of posterior-mean statistics is to predict the parameters based on the microstate by minimizing the regression error:
[(6)) Ee- (s) e112
(117)
20

---

Hessian Geometry of Latent Space in Generative Models
function la learned in this way will predict the mean of the posterior distribution P([]s).
To evaluate the quality of the free energy reconstruction, we compute the RMSE between the ground truth free energy (t) and the reconstructed free energy (t):
RMSE((Fres Fgt) min [P A(Frre(!))1ã
(118)
A
where we minimize over all possible affine transformations accounting for the fact that free energy only defined up to such transformation
Ising Model. Since the Ising model lacks an exact free energy solution for H * O, we construct the ground truth Fising (I' H) by numerically integrating the known magnetization MT, H) and energy E(T, H). The ground truth free energy Fising(2) (T, H) must satisfy the following partial derivative relations with respect to temperature and magnetic field::
I siinng (T, H) = E(T, H), OT
wing (T, , H) = (TT, H).
(119)
0H
To approximate Fising (T, H), we train a feedforward neural network FT(T, H) on the domain H >> 0, where the free energy C²-smooth. The approximation is extended И < using the symmetry Fising (T, - H) Fising (T, H). The loss function enforces consistency with the partial derivatives:
(()) / (1007 0T
OPa OH
E(T, H)
+
M(T
dTah.
(120)
method, we adopt similar approach to integrate the predicted sufficient statistics ST(T, H) and (T,, H) by minimizing the loss:
For the
f (122)
OFo
OF, 0H
2
[(0) -
ST (T(
+
(T, dTdH.
(121)
After obtaining the ground truth free energy, along with the free energy estimated by our Bayesian thermodynamic method, minimize the RMSE with respect to the ground truth over affine
integration and the saaerror-mmaa-a--sstttte transformations to account for the fact that free energy is only defined up to affine transformation. Finally, we evaluate and compare the resulting errors with the baseline.
TASEP Model. For TASEP, which has an analytical free energy solution, we use a similar neural network approach. The ground truth free energy given by:
a> 21 8 > 2 1 a(1 - aa,, a a 3 7
FIASEP(O, (0,
(122)
B(1 B), 8< a, 8<
The network Fo 3) is trained to minimize:
((0) 1
(1 OF. da
2 Ja 8)
0Fs Jola, 3)
+
dod3,
(123)
03
where J., and Jg are the particle currents at the boundaries, analogous to energy/magnetizztion in the Ising model. Finally, the resulting free energy surface is evaluated using the RMSE between the true TASEP free energy and the reconstructed free energy, after accounting for possible affine transformations:
RMSE(Freed Fyt) = min F (a, 9) - A(Frec(a, 3))B.
(124)
A
21

---

Hessian Geometry of Latent Space in Generative Models
B.2. Datasets
Ising Model. Our dataset consists of N 5.4 10 samples of spin configurations on the square lattice of size L x L 128 x 128 with periodic boundary conditions. We consider the parameter ranges 3 = =T [Tmin: Tmax] [1, H [Hmmn Hmaxl = 2- 2] similar to the ranges used in (Walker, 2019). Point (I, is sampled uniformly from this rectangle, and then a sample spin configuration is created for these values of temperature and external field by starting with random initial condition and equilibrating is with Glauber (one-spin Metropolis) dynamics (see, e.g. (Krapivsky et al., 2010)) for 10t x 128 x 128 Re 1.64 x 10t iterations. We represent spin configuration as a single-channel image with color of each pixel taking values +1 and -1. When constructing target probability distributions we choose o = 50 and set the discretization of the square [Tmin: mmax] [Hmin maxx [1, 5) [-2. 2] to be a uniform grid with L 128 x 128 grid cells.
Image-to-image network with U2-Net architecture (Qin al., 2020) is used to approximate posterior pe (t]ss. The network takes as input a bundle of Kundde images concatenated across channel dimension and outputs categorical distribution representing density values in discrete grid points. For simplicity we choose the discretization 'D to be of the same spatial dimensions as the input image. For all our numerical experiments the training was performed on a single Nvidia- HGG compute node with 8 A100 GPUs. We trained Ua-Net using Adam optimizer with learning rate 0.00001 and batch size of 2048 for NuzNet steps 20000 gradient update steps. all our experiments the training set consists of 80% of samples and the other 20% are used for testing.
TASEP Model. We generate a dataset of N = 150000 stationary TASEP configurations a 1d lattice with M 16384 sites. The rates (B) of adding (removing) particles at the left(right) boundary are sampled from the uniform prior distribution over a square [0. 1] x [0, 1]. To reach the stationary state we start from a random initial condition and perform Nsups = 2 x 10 ss 8M2 move attempts, which known to be enough to achieve the stationary state except for the narrow vicinity of the transition line a = 3 < 1/2 between high-density and low-density phases (in this case the stationary state has slowly diffuusigg front of shock wave in it, one needs of order M12 move attempts to form the shock but of order M move attempts for to diffusively explore all possible positions).
We reshape 1d lattice with M 16384 sites into an image of size T, I, 128 x 128 using raster scan ordering. To construct target probability distributions we set a 150 and define the discretization D as uniform grid on [amin- [mmxx] X [8min: 3max]] = [0. 1] x [0, 1] with L x L = 128 x 128 grid cells.
B.3. Net Training
Our task now is to estimate the posterior distribution p(t]s) using the set of samples tx. sn. To do that, we are trying to approximate p(t(s) by representatives from some parametric family of distributions (x choosing to maximize some cost function. is conventional to choose the cost function to maximize the likelihood of the true external parameters I; given 8; over all samples the set:
N
NegativeeooLLikeeiooo(())) = Σ (t.(p)(t.ls:)) =
(125)
RJ N EEr
(f(s)).
where on the right hand side we replaced the summands by their expected values. Minimization of the log-likelihood can be reinterpreted as the minimization of the KL. divergence between the target distribution
Phurger (t|(") = 5(t - t')
(126)
(i.e., the reconstructed labels / are identical to the input labels (') and the predicted distribution J ps(((s)p(ssttdds
(())= = EerP(() KL (Pansa(tte))] D po(t(s)p(s((tdds)
(127)
In practice, to avoid divergences it is convenient to replace the "hard label" target distribution (126) with a smoothened distribution
Ptanger (1|(') = C exp
(-
(- 20 11' 112)
(128)
where a is smoothening parameter and C' is the normalizing constant.
22

---

Hessian Geometry of Latent Space in Generative Models
B.4. Supplementary Figures
-logZ(a, B), free energy
POZZ0 MAKC B), free energy
1.0
1.0
0.8
0.8
1.0
1.0
0.8
0.4
0.8
0.6
0.6
0.4
0.4
a
0.0
a
0.0
0.0
Figure Free energy surface of diffusion model (n 0.1) reconstructed using CLIP distance. Noise injection smooths the free energy landscape, suppressing sharp phase boundaries (compare with Fig. 5).
Figure 10: Free energy surface of a StyleGAN v3 recon- structed using CLIP distance. The convex curvature indi- cates single dominant phase, contrasting with the multi- phase structure of diffusion models (left).
23

---

Hessian Geometry of Latent Space in Generative Models
Image
Clip Distance
Posterior Distribution
Target Parameters
Image
Clip Distance
Posterior Distribution
Target Parameters
Image
Clip Distance
Posterior Distribution
Target Parameters
Figure 11: Training examples for free energy reconstruction. Ground truth images with known generation parameters. CLIP-induced latent distribution. Predicted distribution from our convex model.
24

---

Hessian Geometry of Latent Space in Generative Models
Grid
0.0
0.0
0.2
0.4
0.4
0.6
0.8
1.0
1.0
a
2
3g
3aap
0.0
0.0
0.0
0.2
0.4
0.4
0.4
0.6
0.6
0.8
0.8
0.8
1.0
1.0
1.0
Figure 12: First- and second-order derivatives of the diffusion model's free energy. Discontinuities mark first-order phase transitions and second-order transitions. Regions of constant derivative correlate with visually distinct phases.
25

---

Hessian Geometry of Latent Space in Generative Models
Figure 13: Free energy derivatives of diffusion model's free energy reconstructed with clip distance. Overlaid grid highlights regions of constant derivative, correlating with visually distinct phases.
26


Reviews:
[{"review_id": "TCH7f8sexr", "reviewer": "Reviewer_ACqo", "summary": "This paper introduces a novel approach to exploring the geometric structure of latent\nspaces in generative models by estimating the Fisher metric and investigating phase\ntransitions. Through theoretical developments and empirical validation on statistical\nphysics models and Stable Diffusion. Key findings include the identification of\ndistinct phases within the latent space, fractal-like boundaries between these phases,\nand the property that geodesic interpolations are linear within phases while exhibiting\nnonlinear behavior at phase boundaries. Overall, this work provides new insights into\nthe latent space of generative models.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and hope that the response below will adress the raised points.\n\n- How confident are you that the proposed Fisher metric reconstruction method can be generalized beyond the specific models?\n\n    We are fairly confident it can generalize to other types of generative models, since it is justified by Bryant–Amari–Armstrong theorem for 2D section of a latent spaces. For higher-dimensional spaces it is not guaranteed to work. Beyond stable diffusion models, we tested the method with StyleGAN3, please see Fig.9: \"Free energy surface of a StyleGAN v3 reconstructed using CLIP distance\" in supplementary material. The learned surface has a little curvature and exhibits a single phase, which is in agreement with previously done research by Wang & Ponce (2021).\n  \n- Different feature extractors could yield varying results\n\n    During our evalutation done with CLIP, pixel-wise distances and PPL (perceprual path length) we obtained grid approximations of local metric following Wang & Ponce (2021). We observed that phase boundaries are stable under tested feature extractors. Because of this, we conclude that our algorithm will learn the same phase boundaries. Please see the General Response section in reponses to reviewers wCzF and GBhG for details.\n    \n- Scalability challenges in high-dimensional latent spaces\n\n  Our current method is justified for 2D section of high-dimensional latent spaces as we follow Amari and Armstrong (2013) and Bryant (2024).\n  Though the training procedure can be extended by CNN that predicts parameters of gaussian mixture models, the theoretical ground is the subject of futher research.\n  \n- Phase Transition: What underlying physical or mathematical principles explain these transitions?\n\n    We suppose that this phenomenon is related to the structure of image space, as studied in the ICLR 2023 paper “Verifying the Union of Manifolds Hypothesis for Image Data” by Brown et al. In this paper, it is shown that image data lies on a disjoint union of lower-dimensional manifolds with varying dimensions. The generative process of diffusion models begins with a Gaussian distribution defined on a full-dimensional latent space, and the reverse ODE trajectories end at significantly lower-dimensional, disjoint manifolds that represent real image data. Such a mapping—from a higher-dimensional unimodal distribution to a multimodal data manifold with disjoint supports for each mode—may exhibit a diverging Lyapunov exponent, or, in other words, a diverging Lipschitz constant for the generative mapping from the latent space to the data space.\n\n\nThis phenomenon can be illustrated by the following Proposition, which simulates a lower-dimensional data manifold with disjoint supports.\n\n# Proposition\nSuppose that the (target) data distribution is a bimodal mixture of two gaussians, each with variance $\\sigma$:\n$$\np_0(x) = \\frac{1}{2}\\mathcal{N}(x\\mid -1,\\sigma^2) + \\frac{1}{2}\\mathcal{N}(x\\mid 1,\\sigma^2)\n$$\nThe latent (source) distribution is the standard normal $\\mathcal{N}(x\\mid 0,1)$. Consider the Variance Preserving SDE\n$$\ndX_t = -\\frac{1}{2}\\beta X_t dt + \\sqrt{\\beta} dW_t\n$$\nThen the Lyapunov exponent of the corresponding reverse-time ODE at $x=0$ has the following form\n$$\n\\lambda=\\frac{\\beta}{2}\\left(1+\\frac{1-\\sigma^2}{\\sigma^4}\\right)\n$$\nand it diverges to infinity as $\\sigma$ goes to zero. In this case the point $x=0$ could be interpreted as a phase transition boundary.\n\n# Proof\nWe begin with the reverse probability flow ODE written as\n$$\n\\frac{dX_s}{ds}=-f(X_s,t)+\\frac{g(t)^2}{2}\\nabla_x\\log p_t(X_s),\n$$\nwhere the drift term is \n$$\nv(x)= -f(x,t)+\\frac{g(t)^2}{2}\\nabla_x\\log p_t(x).\n$$\nLinearizing around $x=0$, the Lyapunov exponent is defined by\n$$\n\\lambda=v'(0)=-f'(0,t)+\\frac{g(t)^2}{2}\\frac{d^2}{dx^2}\\log p_t(x).\n$$\nThe density p_t(x) is computed via convolution with the gaussian noising kernel\n$$\np_t(x)=\\int_{-\\infty}^{\\infty} p_0(y)\\mathcal{N}\\Bigl(x\\Big|e^{-\\frac{1}{2}\\beta t}y,1-e^{-\\beta t}\\Bigr)dy.\n$$\nSince convolution of gaussians is still gaussian we obtain\n$$\np_t(x)=\\frac{1}{2\\sqrt{2\\pi}\\sigma_{1}(t)}A(x)\n$$\nwhere\n$$\nA(x)=e^{-\\frac{(x-\\mu(t))^2}{2\\sigma_1^2(t)}}+e^{-\\frac{(x+\\mu(t))^2}{2\\sigma_1^2(t)}}\n$$\nand \n$$\n\\sigma_{1}^2(t)=e^{-\\beta t}\\sigma^2+(1-e^{-\\beta t})\n$$\nThen by computing derivatives in the definition of Lyapunov exponent we get\n$$\n\\lambda = \\frac{\\beta}{2} + \\frac{\\beta}{2} \\frac{e^{-\\beta t} - \\sigma_1^2(t)}{\\sigma_1^4(t)}\n= \\frac{\\beta}{2} (1 + \\frac{e^{-\\beta t} - \\sigma_1^2(t)}{\\sigma_1^4(t)})\n$$\nAs time goes to 0 we get $\\sigma_1 \\rightarrow \\sigma$, and have diverging Lyapunov exponent for small $\\sigma$\n$$\n\\lambda=\\frac{\\beta}{2}\\left(1+\\frac{1-\\sigma^2}{\\sigma^4}\\right)\n$$\nsuggesting divergence of close reverse ODE trajectories.\n\nEnd of the Proof.", "claims_and_evidence": "Claims : A novel method to reconstruct the Fisher metric in latent spaces of\ngenerative models.\n\nEvidence 1: The authors give theoretical foundations provided through Theorems\n3.1 and 3.2.", "methods_and_evaluation_criteria": "Using u2-net and CLIP for different generative models. The authors compare the\nRMSE in the statistical physical models experiment and did geodesic\ninterpolation analysis and phase transition identification in diffusion models\nexperiment.\n\nProblems : When doing the feature extraction, different feature extractors could\nyield varying results. What may happen if the authors use different feature\nextractors.", "theoretical_claims": "The research extends our understanding of generative models beyond traditional\nperspectives, revealing their intricate geometric structures and phase transition\ncharacteristics. Mathematical details are correct, the proof process is rigorous, and the\nuse of symbols is standardized. The paper is very solid in theoretical derivation.", "experimental_designs_or_analyses": "Experiment primarily validated on specific models(Ising and TASEP) and stable\ndiffusion models. But there exist Generalizability Concerns. The experiment primarily\nvalidated on specific models but it is unclear universal applicability across all\ngenerative models.", "relation_to_broader_scientific_literature": "none", "essential_references_not_discussed": "A brand new information geometry method and I cannot give relative references.", "other_strengths_and_weaknesses": "(1) Strengths:\n\nTheoretical Innovation: Bridges multiple disciplines: statistical physics,\ninformation geometry, machine learning\n\nMethodological Contributions: Novel approach to reconstructing Fisher metric\nand give theoretical convergence guarantees\n\n(2) Weakness:\n\nGeneralizability Limitations: what about other feature extractors and generative\nmodels ?\n\nComputational Complexity: Theoretical proofs suggest scaling challenges and\nthere may potential computational overhead for large-scale models", "other_comments_or_suggestions": "none", "questions_for_authors": "1) Generalizability: How confident are you that the proposed Fisher metric\nreconstruction method can be generalized beyond the specific models and about\ndifferent generative models\n2) Feature Extraction Sensitivity: How robust is your method to different feature\nextraction techniques\n3) Computational Complexity: could you addressing scalability challenges in high-\ndimensional latent spaces?\n4) Modeling application: how could apply this analysis theory and improve the\nmodel performance ?\n5) Phase Transition: observed fractal-like boundaries in latent spaces? What underlying physical or\nmathematical principles explain these transitions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "none"}}, {"review_id": "lvPwJP2hTf", "reviewer": "Reviewer_Evof", "summary": "This paper leveraged the information geometry framework to understand the manifold geometry in statistical mechanics models parameter space and generative model latent space. \n\nThey provided nice theoretical results connecting the posterior distribution of parameter given infinite samples and the Bregman divergence of the parameter and the true ones. Further they tried to use this connection to learn the partition function Z, whose Hessian gives us the Fisher metric of the manifold. Then they use neural networks to estimate posterior distributions given samples and to learn the log partition function. They validated the pipeline on some classic statistical physcis model showing it can learn the free energy better than baseline. They also used it to study the latent space geometry in diffusion models, which elucidates a fractal structure in latent space.\n\n## Update after rebuttal\nThe authors clarified most conceptual concerns regarding the method. (esp. the concerns about latent vector falling off shell in the final interpolation experiment.) The reviewer is happy to remain their score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank author for the elaborated questions regarding theory and method.\n\n- The proposed approach seems a bit overly complex, not clear whether it is better than established approach, that is, pulling back the euclidean metric in feature space [Wang & Ponce 2021] [Shao, Kumar, & Fletcher, 2018].\n\n  We admit that the proposed algorithm is more complex than Wang et al. (2021) and Shao et al (2018). However, we would like to highlight that this complexity comes from the broader coverage of the algorithm, namely, its ability to work with stochastics sampling, where pulling back the euclidean metric in feature space is impossible and results in stochastic metric tensor.\n\n\n- Eq.19 is not very different from the previous Jacobian formulations\n\n  For numerical comparison with previous work please refer to the General Response section in responses to reviewers wCzF and GBhG for details. \n\n- The transition from Eq.21 to 24. esp. 22. Why we can learn $log Z_θ(t)$ by minimizing the JS divergence in Eq.22? Do we have some theoretical properties about the objective in Eq.22? \n\n  Regarding the eq.22, in general one could use other loss functions for distance between distributions. We discuss after Theorem 3.1 that MSE loss exhibits vanishing gradients and thus is not suitable in our case. Since Jensen-Shannon divergence is the proper metric on probability distributions therefore its convergence to zero guarantees the convergence of MSE loss.\n\n- Also does Eq. 21 require the latent space you are studying is compact, or the integration is impossible?\n\n  Yes, latent space has to be compact and we require that integration over it is possible.\n\n- If the generative mapping is deterministic (in a case delta measure), does the theory still work?\n\n    Since delta measure could be uniformly approximated by Gaussians and for any Gaussian the theory is justified, it can be proven the approach still works in the limit. \n\n- This design has the risk of falling off the manifold of Gaussian hypersphere\n\n  Indeed, this is a valid point which is missed in the main text. To prevent our interpolation to fall off the Gaussian hypersphere in all our experiments we employed a normalization of latent vector z.", "claims_and_evidence": "- In the paper it sometimes said colloquially, “*estimating the log partition function log Z(t) by training a network to simulate p(t|x)*” But we can only know the log Z up to affine transform. So maybe we can qualify the claim about estimating log Z(t) ?\n- The formulation makes a lot of sense for statistical mechanics setting, and I’m happy to see it learning the correct partition function. \nBut for the generative image model latent space setting (with feature extractor), it seems a bit overly complex. I’m not clear / convinced that this framework is better / more informative than the previous ones to compute the Riemannian metrics for the latent space of generative models (see related works [1,2]). So I don’t buy the claim that “*Notably, the approach proposed below is theoretically justified for any generative model*”\n\n[1] Shao, H., Kumar, A., & Thomas Fletcher, P. (2018). The riemannian geometry of deep generative models. \n\n[2] Wang, B., & Ponce, C. R. (2021). The geometry of deep generative image models and its applications. ICLR", "methods_and_evaluation_criteria": "Yes. Use of solvable statistical physics models as a test case suits the theoretical framework of the authors. \n\n**Question:** \n\n- **Major point** \nThe Eq 16-19 is a nice theoretical formulation for using feature extractor. \nBut in the end seems Eq.19 is not very differently from the previous Jacobian formulations, i.e. computing squared distances in some feature space and pulling back to the latent space. [2] In the end if we are using feature extractors, how does the metric you got for diffusion latent manifold similar / different from the metric obtained from pulling back the euclidean metric in feature space? \nThat could be done by simple finite difference method if you have all the sample images, no need to backprop through the diffusion sampling process. I guess it may not be more expensive than the MLP training in your cae.\n- When you are using feature extractors, why should we learn log Z(t) ? I feel learning a a bivariable distance function d(t, t’) could also be possible or even faster to approximate the D_log Z(t,t’) directly.\n- **Minor point:** In methods Sec 3.1, the neural network design were quite suitable for 2d settings, i.e. the  Ising models are 2d and the parameter space is also 2d, so you can do 2d→2d unet mapping. But seems it’s hard to generalize this to higher dimensional parameter space?\n\n[2] Wang, B., & Ponce, C. R. (2021). The geometry of deep generative image models and its applications. ICLR", "theoretical_claims": "I skim through proof of Theorem 3.1 and Lemma 3.3 which seems to be correct. I didn’t check theorem 3.2 \n\n**Question:** \n\n- **Major point:** \nI cannot quite follow the logic from Eq.21 to 24. esp. 22. Why we can learn *log Zθ(t)* by minimizing the JS divergence in Eq.22? Is it because Theorem 3.1? Do we have some theoretical properties about the objective in Eq.22? Also does Eq. 21 require the latent space you are studying is compact, or the integration is impossible?\n- **Minor point:** \nfrom Eq. 16-19 the formalism is slightly off, I feel it’s mixing the deterministic sampling case and the stochastic sampling case…. esp. Eq. 19\n- **Minor point:** In the degenerate case, where the generative mapping is deterministic, or $p(x|t)$ is delta measure, does the theory still work? This is relevant since for diffusion, if sampler is an ODE solver, then the mapping is deterministic. Similarly for latent space of GAN. I feel authors need to show this to justify the claim “*approach proposed below is theoretically justified for any generative model*”\n    - My hunch is that it (Eq. 4) will fall back to the Jacobian metric, which is the pull back metric of Euclidean metric in x space. Then it will be quite similar to the metric studied by previous works [0][3].\n\n[1] Shao, H., Kumar, A., & Thomas Fletcher, P. (2018). The riemannian geometry of deep generative models. \n\n[2] Wang, B., & Ponce, C. R. (2021). The geometry of deep generative image models and its applications. ICLR", "experimental_designs_or_analyses": "- Experimental design and results in Fig. 3-4 are convincing, showing that the inference method can recover known results.\n\nQuestions: \n\n- **Major point: Random selection of 3 initial latents and chart the 2d plane between them.** \nI feel this design has the risk of falling off the manifold of Gaussian hypersphere. Since the trained latents were sampled from N(0,I), as long as your z_0,z_1,z_2 are not too close, many of their interpolations will be “off” the distribution and have incorrect norm, thus I’m not sure the score function / the sampler can do the right thing for those samples. \nCan you consider a 2d submanifold on the sphere, similar to the way people interpolate latent space in GAN era ([3] [4] Fig1E).\n    - In previous works, people seems to find the manifold can look quite continuous if we perturb the initial state in the “good directions” found by PC (see [5] Fig.5 and supp). So I’m not sure if the fractal structure you found in your Fig. 6 is because we are charting a subspace in latent space that were unnatural for diffusion models.\n- **Minor point:** For Fig. 6, it’s very cool to show this fractal boundary perceptually, is there a way to quantify the “fractalness” of it?\n- **Minor point:** For Fig. 7C, we see pretty linear border separating the “phase” on the 2d space. I feel part of it is due to the model you used to approximate $\\log Z(t)$ is a MLP with ReLU, which is piecewise linear. Thus their separating boundary have to be piecewise linear in a sense.\n\n[3] White, T. (2016). Sampling generative networks. arXiv preprint arXiv:1609.04468.\n\n[4] Wang, B., & Ponce, C. R. (2022). Tuning landscapes of the ventral stream. *Cell Reports*, *41*(6).\n\n[5] Wang, B., & Vastola, J. J. (2023). Diffusion models generate images like painters: an analytical theory of outline first, details later. *arXiv:2303.02490*.", "relation_to_broader_scientific_literature": "- The multiple links drawn between statistical physics and generative models were intriguing, different from many other previous links in DL theory.", "essential_references_not_discussed": "In the introduction section “*Riemannian geometry of latent space*” the authors forgot to cite / discuss one important paper along this line [^2], which also managed to compute and analyzed the structure of a Riemannian (Hessian) metric for the latent space of GANs. Since GAN is a deterministic mapping from latent space to image space, their Hessian metric of latent space is derived by pulling back a image distance function (e.g. LPIPS) and compute its Hessian at each point. Thus, in their case the potential function is $\\phi_x(x’)=D(G(x),G(x’))$. The potential function is local to each point, and not necessarily global. But that framework can deal with higher dimensional manifolds > 2d, though without the gurantee as in *(Bryant–Amari–Armstrong)*. \n\nThe overall conceptual framework of the current paper share many similarities with [3], so it would be nice to discuss the connection and shared spirit with them. \n\n[2] : Wang, B., & Ponce, C. R. (2021). The geometry of deep generative image models and its applications, ICLR, 2021", "other_strengths_and_weaknesses": "- The authors put efforts into the formalism and well made illustrations; they helped understanding a lot.\n- The connection the authors draw between generative models and statistical physics models are interesting and quite original.\n- Fig. 6 the visualization of the fractal and self-similar structure of the phase boundary within diffusion model latent space is quite novel and intriguing!", "other_comments_or_suggestions": "No.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9LsYyAQZ1t", "reviewer": "Reviewer_wCzF", "summary": "This paper proposes a novel method to approximate the Fisher information metric. It shows superior performance to concurrent methods on Ising and Tasep models. Applied to diffusion models, it reveals a fractal structure of the latent space, and sharp transitions. It also allow for smoother interpolations.\n\n## Update\n\nI will keep my rating as is, since I won't be able to evaluate the paper revision and the newly added presentation/details. Moreover, experimental results are still not particularly strong and convincing when considering std of metrics.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting gaps in our evaluation and presentation. The revision will strengthen empirical validation with baselines and metrics and improve clarity via algorithms and reproducibility details. To address concerns about comparison with other methods please refer to tables below. We hope these changes will address the concerns and demonstrate the method’s utility for analyzing generative model geometries.\n# The General Response\nWe would like to thank the reviewers for recognizing our method to approximate Fisher information metric as novel and theoretically grounded.\nSince most reviews raise the question of comparing our work with Wang & Ponce (2021) and Shao et al. (2018), we would like to provide a detailed comment on the distinction between our proposed method and these prior works.\n\nShao et al. studies VAE and computes geodesic curves based on pullback metric induced by euclidean metric in pixel space. Wang et al. consider GAN models and defines metric on the latent space via pullback of L2 distance in the feature space of VGG-19 (LPIPS distance). In both cases generative models are deterministic: each latent Z produces only a single image X.\n\nThe key distinction of our work lies in its consideration of a broader class of models, specifically, models with stochastic generation, where a single latent Z corresponds to a distribution in the image space p(X|Z). Diffusion models with stochastic sampling (and all statistical physics models) cannot be addressed within the formalism of Shao et al. (2018) or Wang et al. (2021).\n\nTo fulfill the request of the reviewers we compare our algorithm with Shao et al. (2018) and Wang et al. (2021) in deterministic sampling regime. We get pullback metric from the euclidean metric in the space of CLIP embeddings. The Jacobian is estimated via finite differences. We use three evaluation scores: CLIP, pixel and Perceptual Path Length (PPL), which measure the average path length in CLIP embedding space, pixel space and feature space of VGG-19. We compute them as the cumulative L2 distance between consecutive feature vectors (or images for pixel space), then average this perceptual divergence measure across multiple trajectories.\n\n    Average CLIP Lengths ± std (bootstrap):\n    --------------------------------------------------\n    Geodesic ours: 72.2951 ± 3.7516\n    Linear:        73.6076 ± 3.5428\n    Geodesic:      73.5764 ± 4.3664\n\n\n    Average Pixel Lengths ± std (bootstrap):\n    --------------------------------------------------\n    Geodesic ours: 2769146.3818 ± 23813.2801\n    Linear:        2757206.0770 ± 27731.2199\n    Geodesic:      2739432.4217 ± 35251.5666\n\n\n    Average Perceptual Path Lengths ± std (bootstrap):\n    --------------------------------------------------\n    Geodesic ours: 3.1172 ± 0.1587\n    Geodesic:      3.1857 ± 0.2134\n    Linear:        3.1725 ± 0.2256\n\n\nWe observe that our interpolation is on par with Shao et al. (2018) and Wang et al. (2021) in the case of deterministic sampling. We additionally compute the curvature of each trajectory as the mean angular change per unit length between consecutive path segments. Our analysis reveals that trajectories constructed using the Wang metric show significantly higher curvature and frequent turning compared to our method. We attribute this behavior to the finite differences, which introduce high frequency noise in metric components. In the case of diffusion, the Jacobian is hard to obtain via backpropagation as suggested in Shao et al. (2018) due to high computational cost.\n\n    Average Mean Curvature ± std (bootstrap):\n    --------------------------------------------------\n    Geodesic ours: 0.3671 ± 0.6909\n    Geodesic:      1.3261 ± 0.5328\n    Linear:        0.0000 ± 0.0000", "claims_and_evidence": "The paper claims that their method allows a good approximation of the Fisher information metric, which allows its use in thermodynamic models and on analysing latent space of diffusion models. To me, the empirical evidence is not strong enough (see below).", "methods_and_evaluation_criteria": "Method: The method is sound and well thought. \n\nEvaluation criteria: The first evaluation criteria is a validation on exact statistical models for thermodynamics. I do not know well these models, and thus I am unsure of the interest of these results. \nThe second evaluation criteria is on the analysis of the latent space of diffusion models. The main weakness is the lack of comparisons with other methods. Why does the proposed method reveal things that would not be revealed by other methods? For example, it would be worth adding a simple baseline measuring the generator's local curvature (e.g. estimated with finite difference), which could also reveal these phase transitions. Moreover, it lacks metrics. Validating an idea based on a few visualizations is not rigorous enough. A potential metric could be the Perceptual Path Length introduced in StyleGan paper.", "theoretical_claims": "I did not check the proofs. However, let us note that the theory proves the feasibility of their method rather than its superiority compared to other methods.", "experimental_designs_or_analyses": "As mentioned above, the evaluation criteria is weak in my opinion. The first part is on toy data. The second part lacks comparisons with baselines.", "relation_to_broader_scientific_literature": "The method described here is a novel method to approximate Fisher information metric in my knowledge.", "essential_references_not_discussed": "Not essential but would be worth discussing: 1: \"Metrics for Deep Generative Models\", Chen et al., AIStats 2018. \n2: \"Optimal transport maps for distribution preserving operations on latent spaces of Generative Models.\", Agustsson et al., 2018.", "other_strengths_and_weaknesses": "Additionally to the lack of strong experiments, a second weakness is the paper clarity and the low level of details. To me, the method is not detailed enough. For example, what is described in 3.1 and 3.2 should be summarized in an Algorithm. Moreover, details required to reproduce the experiments are lacking, which is a problem.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qb8MOTQKGx", "reviewer": "Reviewer_GBhG", "summary": "## Update After Rebuttal\n## I maintained my score. Please see my response to the reviewer below for my reasons.\n----\nThis work presents a novel technique for analyzing latent space geometry in diffusion models. Based on the reconstruction of the Fisher Information metric, this method approximates the posterior distribution of latent variables given synthetic samples generated from a diffusion model, and uses this information to learn the log-partition function of the variable $t$. To develop this method, the work relies heavily on the theoretical and mathematical works done in\n[Amari and Armstrong (2013)](http://arxiv.org/pdf/1312.1103) and [Bryant (2024)](https://arxiv.org/abs/2405.06998) for its own derivations and assumptions. \n\nWhen applied to diffusion models, this new method highlights structures of phase transition in the latent space, parameterized by interpolation values $\\alpha$ and $\\beta$ which represent $t$, allowing the authors to find geodesics or shortest possible paths between latent variables --- highlighting the complex behavior of the generation process in diffusion models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate provided feedback and agree that the paper will benefit from additional quantitative studies.\n\n- Providing additional quantitative result, like Table (2) of Shao et al. (2018)\n\n  Please refer to the General Response below.\n\n- Adding experiments for non-stable diffusion generative models\n\n  In addition to stable diffusion, we have applied the proposed approach to StyleGAN3 (2021). However, it resulted in geodesics close to linear interpolation and overall our findings coincide with Wang & Ponce (2021), so these experiments are not included in the main paper text. Please find Figure 9: \"Free energy surface of a StyleGAN v3 reconstructed using CLIP distance\" in supplementary material.\n\n- Writing an algorithm section detailing how the model is trained, including the details of the algorithm found in Shao et al. (2018)\n\n  We agree that these algorithmic details should be written explicitly.\n\n# The General Response\nWe would like to thank the reviewers for recognizing our method to approximate Fisher information metric as novel and theoretically grounded.\nSince most reviews raise the question of comparing our work with Wang & Ponce (2021) and Shao et al. (2018), we would like to provide a detailed comment on the distinction between our proposed method and these prior works.\n\nShao et al. studies VAE and computes geodesic curves based on pullback metric induced by euclidean metric in pixel space. Wang et al. consider GAN models and defines metric on the latent space via pullback of L2 distance in the feature space of VGG-19 (LPIPS distance). In both cases generative models are deterministic: each latent Z produces only a single image X.\n\nThe key distinction of our work lies in its consideration of a broader class of models, specifically, models with stochastic generation, where a single latent Z corresponds to a distribution in the image space p(X|Z). Diffusion models with stochastic sampling (and all statistical physics models) cannot be addressed within the formalism of Shao et al. (2018) or Wang et al. (2021).\n\nTo fulfill the request of the reviewers we compare our algorithm with Shao et al. (2018) and Wang et al. (2021) in deterministic sampling regime. We get pullback metric from the euclidean metric in the space of CLIP embeddings. The Jacobian is estimated via finite differences. We use three evaluation scores: CLIP, pixel and Perceptual Path Length (PPL), which measure the average path length in CLIP embedding space, pixel space and feature space of VGG-19. We compute them as the cumulative L2 distance between consecutive feature vectors (or images for pixel space), then average this perceptual divergence measure across multiple trajectories.\n\n    Average CLIP Lengths ± std (bootstrap):\n    --------------------------------------------------\n    Geodesic ours: 72.2951 ± 3.7516\n    Linear:        73.6076 ± 3.5428\n    Geodesic:      73.5764 ± 4.3664\n\n\n    Average Pixel Lengths ± std (bootstrap):\n    --------------------------------------------------\n    Geodesic ours: 2769146.3818 ± 23813.2801\n    Linear:        2757206.0770 ± 27731.2199\n    Geodesic:      2739432.4217 ± 35251.5666\n\n\n    Average Perceptual Path Lengths ± std (bootstrap):\n    --------------------------------------------------\n    Geodesic ours: 3.1172 ± 0.1587\n    Geodesic:      3.1857 ± 0.2134\n    Linear:        3.1725 ± 0.2256\n\n\nWe observe that our interpolation is on par with Shao et al. (2018) and Wang et al. (2021) in the case of deterministic sampling. We additionally compute the curvature of each trajectory as the mean angular change per unit length between consecutive path segments. Our analysis reveals that trajectories constructed using the Wang metric show significantly higher curvature and frequent turning compared to our method. We attribute this behavior to the finite differences, which introduce high frequency noise in metric components. In the case of diffusion, the Jacobian is hard to obtain via backpropagation as suggested in Shao et al. (2018) due to high computational cost.\n\n    Average Mean Curvature ± std (bootstrap):\n    --------------------------------------------------\n    Geodesic ours: 0.3671 ± 0.6909\n    Geodesic:      1.3261 ± 0.5328\n    Linear:        0.0000 ± 0.0000", "claims_and_evidence": "## Claim \n\n(1) For two-parameter systems, like the classic Ising model and TASEP system, the work introduces metric which reconstructs Fisher metric outperforms existing baselines designed for reconstructing thermodynamics quantities. \n\n(2) The introduced method when applied in diffusion models demonstrate fractal structure of phase transitions in latent space, illustrated by abrupt changes in the reconstructed Fisher metric.\n\n(3) The interpolation based on geodesics' information illustrates a smoother interpolation in contrast to the conventional linear interpolation. Moreover, the authors also claim that the diffusion model exhibits a divergent Lipschitz constant with respect to the latent space at phase boundaries, using geodesics' information as part of their analysis.\n\n\n## Evidence\n\n(1) To support the first claim, the work first show that the posterior $p(t | x)$ satisfies \n$\\underset{N \\rightarrow \\infty}{\\text{lim}} \\big ( p(t | x_1, \\dots x_N) \\big )^{1 / N} \\overset{a.s.}{=} \\mathrm{e}^{-D_{\\log Z(t)} (t, t')}$ where $D_{\\log Z(t)} (t, t') = \\log Z(t) - \\log Z(t') - \\langle \\nabla_t' \\log Z(t'), \\, t - t' \\rangle$ is the Bregman Divergence. This full proof is detailed in Appendix A. There are some assumptions made, but the main assumption is that the general distribution belongs to the exponential family. \n\nThen, with the posterior $p(t | x_1, \\dots x_N)$, the log-partition function $log Z(t)$ can be estimated with the trained parameter $\\theta$ using the following loss \n\n$\n    \\mathcal{L}(\\theta) = \\int_S \\mathrm{D}_\\mathrm{JS} \\big ( p(t | x_1, \\dots x_N) \\,\\, || \\,\\,\\, p_{\\log Z_\\theta}  (t | t') \\big ) dt'\n$\n\nUsing $\\mathcal{L}$, they computed the free energy of the Ising model and TASEP respectively. They recorded the free energy RMSE (root mean squared error) in Table (1), as well as the RMSE of the partial derivatives w.r.t the function $F$ or the Hamiltonian (total energy). Based on Table (1), it's clear that the work's method performs better since it has much lower RMSE than its baselines. \n\nThe accuracy of the computed free energy is verified with the depiction of the actual free energy of the two simple models, see figures (3 and 4).\n\n\n(2) To tackle the second claim, the authors rely on the usage of a feature extractor $\\mathcal{E}$ (in this case CLIP) to approximate the posterior $p(t | x_1, \\dots x_N) \\approx \\mathrm{e}^{- \\frac{N}{2} \\lVert \\mathcal{E}(x) - \\mathcal{E}(x') \\rVert^2}$, where $x \\sim p(x|t)$ and $x' \\sim p(x | t')$, for the high-dimensional (or image) setting.\n\nUsing the loss function $\\mathcal{L}$ (mentioned above), the authors trained an MLP which represents $g_F (t) = \\nabla^2 \\log Z_{\\theta^*} (t)$ obtained via $\\theta^* = \\text{argmin}_\\theta \\mathcal{L} (\\theta)$. Using this network, they are able to observe the phase transitions detailed in figure (7). Keep in mind, they represent t as two interpolation parameters, $\\alpha$ and $\\beta$. \n\n(3) With the fisher metric $g_F(t) = \\nabla^2 \\log Z_{\\theta^*} (t)$, the method is extrapolated to explore the geodesics' geometry, or shortest path geometry. To do this, the authors rely on the work of [Shao et al. (2018)](https://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w10/Shao_The_Riemannian_Geometry_CVPR_2018_paper.pdf) for its approach to find the geodesic along a perturbation/interpolation trajectory of two images, by minimizing the curve length $L[\\gamma(t), g_f(t)]$ (see Eq. 25). \n\nAlthough the results are mostly qualitative, we can see that in figures (6 and 7) as well as those in the Appendix, figures (11 and 12) --- that the geodesics' interpolation obtained via minimizing $L$ is much smoother. \n\n## Strength\n\n(1) This is a very interesting paper which presents a method on obtaining information from the (intractable) partition function of diffusion model. Through some formulations, the partition function (based on t), which the work uses to analyze the behavior of diffusion model, is shown to be very informative. Moreover, the results are very interesting.\n\n(2) The formulations/derivations are rigorous and based on or inspired heavily by works done by [Amari and Armstrong (2013)](http://arxiv.org/pdf/1312.1103) and [Bryant (2024)](https://arxiv.org/abs/2405.06998). Hence, I think the paper has a nice story in which it follows very well. The presentation is good.\n\n\n## Weakness\n\n(1) With the exception of Table (1), in general the results do feel more qualitative than it is quantitative. Also, the results in Table (1) indicate that for the two simple models. It would be nice to have something similar for comparing geosidics and linear interpolation. Perhaps, something like Table (2) of [Shao et al. (2018)](https://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w10/Shao_The_Riemannian_Geometry_CVPR_2018_paper.pdf). \n\n(2) For the diffusion model analyses, the experiment was conducted only on a stable diffusion model. It would be nice to see this behavior for non-stable diffusion models as well.", "methods_and_evaluation_criteria": "In general, I believe the evaluation makes sense. Unfortunately, there are no baselines for the proposed method in high dimensional, except for the 2D model case (since you can compute the actual free energy for Ising and TASEP). It is rather difficult to have actual baselines (to my knowledge).", "theoretical_claims": "The paper definitely makes a lot of theoretical claims. Thus, relying heavily on works done by others, the authors were able to rigorously formulate their approach and also proofs behind it. See Appendix A for the most important proof.", "experimental_designs_or_analyses": "Yes, I did. Please see what I've said above. Nonetheless, I do not see problems with their experimentation, but rather I think their mathematical works can be written and organized slightly better.", "relation_to_broader_scientific_literature": "I think this work is pretty important. We know that diffusion models are just energy-based models (EBMs), which learns the score or the gradient of the energy function (or log-p). Unlike normal EBMs, to compute the log-likelihood or an estimate to the energy, we have to perform a numerical integration of the probability flow ODE, detailed in [Song et al. (2021)](https://arxiv.org/abs/2011.13456). This is simply a way of telling how good a diffusion model learns. But it doesn't fully tell us how diffusion models behave. \n\nThis work presents an interesting approach and a point of view of how we can interpret the intractable partition function, which we avoided in training diffusion models, such that we can analyze the behavior of diffusion models.", "essential_references_not_discussed": "Since you mentioned Ising model, I think it's also fair to cite Hopfield models in the work as well for relevancy. See [Amari, s-i. (1972)](https://ieeexplore.ieee.org/document/1672070) and [Hopfield J. (1982)](https://www.pnas.org/doi/10.1073/pnas.79.8.2554) \n\nAnother interesting work which relies on the analysis of the Jacobian of the score or Hessian of the energy is [Achilli et al. (2024)](https://arxiv.org/pdf/2410.08727?).", "other_strengths_and_weaknesses": "See Claim and Evidence Section", "other_comments_or_suggestions": "Overall, I believe this is a good work, but it needs an additional quantitative result. For example, something like Table (2) of [Shao et al. (2018)](https://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w10/Shao_The_Riemannian_Geometry_CVPR_2018_paper.pdf) is probably sufficient. \n\nMoreover, there are minor notation errors that should be correct --- and the semantics of some variables (especially those in the Appendix) should be included to make it clear to the readers. \n\nIn the main text:\n\n(1) Eq. (22) is labeled as $\\mathcal{L}_1$ but shouldn't it just be $\\mathcal{L}$ instead? I don't see this subscript 1 being used at all. \n\n(2) In section 4.1, on line 304-305, you should use $\\partial$ instead of $\\mathrm{d}$ to indicate partial derivative. \n\nIn the Appendix:\n\n(1) On line 617, it should be \"where $\\psi(s, x)$ ...\" instead of \"where $\\psi(x, s)$ ...\" just for consistency.\n\n(2) What exactly does the variable $\\mathbf{N}$ mean on lines 720 and 753? Do you mean to use $\\mathbb{N}$ instead?\n\nFinally, here is my ***main suggestion***. I think you should write an algorithm section detailing how you train your model $g_F(\\theta, t)$. It should include the usage of the feature/encoder model. Furthermore, please include the details of the algorithm found in [Shao et al. (2018)](https://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w10/Shao_The_Riemannian_Geometry_CVPR_2018_paper.pdf) which you relied on, in the Appendix. It will help keep things clear for the readers.", "questions_for_authors": "See Above Section", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: There is an even split between the scores of the reviewer, but sadly there was no discussion whatsoever on the topic of reaching a consensus. On the positive side, there was a healthy interaction between the reviewers and the authors post-rebuttal.

I am convinced regarding the novelty and the experimental evaluation of the proposed method.
I would strongly suggest the authors take the time to update the manuscript so it incorporates the reviewers' feedback. I believe the additional results around the Avg mean curvature belong in the main paper and I would like to see a nice discussion around them.

Congratulations!
Decision: Accept (poster)