Identifying Metric Structures of Deep Latent Variable Models
Authors: Stas Syrota
Yevgen Zainchkovskyy
Johnny Xi
Benjamin Bloem-Reddy
Søren Hauberg
Venue: ICML 2025 poster (2025)
Keywords: Identifiability
LVMs.
Primary area: theory->everything_else
TL;DR: We show that geodesic distance measure in the latent space of a deep latent variable model is statistically identifiable.

Abstract: Deep latent variable models learn condensed representations of data that, hopefully, reflect the inner workings of the studied phenomena. Unfortunately, these latent representations are not statistically identifiable, meaning they cannot be uniquely determined. Domain experts, therefore, need to tread carefully when interpreting these. Current solutions limit the lack of identifiability through additional constraints on the latent variable model, e.g. by requiring labeled training data, or by restricting the expressivity of the model. We change the goal: instead of identifying the latent variables, we identify relationships between them such as meaningful distances, angles, and volumes. We prove this is feasible under very mild model conditions and without additional labeled data. We empirically demonstrate that our theory results in more reliable latent distances, offering a principled path forward in extracting trustworthy conclusions from deep latent variable models.

Identifying Metric Structures of Deep Latent Variable Models
Stas Syrota' Yevgen Zainchkovskyy Johnny Xi²
Benjamin Gloem-Reddy 2 Soren Hauberg
Abstract
Latent representation
Deep latent variable models learn condensed rep- resentations of data that, hopefully, reflect the inner workings of the studied phenomena. Un- fortunately, these latent representations are not statistically identifiable, meaning they cannot be uniquely determined. Domain experts, therefore, need to tread carefully when interpreting these. Current solutions limit the lack of identifiability through additional constraints the latent vari- able model, c.g. by requiring labeled training data, or by restricting the expressivity of the model. We change the goal: instead of identifying the latent variables, we identify relationships between them such as meaningful distances, angles, and vol- umes. We prove this is feasible under very mild model conditions and without additional labeled data. We empirically demonstrate that theory results in more reliable latent distances, offering principled path forward in extracting trustworthy conclusions from deep latent variable models.
Latent Euclidean distances
Figure 1: Latent representations of transcriptomic data (top row) changes with model retraining. Each column corre- sponds model trained from scratch. The latent variables are not identifiable and change between training runs. Pair- wise Euclidean distances (bottom row, averaged across cell types) also change significantly between runs. This lack of identifiability prevents us from reliably using the latent representations to understand the underlying biology.
1. Introduction
Latent variable models express the density of observational data through set of latent, i.e. unobserved, variables that idcally capture the driving mechanisms of the data- generating phenomena. For example, the latent variables of a variational autoencoder (Kingma & Welling, 2013; Rezende et al., 2014) trained on protein data, can reveal the underlying protein evolution which can help domain experts understand a problem of study (Riesselman al., 2018; Ding et al., 2019; Detlefsen et al., 2022).
variables as the analysis becomes subject to the arbitrariness of model training. Fig. exemplifies the issue, where two independently trained latent representations are estimated from transcriptomic data (Tasic al., 2018). While clusters are similarly estimated across models, the relationships between clusters vary significantly, preventing from extracting intra-cluster information from the plots. This fundamental issue has sparked the development of training heuristics to limit the issue (Kobak & Berens, 2019), and the practice of analyzing latent variables has been both disputed (Chari & Pachter, 2023) and defended (Lause et al., 2024). This entire discussion could have been avoided only distances between latent variables had been identifiable.
Unfortunately, latent variables are rarely identifiable, i.e. they cannot be uniquely estimated from data. This lack of uniqueness prevents reliable analysis of the learned latent
'Department of Applied Mathematics and Computer Science, Technical University of Denmark. Lyngby, Denmark 'Department of Statisties, University of British Columbia. Correspondence to: Stas Syrota <stasy @dk
Providing identifiability guarantees has been heavily inves- tigated. We survey key results in Secs. 2 and 6, but the exec- utive summary that current approaches require significant model restrictions (c.g. linearity assumptions), labeled data, or a combination of both. These solutions are underwhelm-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Identifying Metric Structures of Deep Latent Variable Models
ing to practitioners who often lack a labeling mechanism and are keen to leverage contemporary generative models.
Latent space
Observation space D
Probubilistic PCA
fa
In this paper, we change the identifiability question to ar-
rive at working tools that do not require data labels and
only impose minimal restrictions on the used models. In-
stead of identifying the latent variables, we identify the
A.. 1 15
fb
Po. = Po
relationship between latent variables. For example, instead of identifying the coordinates of the latent variables we iden- tify pairwise distances. In our experience, this matches the needs of domain experts who rarely assign meaning to the coordinates of latent variables. Using differential geome- try, we prove strong identifiability guarantees on pairwise distances, angles, and more. We empirically validate our theory on four different generative models.
fm
D
General case
f+ Po. Po.
Figure 2: Indeterminacy transformations characterize the identifiability equivalence class. Top row: Probabilistic PCA has linear decoders, such that the indeterminacy trans- formations are rotations. Bottom row: In general deep latent variable models the indeterminacy transformations are the general class of diffeomorphisms acting on the latent space.
2. Background and notation
Before stating our main questions and results, we recap the prerequisite background information. We position our work relative to the existing literature in Sec. 6.
Deep latent variable models learn densities of data X D parametrized by latent variables Z E Z, such that p(X)= = I ((X(ZZ)(ZZ)l (Tomczak, 2024). We consider models with continuous latent variables, i.e., Z C IR". Exam- ples of this model class include probabilistic PCA (Tip- ping & Bishop, 1999), variational autoencoders (Kingma & Welling, 2013; Rezende et al., 2014), normalizing flows (Tabak & Ganden-Eijnden 2010; Lipman et al., 2022), dif- fusion models (Ho et al., 2020) and more.
where F and Pz are the sets of possible generator functions and distributions on the latent space, respectively. Designing deep latent variable model means specifying JF and Pz.
Identifiability concerns the uniqueness of parametrizations. We say that two parameters 0 and O' are equivalent, 0 ~ O', if the resulting distributions Po and Po are the same. The induced equivalence class is denoted [0] = {0' : Pa Po Informally, this class captures the different ways in which specific density can be parameterized. Following Xi & Bloem- Reddy (2023), we say that a model is strongly iden- rifiable if [0] is singleton, i.e. the model parametrization is unique, while model is weakly identifiable if it can be identified up to the equivalence class [0].
Formally, define model as tuple of random variables (2, X) where the latent Z drives the observations X through a measurable generator function f Z + D. often called the decoder, and a noise mechanism h : D x D + 'D that makes the relationship stochastic through a noise term e,
L' ~ Pz. P x'= ((Z))(
(L)
As an example, in probabilistic PCA (Tipping & Bishop, 1999), the latent variables can only be identified up to an unknown rotation due to the rotational symmetry of the Gaussian distribution. We then write the equivalence class as [0] (R0), where R is any rotation matrix (Fig. 2, top).
where Zt and e are assumed independent. We further adopt a standard regularity assumption that h and P. are such that e" d ch, h(f(Z"),,") - h(f(Z") e²) if and only if f(29) _ .(2²). Here _ denotes equality in distribution and the assumption ensures that the noise e does not interfere with the causal relationship between X and Z.
Indeterminacy transformations provide means to char- acterizing the equivalence class of a latent variable model M (Xi & Bloem-Reddy, 2023). Given two parametriza- tions of a model 0,, = (Sa, Pz.) and 0, = -fo, Pz,) with resulting marginal distributions Ps. =Po.. . an indetermi- nacy transformation
Statistical model arises when we learn the parameters of the generative model given realizations x of X. Learn- ing the generative model means estimating its parameters 0 (f, P2). which represent the decoder and the latent distribution, respectively. These give rise to the marginal distribution of the data Po that quantifies model fit. For- mally, we define a model M as
at (0., 0b) is measurable function Aa,s Z Z such that Po. Po, and fa A ft; c.f. bottom panel of Fig. 2. Xi & Reddy (2023) prove that the set of all indeterminacy transformations, denoted A(M), fully determines the equivalence class [0). This result establishes the equivalence between parameter iden- tifiability and indeterminacy transformations of the latent
M (F, P2) = (Po on D 0
(2)
2

---

Identifying Metric Structures of Deep Latent Variable Models
space and their associated decoders.
such as M-flows (Brehmer & Cranmer, 2020), normalizing flows, and diffusion models. VAEs need not satisfy A2. On the other hand, A3 can be empirically validated after model training (Shao et al., 2018), and experiments (Sec. 5) show that our methodology is effective in this setting.
Identifiable task captures latent computations with identifi- able outcomes (Xi & Kloem-Reddyy 2023). Here, task is defined by first selecting latent points 2, = s(0, Xm) E Z, and secondly by evaluating the task 1(0, Xm, 2.). The se- lection mechanism can e.g. be the inverse decoder, while a
3.2. Identifiability of distances
task could be independence testing causal discovery or
measuring the distance between latent representations.
In this paper, we shift focus from identifiability of latent rep- resentations (or equivalently, model parameters) and instead identify the relations between them. As our main focus, we seek to establish a distance measure that is invariant under the indeterminacy transformations ((M) of deep latent variable model and therefore identifiable.
Following Proposition 3.1 from Xi & Bloem-Reddy (2023), we can state the sufficient condition for the identifiability of a task in terms of indeterminacy transformations.
Definition 2.1. A task (s, t) is identifiable up to [0] if, for each A E (M) and Xm E 'D with Zn E Z:
deep latent variable model (F, Pz) and (M) its set of indeterminacy transfor- mations. We want to identify latent distances, i.e. find a 'meaningful' distance function : x Z + R.. such that given a parametrization 0, for any 21, 22 E 2 and A E (M) the following is staisfied:
Problem 1. Consider
t(0, Xm: Z,,) = t(A0, Xm Alza)) and s( (AO, xm. = A ((, x)));
(3)
where with la 0b E [0], we have Ab,- = 0, = (fa A-1, A4 Pz.) = (Fo, Pz,) and Ag Pz,, denotes the push- forward of the probability measure Pz..
d(z1, 22) d( (((z)). A(z2))
(4)
3. Problem statement
The inclusion of 'meaningful" in the problem definition emphasizes that solutions can be constructed that satisfy Eq. 4 without being of particular value, e.g. the trivial metric
We address the challenge of making pairwise distances sta- tistically identifiable in modern deep generative models without impractical assumptions. Next, we outline our as- sumptions and formalize the objective, which we solve in Sec. 4 and we show that our approach ensures not only identifiable distances, but also a broader set of identifiable metric structures.
d(z), x2) I(z1 # z2)
(5)
is identifiable, but reveals little about latent similarities. Instead, we want the distance to reflect and respect the underlying mechanisms behind the observed data.
4. Main results
3.1. Assumptions on F and
Strategy and results at a glance. In the following, we show that distances, angles, volumes, and more, can be identified in latent variable models that satisfy the weak assumptions in the previous section. Our proof strategy is to connect indeterminacy transformations from the identifi- ability literature with charly from the differential geometry literature. Once this connection is in place, our results easily follow. Furthermore, we use the connection to show that identifying Euclidean distances in the latent space is either impossible or requires foreing the decoder to have zero cur- vature. Below we present results with proof sketches and leave details to Appendices A and B.
Following typical literature (Xi & Bloem-Reddy, 2023; Shao et al., 2018), we further impose assumptions on the space of our decoder functions F and the latent space Z.
We consider decoders that are smooth functions f 2 - D such that for each f E F:
A1 2 is compact
A2 f is injective
A3 The differential of f, df, has full column rank
A4 All f E F have the same image. That is, for any far fo F. we have fz(z) fo(Z) : MCD
Assumptions A2-A4 are repeated from Xi & Mloem-Reddy: whereas we add assumption Al and require f be smooth. Together, these allow to treat the image of the decoder as smooth manifold. Assumption A1 is purely technical and can be interpreted as (after model training) we consider a compact subset of the latent space, e.g. the range of float- ing point numbers. Jointly, the assumptions may appear restrictive, but they are satisfied by contemporary models
4.1. Identifiability via geometry
We begin by focusing on the family of decoders FF and analyzing the properties of their image.
Lemma 4.1. Let Z and 'D be two smooth manifolds and f E F. then f is a smooth embedding and f(Z) C D is a submanifold in D. In particular, f : - f(Z) is a diffeomorphism.
3

---

Identifying Metric Structures of Deep Latent Variable Models
Latent spaces
fa Observation space
From the perspective of differential geometry, if we are to collect all the coordinate charts of the form (f-1,M stemming from the indeterminacy transformations in A (M) into smooth atlas, then Lemma 4.3 tells us that the gener- ator transforms play the of role of smooth transition maps between them, as illustrated in Fig. 3.
Ant
2g
M
fs
Lemma 4.1 tells us that the image of our decoder functions is smooth manifold. Due to the 'Existence of Riemannian Metries' result by Lee (2003), it admits a Riemannian metric g that for each point on the manifold defines an inner product in its tangent space at denoted by T..M. The tuple (M,g) g) defines the Riemannian manifold structure that allows measurements on general smooth manifolds and is the theoretical foundation for our methodology.
Figure 3: Decoders Sa and fo parametrize the same manifold M D when Pg = (fa, Pz.) and 0, (fb, P2.) give the same marginal distributions Po. Pa..
Proof sketch. Smoothness and assumption A3 lead to f E F being a smooth map of constant rank (smooth immersion) while assumptions A1 and A2 make sure that the image of does not self-intersect. Given these properties. the claim follows from standard results in differential geometry.
For general data manifolds, there is neither a unique nor a known metric g. However, given decoder function f and chosen metric (often Euclidean) g" in the ambient space D, we can construct the pullback metric gf in the latent space Z by pulling the ambient metric back to the latent space using the decoder.
One consequence of Lemma 4.1 is that given two trained models 02 and O, with equivalent marginal distributions Po. = Po., the resulting decoder functions fa. fu act as reparametrizations of the same manifold fz( fz) = M. In particular the tuples (fa²). .M) and (fr .M) can be seen as coordinate charts of the manifold M. This situa- tion is the main subject of our analysis and is illustrated in Figure 3. In what follows, we will label the latent spaces by the associated models such that for 6g we will have Z, and similarly for 0g.
Detinition 4.4. Let 2 be smooth manifold and (D,g") be Riemannian manifold. Furthermore, let f 2 - M C D be map satisfying assumptions A1-A3, the pullback metric f g" on 2 is defined as:
() = = (d ()'
(7)
We will go between the different charts (or latent spaces) by using generator transformations in Definition 4.2 that push and pull along the respective decoders.
for any tangent vectors u, u E T, In Eq. 7, 7. means that we use ambient metric evaluated in the tangent space T MM The notation (ff means that the differential map of f at E Z maps the vector u E TpZ to f(u) e M. We will denote the pullback metric as g¹ = f" for shorter notation and let the domain of be implicit from the definition of f.
Definition 4.2. Given two equivalent parametrizations 0g- (Ja, P2.) and 0g- (So, Pz.) of a model with Po. Po..
we define the generator transformation Au,: Zn > 2, is
(z, = =f o fa(z), for z E Za.
(6)
The result of this important construction is that:
Lemma 2.1 in Xi & Reddy (2023) shows that any indeterminacy transformation A E (M) must be almost everywhere equal to the generator transformation. Whereas Xi & Bloem- focus on proving this result and us- ing it for characterizing identifiability issues in general, we use this construction to show that it preserves the geomet- ric properties of the manifold and, in particular, that the geodesic distance function (formally defined in Eq. 9) is invariant w.r.t to the entire set A(M).
it allows us to construct a Riemannian metric on M that respect the intrinsic properties of the Riemannian manifold (M.g). g). In this setting, the pullback met- ric gf represents some intrinsic g in the coordinates defined by Z and f.
we can make all the measurements from the latent space Z using of as this construction makes the Ric- mannian manifolds (Z, ( and (M, g) the same, from a geometric perspective. Thus, we can concentrate our attention on (Z,g'), while being consistent with (M,g) without worrying about
Lemma 4.3. Given two equivalent parametrizations fg (Ja, P2..) and Op- = (Jo, Pz.) of a model with Po. = Pow, the generator transformations Az,(Zz) and A A = Ab,a (z) are diffeomorphisms.
The pullback metric merely measures the length of a latent curve by first decoding the curve and measuring its length according to the data space metric. This is quite 'meaning- ful' metric in line with the requirements of Problem 1.
Proof sketch. The result follows from the fact that a compo- sition of diffeomorphisms is a diffeomorphism. The genera- tor construction only possible because of assumption A4.
4

---

To solve Problem I we use the geodesic distance from Defi- nition 4.6 and show that it is identifiable in Theorem 4.7. Definition 4.6. Let (Za: gf=) ) be Riemannian manifold, then for 21, Z2 E Z we define the geodesic distance function Identifying Metric Structures of Deep Latent Variable Models
fa
D
Theorem 4.7. Lei 02 (Za,Pz.) and 0, = (Sor Pzo) with Po. aa,and and let Aa,b be the generator transforr between the parameters. Furthermore, let (Za.gi=) and (Zo, gh)) be the associated Riemannian manifolds. Then, the geodesic distance between 21 and 22 is identifiable and
2g
9
M
f6
Agta (Z1,Z2) dgts (Aa,b(Z1). Aa,)(Z)))
(10)
Figure 4: Pullback metries assign a local inner product to latent spaces corresponding to measuring along the mani- fold spanned by the decoder. In the left panels, the white ellipsis corresponds to unit circles under the pullback metrie, corresponding to a local Euclidean metric in the observation space D. Geodesics (yellow curves) minimize length ac- cording to the pullback metric, corresponding to minimizing the length of the decoded curve along the manifold.
for some 21, 22 Za be two points in the latent space that correspond to some X1, X2 E M on the manifold.
Proof sketch. First formulate the task of computing the geodesic distance in terms of Definition 2.1, then check the definition for the selection function by plugging in and for the task output by leveraging Theorem 4.5.
4.2. Identifiability of Euclidean distances
In the framework of pullback metrics defined by differ- ent decoders that span the same manifold, the generator transformations that comprise the space of indeterminacy transformations isometries that preserve angles, length of curves, surface areas, and volumes on the manifold.
Theorem 4.7 represents solution to Problem 1 and came as a result of treating the question of identifiability from the geometric perspective. While Section outlines alternative approaches to the same problem, in the following we show that these must necessarily impose implicit constraint of flatness on the models.
Theorem 4.5. Let 0g = (fa; Pz..) and 0g = (fb, Pz.) with Pv. Po,and and let gf) and (Zt, gyو) be the associated Riemannian manifolds, then the generator transform is isometry and it holds that:
= R be the latent space and (2,g') the associated Riemannian manifold. Furthermore, let gFo denote a metric tensor that is proportional to the Euclidean metric tensor g", then:
Proposition 4.8. Let =
(Au,)) go =g
P1 If we choose gEv as our metric in the latent space, that equivalent to assuming g gh, then (v) can only be identifiable if the associated parametrize flat manifold M within the ambient space D i.e. M has zero curvature.
Thus, making (2a, g'0) and (2,,g isometric. This makes Riemannian geometric properties such as lengths of curves, angles, volumes, areas, Ricci curvature tensor, geodesics, parallel transport, and the exponential map identifiable.
F
Proof sketch. First, we show that Eq. is satisfied, es- tablishing the isometry property, then we refer to results
P2 If we choose the Euclidean distance to be identifiable, equivalent to assuming gf =g", then PI applies and the associated f E F are such that the generator trans- forms are isometries of R", i.e. translations, rotations, or reflections.
in Riemannian geometry to establish the isometric invari- ance of partieular property. To obtain identifiability, each property is expressed in terms of a task from Section 2 and Definition 2.1 is shown to be satisfied.
Proof. Distance measures proportional the Euclidean distance measure are characterized by the pullback metric g' being constant everywhere. If g' is constant everywhere, its directional derivatives vanish and the curvature is zero. The second point follows from Theorem 4.5 and standard linear algebra, c.g. Friedberg et al. (2014).
dyt (Z1, 22) inf S' (1((t)) dt
7.
4.3. Main takeaways
where y: :0,TT] + Za is a latent curve from Z1 to Zz.
Our results can be summarized by the following takeaways:
Figure 4 illustrates how the geodesic distance measures the length of the shortest curve (geodesic) under the pullback metric. This is equivalent to finding the shortest curve along the manifold spanned by a decoder.
The Riemannian metric space of a deep latent variable model identifiable (Theorem 4.7) making distance measurements the latent space identifiable. This solves Problem 1.
5

---

Identifying Metric Structures of Deep Latent Variable Models
Riemannian geometry properties of the learned mani- fold are identifiable (Theorem 4.5). Examples beyond distances include angles, volumes, and more. Jointly these provide a rich language for probabilistic data analysis in the latent space.
as (V(d,,p))) = o(d. (x.))/ (x.))) with o and p denoting standard deviation and the mean across 30 seeds, and d, is placeholder for the distance measure used (dg or dg). The coefficient of variation is a unitless measure of variability, where low values indicate less variability. We use this to compare the variability of Euclidean and geodesic distances.
Using Euclidean distances in the latent space is either not identifiable or must come at the cost of imposing flatness constraints on the model (Proposition 4.8).
We consider two models and four datasets. First, a model that satisfies all our assumptions, and second, a model where we disregard the injectivity assumption. To compute geodesics we parametrize them by spline connecting two points in the latent space and minimize its energy. The discussion around Definition A,23 in Appendix A covers how this leads to a geodesic. has been noted that taking decoder uncertainty into account key to good performance (Arvanitidis et al., 2018; Hauberg, 2018) and we follow the ensemble- based approach from Syrota et al. (2024), implying that we train an ensemble of 8 decoders. Details on computing geodesics and experimental details are in Appendices C and D. The code to reproduce our results is available in the project repository GitHub'.
Any task whose identifiability boils down to the identi- fiability of the Riemannian metric is identifiable if the properties of the manifold allow.
To exemplify the last point, consider the Fréchet mean that generalizes the well-known mean to manifolds (Pennec, 2006). This is obtained by finding the point with minimal average squared distance to the data.
jiFrcchet
argmin 2
d2 glo (2), 2i)
(11)
As the mean is defined as an optimization problem, there might exist multiple means, which violates the usual notion of identifiability. First, it is worth noting that the solution set is identifiable, although, computes a single optimum. In some situations, this single- ton can, however, be identified based on properties of the manifold M. Karcher (1977) and Kendall (1990) provide uniqueness conditions that connect the radius of the smallest geodesic ball containing the data with the maximal curvature of the manifold. Importantly, these are, principally, testable conditions such that it should be feasible to computationally test if a computed mean is identifiable. Identifiability of some statistical quantities is, thus, within reach.
MNIST and CIFAR10 with A A1-44 satisfied. We use M- flows (Brehmer & Cranmer, 2020) to construct model with an injective decoder. We train this model on a 3-class subset of MNIST (Deng, 2012) with 2D latent space for visualization purposes and full CIFAR 10 (Krizhevsky et al.). An example of a geodesic curve from digit 7 to digit O from the test set is visualized in Fig. 5, The geodesic crosses class boundaries where they are well-explored by the model and offer little uncertainty.
practice, one usually only
The left side of Fig. 7 shows histograms of the coefficient of variation for the 100 point-pairs, where we see a narrower distribution with both lower mean and spread for geodesic distances. We perform a one-sided Student's f-test for the null hypothesis that geodesic distances vary less than the Eu- clidean (Table 1) and find strong evidence for the hypothesis. This demonstrates that identifiability improves reliability.
5. Experiments
Identifiability is a theoretical concept studied in the asymp- totic regime of infinite data. Our experiments aim to demon- strate that this asymptotic property is practically exploitable in standard models using off-the-shelf methods.
FMNIST and CELEBA with A2 relaxed and A3 veri- fied. The general VAE model is known to be effective due to its flexible decoder parametrized by a neural net- work with arbitrary architecture that is not guaranteed to be globally injective (A2) nor to have full rank Jacobian (A3).
Theorem 4.7 proves that geodesic distances are identifiable, while Euclidean ones are not. This suggests that geodesic distances should be more stable under model retraining than the Euclidean counterpart. To test this hypothesis, we train 30 models with different initial seeds and compute both Euclidean (d)) and geodesic (dg) distances in the latent space between 100 randomly chosen unique point pairs ((p: = (x,, xx) 100
5
7
Figure 5: An example latent geodesic from a M-flow model trained on three classes from MNIST. The back- ground color indicates model uncertainty.
Xz, X2 .,, j # k) from the test set. We emphasize that the pairs are the same across all models, allowing us to measure the variances of the dis- tances across models.
To assess the stability of the distance measures, we compute the coefficient of variation for each pair, which is evaluated
6

---

Identifying Metric Structures of Deep Latent Variable Models
Figure 6: Geodesic (top) and Euclidean (bottom) interpo- lations, highly similar, but dis- tances still differ significantly (Fig, 7).
MNIST
FMNIST
CIFAR10
CELEBA
that characterize when, and what types of non idd-iifiiibbiitty are acceptable in deployments of deep generative models without significant constraints.
i-statistic -8.64
16.75
-42.83
-22.33
p-value
1.00
1.00
1.00
1.00
Latent space geometries have been studied in various con- texts to define more 'meaningful' latent interpolations and distances (Tosi et al., 2014; Arvanitidis et al., 2018; Beik- Mohammadi et al., 2021). While Hauberg (2018) alludes to connections between latent space geometries and iden- tifiability, no formal statement has previously been made. Our work, thus, brings further mathematical justification the algorithmic tools developed for latent space geometries. Detlefsen et al. (2022) has previously demonstrated that la- tent space geometries recover evolutionary structures from models of proteins that are invisible under an Fuclidean latent geometry. Our work adds credibility to these findings, which can now seen as identifiable.
Table 1: One-sided Student's t-test for the variability of geodesic versus Euclidean distances
We train this model the FMNIST (Xiao et al., 2017) and CELEBA (Liu et al., 2015) datasets where the architecture is composed of convolutional and dense layers with ELU activation functions. The latent space dimension is 64 and we further employ Resize-Conv layers (Odena et al., 2016) to improve image quality. We follow the approach from Shao et al. (2018) to validate that the decoder Jacobian is, indeed, always full rank. An example geodesic shown in Fig. 6 alongside Euclidean counterpart, where do not observe a significant difference between generated images.
Causality strongly relies on identifiability as there is lit- tle point in recovering the 'true' causal model if it is not guaranteed to be unique. The goal of causal representa- tion learning (Schölkopf et al., 2021) is closely linked to our quest for identifiable representations, or, at least, re- lationships between such. Our approach, however, is not immediately applicable to many questions of causality as these often amount to establishing independence between variables (Peters et al., 2017). Under the geometric lens, we do not have canonical coordinate system in the la- tent space, which complicates splitting the latent space into factors to be considered independent.
Figs. 7b and 7d (right side) show that the coefficient of variation for geodesic distances has both lower mean and standard deviation than Euclidean distances. The one-sided Student's i-test again validates this observation (Table I). This demonstrates that geodesic distances remain more reli- able than Euclidean ones even when the injectivity assump- tion may be violated.
6. Relations to existing work
We connect questions of identifiability with results from differential geomelry. To knowledge, no previous stud- ies have formally connected these otherwise disjoint fields. Our work is, however, linked to a large body of prior works.
Transformations of the latent space have proven valuable in various areas, including disentangled and equivariant learning, highlighting their mathematical and conceptual connections. Higgins al. (2018); Wang et al. (2021); Zhu et al. (2021) assume disentangled latent space where transformations decompose into individual factors of varia- tion and the goal is to find representations that respect this structure. This can be interpreted as equivariance of the generator transformations Aa,b (Definition 4.2). Instead of enforcing specific properties of As,b, our theory analyzes the natural properties of the generator transformations, and we find that just by learning generative model, Ao,b will automatically respect the latent Riemannian geometry (The- orem 4.5). Thus, our theory is valid regardless of whether a
Identifiability is well studied in the ICA or source separa- tion literature (Comon, 1994; Hyvärinen & Pajunen, 1999). The analysis of identifiability in deep generative models stems from a connection between VAEs and ICA first no- ticed by Khemakhem et al. (2020). Many works either constraints, focus on formulating adentifiability-ehhanee typically placed on the decoder or latent distribution, or obtaining data from more diverse sources, c.g., multiple environments or multiple views (Kivva et al., 2022; Hyvari- nen et al., 2019; Gresele et al., 2019; Locatello et al., 2020; 2019b; Shu et al., 2019). On the other hand, few works exist

---

Identifying Metric Structures of Deep Latent Variable Models
0.30
Gootel: Dacidca
Gooted::
0.90
Dacidice
0.35
0.95
0.20
0.00
0.15
0.15
6.10
0.00
0.00
0.00
0.00
0.00
0.10
0.15
0.90
0.95
0.30
0.35
0.05
0.04
0.06
0.06
0.10
0.19
Coeflicient of
Coefficient variation
(a) M-flow model MNIST
(b) VAE FMNIST
0.95
Gookel:
0.15
Gooke::
0.90
0.10
0.15
0.10
0.04
0.01
0.02
0.03
0.04
0.05
0.000
0,010
0.016
0,025
0.050
Coefficient of variation
Coefficient of varistion
(c) M-floww model on CIFARIO
(d) VAE on CELEBA
Figure 7: Histograms of coefficients of variation for Euclidean and geodesic distances on MNIST (7a), FMNIST (7b). CIFAR (7c) and CELEBA (7d). Geodesic distances vary significantly less, which is quantified in Table 1.
disentangled latent space exists in the sense of Higgins et al. (2018).
7. Weaknesses and open questions
Our work provides strong identifiability guarantees for es- sential quantities such pairwise distances in contemporary generative models. However, our approach is not problem- free and we highlight some pitfalls to be aware of.
Disentanglement can be seen as a 'poor man's 's causality' (Detlefsen Hauberg, 2019), where the key generating fac- tors are sought to be axis-aligned in the latent space. This is generally known to be mathematically impossible (Locatello et al., 2019a), much in line with proposition 4.8. Similar to identifiability, this difficulty has been addressed by induc- tive biases (Bouchacourt et al., 2018) or (weak) data labels (Locatello et al., 2020; 2019b; Shu et al., 2019). Empirical studies, however, hint that some key factors can be recov- ered in practice (Higgins et al., 2016; Dittadi et al., 2020; Suter et al., 2019). Rolinek et al. (2019) noted that standard disentanglement pipelines only work when the variational distribution, parametrized by the encoder, is restricted to diagonal covariance. This suggests that current results in disentanglement may be artifacts of a poor Bayesian approx- imation. Our work suggests that disentanglement is perhaps better achieved by looking for "geometric factors', such as principal geodesics (Fletcher et al., 2004).
Observation metries matter. Our identifiable geometric structure relies on the idea of locally bringing the obser- vation space metric into the latent space. This has many benefits but also raises the question of choosing the observa- tion space metric. This choice will directly impact the final latent distances. Did we then replace one difficult problem (identifiability) with another (choosing observation metric)? We argue that most data is equipped with units of measure- ment, which greatly simplifies the task of picking a suitable metric in the observation space. Furthermore, being explicit about how data is compared improves the transparency of the conducted data analysis. Finally, emphasize that we are not proposing to bring the Euclidean distance from observation space into the latent space, but only to do so infinitesimally, i.e. measuring along the manifold.
Relative representations are explicitly constructed from the original representations and set of anchor points to be invariant under angle-preeerving transformations (using cosine similarity) (Moschella et al., 2022) or isometries (us- ing a proxy for geodesic distances) Yu et al. (2024). Our approach instead provides mathematical guarantees on in- variance under isometries of the original representations and establishes the link to statistical identifiability.
Identifiability comes at a (computational) cost. Eu- clidean distances are cheap to compute, unlike the geodesic distances consider. For any geodesic distance, must solve an iterative optimization problem. Fortunately, this is locally convex problem, that only requires estimating a limited number of parameters, so the computation is feasi- ble. Yet remains significantly more costly than computing
8

---

Identifying Metric Structures of Deep Latent Variable Models
Euclidean distance. We argue that when identifiability is important, e-g. in scientific knowledge discovery and hy- pothesis generation, the additional computational resources are well-spent. Proposition 4.8 effectively tells us that we must choose between (cheap) flat decoders or (expensive) curved ones: we cannot get the best of both worlds.
as many tools readily exist for statistical computations on manifolds. For example, Riemannian counterparts to aver- ages (Karcher, 1977), covariances (Pennec, 2006), principal components (Fletcher et al., 2004), Kalman filters (Hauberg et al., 2013), and much more readily exist. In principle, it is also possible to devise computational tests to determine if these statistics are identifiable for given model and dataset. This is, however, future work.
Compression matters. Our current work rests on the assumption that the latent space has dimension that is less than equal to the data space dimension, i.e. dim(2) 2) 1 dDm(D). This remains the standard setting for representa- tion learning and generative models. However, if we, for the sake of argument, wanted to identify distances between the weights of overparametrized neural networks, then our strategies would not directly apply. Early work has begun to appear on understanding the geometric structure of over- parametrized models (Roy et al., 2024), which suggests that perhaps our approach can be adapted.
Impact statement
This paper improves our collective understanding of which aspects of statistical model can be identified. As the the- oretical understanding translates directly into algorithmic tools, our work has an impact potential beyond the theoreti- cal questions. Being able identify relationships between latent representations of data can aid in the process of sci- entific discovery as we increase the reliability of the data analysis. Our work can also help provide robustness to in- terpretations of neural networks and other statistical models, which may help in explainability efforts.
Injectivity remains an issue. Most of our assumptions are purely technical and easily satisfied in practice. The key exception is Assumption A2 stating that the decoder must be injective. The decoders of contemporary mod- els such as diffusion models and (continuous) normalizing flows (including those trained with flow matching) are in- jective and the assumption is satisfied. However, general neural networks cannot be expected to be injective, such that variational autoencoders and similar models are not identifiable "out of the box'. We have demonstrated that (injective) M-flow architectures can be used for such mod- els, and empirically we observe that the geodesic distance increases robustness in non-injective models. This gives hope that theoretical statements can be made without the injectivity assumption. One such path forward may be to consider notions of weak injectivity (Kivva et al., 2022), which is less restrictive and more easily satisfied in practice.
A Acknowledgments
This work was supported by research grant (42062) from VILLUM FONDEN. This project received funding from the European Research Council (ERC) under the European Union's Horizon research and innovation pro- gramme (grant agreement 101125993). The work was partly funded by the Novo Nordisk Foundation through the Center for Basic Machine Learning Research in Life Science (NNF200C0062606), JX is supported by Natu- ral Sciences and Engineering Research Council of Canada (NSERC) Canada Graduate Scholarship. BBR acknowl- edges the support of NSERC: RGPIN2020-04999, RGPAS- 2020-00095.
8. Conclusion
References
In this paper, show that latent distances, and similar quantities, can be statistically identified in a large class of generative models that includes contemporary models with- out imposing unrealistic assumptions. This is a significant improvement over existing work that tends to impose ad- ditional restrictions on either model or training data. Our results are significant when seeking to understand the mech- anisms that drive the true data- entta-eenrratig process, e.g. in scientific discovery, where reliability is essential.
Arvanitidis, G., Hansen, L. K., and Hauberg, S. Latent space oddity: on the curvature of deep generative models. In International Conference on Learning Representations (ICLR), 2018
Beik-Mohammadi, H., Hauberg, S., Arvanitidis, G., Neu- mann, G., and Rozo, L. Learning riemannian manifolds for geodesic motion skills. In Robotics: Science and Systems (RSS), 2021.
Practically, it is important to note that our strategy requires no changes to how models are trained. Our constructions are entirely post hoc, making them broadly applicable.
Bouchacourt, D.,, Tomioka, R., and Nowozin, S. Multi-level variational autoencoder: Learning disentangled represen- tations from grouped observations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Our proof strategy relies on linking identifiability with Rie- mannian geometry; link that does not appear to have for- mally been made elsewhere. This link paves way forward

---

Identifying Metric Structures of Deep Latent Variable Models
Brehmer, J. and Cranmer, K. Flows for simultaneous mani- fold learning and density estimation, 2020.
Hauberg, S. Only bayes should learn a manifold. 2018.
Hauberg, S. Differential geometry for generative mod- eling, 2024. URL https:: /www:////www.. compute. atu.dk/-sonn /wwekenndwwhheeriiee ieaerenntia__gemmerttyye gerrrrggeeeeiiieeee modeling.pdk
Chari, T. and Pachter, L. The specious of single-cell ge- nomics. PLOOS Computational Biology, 191 2023.
Comon, P.
Independent component analy- sis, concept? Signal Processing, 36
Hauberg, S., Lauze, F,, and Pedersen, K. S. Unscented
(3):287-314, 1994.
ISSN 0165-1684.
doi:
kalman filtering on riemannian manifolds. Journal of Mathematical Imaging and Vision, 46(1):103-120, May 2013.
URL https: https://ww..eccieeedieeee com/
Higher Order Statistics.
He, K. Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition, 2015. URL https:// arxiv. www.worr//aaoooo abs 1112.1.. 03385.
Deng, L.
The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-144) 2012.
Higgins, Matthey, L., Pal. A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta- vae: Learning basic visual concepts with constrained variational framework. 2016.
Detlefsen, N. S. and Hauberg, S. Explicit disentangle- ment of appearance and perspective in generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Higgins, I., Amos, D., Pfau, D., Racaniere, S., Matthey, L., Rezende, D., and Lerchner, A. Towards definition of disentangled representations, 2018. URL https:
Detlefsen, N. S., Hauberg, S., and Boomsma, Learning meaningful representations protein sequences. Nature Communicationns 13(1), April 2022. ISSN 2041-1723. doi: 10. 10.1103//44467-00000 -22--7743--2922- URL http://dx. doi www.err//.... 1038/
/ wwtp:/iwwwwg.a////o wttp://///////11.022200.00
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Ding, X., Zou, Z., and Brooks III, C. L. Deciphering protein evolution and fitness landscapes with latent space models. Nature communications, 10(1):6644, 2019.
Hyvarinen, A., Sasaki, H., and Turner, R. E. Nonlinear ica using auxiliary variables and generalized contrastive learning, 2019. URL https: httth:/:///aaaiii https////xxxv..o//aaabp 1805.08633
Dinh, L., Krueger, D., and Bengio, Y. Nice: Non-linear independent components estimation, 2015.
Hyvärinen, A. and Pajunen, P. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 12(3):429-439, 1999. ISSN 0893-6080. doi: http:///o.oog////oo/6666993////0 URL httpp::/www..ccleeeddreee. science/rrtaaaeepp/////000000000
Dittadi, A., Träuble, F., Locatello, F., Wüthrich, M., Agrawal, V., Winther, O., Bauer, S., and Schölkopf, B. On the transfer of disentangled representations in realistic settings. arXiv preprint arXiv:2010.14400 2020.
Durkan, C., Bekasov, A., Murray, I,, and Papamakarios, G. Neural spline flows, 2019.
Johnson, J.. Alahi, A., and Fei-Fei, L. Perceptual losses for
Fletcher, P. T.,, Lu, C.,, Pizer, S. M., and Joshi, S. Principal geodesic analysis for the study of nonlinear statisties of shape. IEEE transactions on medical imaging, 23(8): 995-1005, 2004.
real-time style transfer and super-resolution, 2016. URL
https: / //aa//ivvv.///////////./////aata org. wwtp/r/gg//aa//oeeeeee /ow...//1633... 08155.
Karcher, H.
Riemannian center of mass and mollifier smoothing. Communications pure and applied mathe- matics, 30(5):509-541, 1977.
Friedberg, S., Insel, A., and Spence. L. Linear Alge- bra. Pearson Education, 2014. ISBN 9780321998897.
URL https: /bw.koooksooooooooooooooo.kkkk.. googl 1 e dk books?id- KYBODAAAQBAJ.
Kendall,
Probability, convexity, and harmonic maps with small image i: uniqueness and fine existence. Pro- ceedings of the London Mathematical Sociery, 3(2):371- 406, 1990.
Gresele,
L., Rubenstein, P. K., Mchrjou, A., Locatello, F,, and Schölkopf, B. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica, 2019. URL https:t////./...... /ttp://axiii orgg abs 900.005000 06642.
Khemakhem, I., Kingma, D. P., Monti, R. P., and Hyvärinen, Variational autoencoders and nonlinear ica: A unifying
10

---

Identifying Metric Structures of Deep Latent Variable Models
framework, 2020. URLhttts://arxiv..r
Locatello, F,, Tschannen, M., Bauer, S., Rätsch, G., Schölkopf, B., and Bachem, O. Disentangling fac- tors of variation using few labels. arXiv preprint arXiv:195..012 2019b.
1907.044 9.
Kingma, D. P. and Ba, J. Adam: A method for stochastic op- timization, 2017. URL https:: /ate:i///aavv..// org/ abs/ 1412. 6980.
Locatello, F,, Poole, B., Raetsch, G., Schölkopf, B., Bachem, O., and Tschannen, M. Weakly-supervised disentangle- III, H. and Singh, A. (eds.), Proceedings of the 37th International Confer- ence Machine Learning, volume 119 of Proceedings Machine Learning Research, pp. 6348 6359. PMLR. 13-18 Jul 2020. URLhttps::/prooct press v11 https//111111cccatoee htt.1//oooaaaalll.a.aaooo html.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint 2.Xi::3312.6114,2011 2013.
ment without compromises.
Kivva, B., Rajendran, G., Ravikumar, P., and Aragam, B. Identifiability of deep generative models without auxil- iary information, 2022. URL http::::///xxx..orrrr.. abs/2205 110...0044 10044.
ngs mmlr.
Klema, V. and Laub, A. The singular value decomposition: Its computation and some applications. IEEE Transac- tions on Automatic Control, 25(2);164-176, 1980. doi: 10.1 1991 190344
Moschella, L.,, Maiorca, V., Fumero, M., Norelli, A., Lo- catello, F,, and Rodolå, E. Relative representations enable zero-shot latent space communication. arXiv preprint arXh::2209 2022.
Kobak, D. and Berens, P. cell transcriptomics. Nature Communications, 2019.
The art of using t-sne for single-
Odena, A., Dumoulin, V., and Olah, C. Deconvolution and checkerboard artifacts. Distill, 2016. doi: 10.23915/ distill.00003. URL http://ddisiillputt 20111 deconv-checkerboan
Kress, R. Numerical Analysis. Graduate Texts in Mathe- matics. Springer New York, 2012. ISBN 9781461 205999.
URL https:// /bbokss....oooo/oo google ak/bookski. Jv Jv_īBwAAQBAJ.
O'Neill, B. Elementary Differential Geomelry. Academic Press, 1997. ISBN 9780125267458, URL https://
Krizhevsky, A., Nair, V., and Hinton, G. Cifar-10 (canadian institute for advanced research). URL http:// www. cs. toronto... edu/ aedu/-kriz/ /riz/rii ci far. html.
books . google.ooooooooooooooooooooo.. wk/books http://ookk.....aaaa//aaaaaaaaaa
Pennec, X. Intrinsic statistics riemannian manifolds: Ba- sic tools for geometric measurements. Journal of Mathe- matical Imaging and Vision, 25:127-154, 2006.
Lause, J., Berens, P., and Kobak, D. The art of seeing the elephant in the room: 2d embeddings of single-cell data do make sense. bioRxiv, 2024.
Peters, J., Janzing, D., and Schölkopf, B. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Lee, J. Introduction to Smooth Manifolds. Gradu- ate Texts Mathematics. Springer, 2003. ISBN 9780387954486. URL https: /bboos..... google. ak/bookssld=eqffg2 jOceYC. jOceC...
Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep gen- erative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint 2xXi:2210.027747 2022.
Riesselman, A. J., Ingraham, J. B., and Marks, D. S. Deep generative models of genetic variation capture the effects of mutations. Nature methods, 15(10):816-87) 2018.
Liu, 7.,, Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Confer- ence on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4124. PMLR, 09
Rolinek, M., Zietlow, D.,, and Martius, G. Variational au- toencoders recover pea directions (by accident). In Pro- ceedings IEEE Conf. Computer Vision and Pattern Recognition, pp. 132, 2019.
Roy, H., Miani, M., Ek, C. H., Hennig, P., Pförtner, M., Tatzel, L., and Hauberg, S. Reparameterization invariance in approximate bayesian inference. In Neural Information Processing Systems (NeurIPS), 2024.
15 Jun 2019a. URL https: httposs://rrcceeeee /ptp/peeenngngss.eeeeeepppp. mlr. press/v99 hwtpaaceleellllloooooo1111 html.

---

Identifying Metric Structures of Deep Latent Variable Models
Schoenberg, I. J. Contributions to the problem of approximation of equidistant data by analytic fune- tions. part b. on the problem of osculatory interpola- tion. second class of analytic approximation formu- lae. Quarterly of Applied Mathematics, 4:1 4:112-141, 1946. URL https: wwwppp..eaatissc.....ooooooooeoe org/CorpusID: 125667988.
Tipping, M. E. and Bishop, C. M. Probabilistic principal component analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology, 6163),611-62,, 1999.
Tomezak, M. Why deep generative modeling? In Deep Generative Modeling, pp. 1-13. Springer, 2024.
Tosi, A., Hauberg, S., Vellido, A., and Lawrence, N. D. Metries for probabilistic geometries. In The Conference on Uncertainty in Artificial Intelligence (UAI), Quebec, Canada, July 2014.
Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalch- brenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE, 109(5); 612-634, 2021.
Wang. T., Yue, Z., Huang, J., Sun, Q., and Zhang, H. Self- supervised learning disentangled group representation as feature. In Ranzato, M., Beygelzimer, A., Dauphin, Y.. Liang, P., and Wortman Vaughan, J. (eds.), Advances in Neural Information Processing Systems 34 35th Confer- ence on Neural Information Processing Systems, NeurlPS 2021, Advances in Neural Information Processing Sys- tems, pp. systems foundation, 2021. Publisher Copyright: ② 2021 Neural information processing systems foundation. All rights reserved.; 35th Conference on Neural Information Processing Systems, NeurIPS 2021 Conference date: 06- 12-2021 Through 14-12-2021.
Shao, H., Kumar, A., and Fletcher, P. T. The Riemannian Geometry of Deep Generative Models In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 428-4288, Los Alamitos, CA, USA, June 2018. IEEE Computer URL
Society. doi: 10.1109/CVPR.22018.00M https::/do //to:///oiiio// wto...eeemmotteessceeeeey.. 10. 1.0.119//00 2010///VVR..... 2018 00071.
Neural information processing
Shu, R., Chen, Y., Kumar, A., Ermon, S., and Poole, B. Weakly supervised disentanglement with guarantees. arXiv preprint arIiv::9I000. 1910.09772, 2019.
Suter, R., Miladinovic, D., Schölkopf, B., and Bauer, S. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceed- ings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learn- ing Research, pp. 6056-6065, PMLR, 09-15 Jun 2019. URL httpthtps/// htlLhtts::/peeee http:///ooccedeeeee mlr press/v97/ suter19a 9a.119...4 html.
Xi, Q. and Bloem-Reddy, B. Indeterminacy in generative models:
Characterization and strong identifiability, 2023,
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist. a novel image dataset for benchmarking machine learning algorithms, 2017. URL htw.t:tsps//// http///axxvvv//// /rb//bs/ 1708. 07747.
/bs//
Yu, H., Inal, B., and Fumero, M. Connecting neural models latent geometries with relative geodesic representations. UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models, 2024. URL https: / // //openreviee nww. wttp///rrmm/ee///////////o/ www.mmmiiidgyy.mmiie.....m.
Syrota, S., Moreno-Muñoz, P., and Hauberg, S. Decoder ensembling for learned latent geometries. In Vadgama, S., Bekkers, E., Pouplin, A., Kaba, S.-O., Walters, R., Lawrence, H., Emerson, T., Kvinge, H., Tomezak, J., and Jegelka, S. (eds.), Proceedings of the Geometry- grounded Representation Learning and Generative Mod- eling Workshop (GRaM), volume 251 of Proceedings of Machine Learning Research, pp. 277-285. PMLR, 29 Jul 2024. URL htwps:tt///// httptt:p::/pppcooeeee http://ooceeeiin/ ngs mlr .prss// html.
Zhu, X., Xu, C., and Tao, D. Commutative lie group vae for disentanglement learning. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12924-12934, PMLR, 18-24 Jul 2021. URL http:::ppp//oooeeedieeooo .llr. press/v139/ wwtpw/////euei1e zhu21f f..ull............m
Tabak, E. and Vanden-Fijnden, E. Density estimation by dual ascent of the likeliood,,odd Communications in Mathematical Sciences, 8(1):217-233 2010.
Tasic, B., Yao, Z.,, Graybuck, L. T., Smith, K. A., Nguyen, T. N., Bertagnolli, D., Goldy, J., Garren, E., Economo, M. N., Viswanathan, S., et al. Shared and distinct tran- scriptomic cell types across neocortical areas. Nature, 563(7729):22-78,22 2018
12

---

Identifying Metric Structures of Deep Latent Variable Models
A. Riemannian Geometry
This appendix covers the necessary concepts and results from differential geometry. For more full treatment, the reader referred to excellent sources such the book by Lee (2003).
The notation in this appendix is self-contained and separated from the main sections of the paper. Thus, the symbols that were used in the main body can be reused in the following but denote different concept. The notation is introduced as we go along and should not lead to confusion.
To gradually build up the necessary constructions bottom-up, we start from the concept of a topology that we will morph into a Riemannian manifold by progressively adding structure.
A.1. Topological concepts
topology T on a set X is a collection of subsets X, that defines which sets are open in X. This gives rise to the notion of a neighborhood of a point/ E X for an abstract set X.
Definition A.1. A topology T on X is a collection of subsets of P(X) satisfying the following axioms:
1. X and are in T.
2. The union of any family of subsets in T are in T.
3. The intersection of any finite family of subsets in T are in T.
The tuple (X, is called topological space, and the elements of ' are called open sets.
To construct topology, we need basis.
Definition A.2. A basis for topology T set X is a collection B of subsets of X such that:
1. For each x X, there least one B E B such that e B.
2. I E O B2 for B1, B2 E B, then there is B3 E B such that r E B3 S n B2.
In cases where work with subsets of X, Definition A.3 (Subspace Topology). Let X be topological space with topology T and Y C X. The subspace topology on Y is defined Ty (Y (UNUEE TT.
use the topology of X to define topology on the subsets.
The notion of topological leads the construction of topological manifold that is the prerequisite to Riemannian manifold.
Definition A.4. Suppose M is a topological space. We say that M a topological manifold of dimension n or a topological n-manifold if it has the following properties:
1. M is a Hausdorff space: for every pair of distinct points p. E M, there are disjoint open subsets U, V C M such that
p E U and q E V.
2. M is second-countable: there exists countable basis for the topology of M.
3. M is locally Euclidean of dimension n: each point of M has neighborhood that is homeomorphic to an open subset of g".
Definition A.4 specifies what we mean by a homeomorphism.
Definition A.5. Let X and Y be topological spaces, map F F + Y
continuous if the preimage of every open set in Y is open in X.
vV C Y open + F-1(V) C X open
injective if F(x)= = F(y) implies x y.
surjective if for every y E Y there an x E X such that F(x) = y.
13

---

Identifying Metric Structures of Deep Latent Variable Models
bijective if it is both injective and surjective.
homeomorphism is bijective and both F and F-1 are continuous.
If F a homeomorphism, then X and are called homeomorphic. F, on the other hand, is not bijective but only injective, we call F' a topological embedding.
Definition A.6. Let X and Y be topological spaces; a continuous injective map F X + Y is called a topological embedding it is homeomorphism onto its image F(X)= S Y in the subspace topology.
The notion of local homeomorphism defined in the following is fundamental to the construction of a Riemannian manifold. Definition A.7. For two topoloical spaces X and Y, a continuous map X is called a local homeomorphism every point p E X has neighborhood S such that FUU) is open in and F restricts a homeomorphism from U to F(U).
Definition A.4 requires the existence of local homeomorphism from M to R" for every point E M defined on a neighborhood of p. Each such local homeomorphism with the corresponding restriction U M is called a coordinate chart.
Definition A.8. A coordinate chart on M is a pair (U, Φ) where is an open subset of and Φ U + Ú is a homeomorphism from to an open subset Ü = (UU) C R".
It follows from the definition of topological manifold M that every point p E M lies in the domain of some chart. U is called coordinate domain and 6 local coordinate map. The foundation of a smooth manifold is a maximal smooth atlas containing coordinate charts that are smoothly compatible.
Definition A.9. Let . be topological n-manifold. Two charts (U, Φ) and (V, () are called smoothly compatible if UNV Ø or their transition map
1 N : : (V (UNV)
a diffeomorphism in case UNV #
In Definition A.9, note how the smoothness of the transition map can be analyzed in terms of the smoothness of maps between open subsets R", namely the domains the associated coordinate charts.
Definition A.10. A diffeomorphism between open subsets U and V of I2 is a bijective map F U + such that both F and F are continuous and differentiable.
A.2. Smooth manifolds
We now in position to define smooth manifold. Definition A.11. Let M topological manifold,
an atlas A is collection of charts whose domains cover M
a smooth atlas is an atlas A such that any two charts in AA are smoothly compatible
a maximal smooth atlas is a smooth atlas that is not properly contained in any larger smooth atlas
M together with a maximal smooth atlas A is called a smooth manifold denoted by (M,
In the following, define smoothness for map between two smooth manifolds.
Definition A.12. Let M, N be smooth manifolds, and let F M - N be any map. We say that F is a smooth map for every p E M, there exist smooth charts (U, sp) containing p and (V, vb) containing F(p) such that F(U) S V and the composite map & Fo yp is smooth from (U)) to (VV). This means that v Fo ip a map between subsets of g" and RM and we can apply the usual real caleulus.
Since a manifold does not have the usual operations of the Euclidean vector space, one way to construct tangent vector space manifold M at point p E M define it in terms tangent vector to some curve 7 M.
14

---

Identifying Metric Structures of Deep Latent Variable Models
Definition A.13. Let M be smooth manifold, and let 71, 72 (0))= (2) E M, then n and "/2 said to be equivalent if the following holds:
(-e,e) + M be smooth curves in M. Suppose that
d(p o (1) dt
d(y o 72)
dt
1=U
t=
This defines an equivalence relation on the set of all smooth curves through p. and the equivalence classes are called tangent vectors of M at The tangent space T.M to M at is then defined as the set of all tangent vectors at p and does not depend on the choice of the coordinate chart yo.
vector space structure the tangent space T is defined by using the coordinate charts that map between subsets of R. and allow us to do vector addition and scalar multiplication. Lee (2003) shows that the resulting construction is independent of the choice of the charts and that T,M is an n-dimensional real vector space.
Considering maps between manifolds, we want to link the tangent space of one with the tangent space of the other. This done by defining the differential dF of a point p, which a linear mapping from one manifold's tangent space to another's.
Definition A.14. Let M and N be smooth manifolds and F M N be smooth map. For each E M we define map:
d.p: T.M T
called the differential of F at which is linear map between the tangent spaces. The following property defines the differential:
d(ppo) dFp(u)=
dt t=
where 7 is smooth a curve in M through p with 7'(0) = v and yp is a coordinate chart around p. This construction is independent of the choice of the chart yo as shown in (Lee, 2003).
The differential allows us to assess the rank of map between two manifolds.
Definition A.15. Given two smooth manifolds M and map F - N has constant rank at p E M the linear map d.F T - TF(p) has rank r. F is called a smooth submersion if its differential surjective each point (rank F dim N). It is called a smooth immersion if its differential is injective at each point (rank F = dim M)
To define new submanifolds as images of maps, we need the concept of a smooth embedding.
Definition A.16. Let M and N be smooth manifolds, a smooth embedding of M into N is smooth immersion F M -> N that is also topological embedding, i.e., homeomorphism onto its image F(M) N in the subspace topology.
Theorem A. 17 tells us when an injective smooth immersion is also smooth embedding.
Theorem A.17, (Proposition 4.22 in (Lee, 2003)) M and N be smooth manifolds, and M N is an injective smooth immersion. If any the following holds, then F a smooth embedding.
an open or closed map.
F is a proper map.
M is compact.
M has empty boundary and dim dim
Theorem A.18 tells that images of smooth embeddings submanifolds with smooth properties.
Theorem A.18. (Proposition 5.2 in (Lee, 2003)) Suppose M and N are smooth manifolds and F M + is a smooth embedding. Let S F(M). With the subspace topology, S is a topological manifold, and it has a unique smooth structure, making into an embedded submanifold of N with the property that F is diffeomorphism onto its image.
15

---

Identifying Metric Structures of Deep Latent Variable Models
A.3. Riemannian manifolds
Proposition 13.3 of (Lee, 2003) proves the existence of Riemannian metric g in any smooth manifold N, where g is smooth, symmetric covariant 2-tensor field on M that is positive definite at each point p N and defines an inner product in tangent space Ty.N. The tuple (N, 9) is called Riemannian manifold.
Often, in modeling situations, we think of Theorem A.18, that is, we imagine an embedded submanifold S S N in some ambient space N. In this case, we say that F .M + N is a parametrization of S. If there is a Riemannian metric gd on N, there is a way to measure lengths vectors in a tangent space to a point on S using gx as it embedded in this larger vector space that has metric. We can use this to construct metric on by pulling g" by F.
Definition A.19. For a map F; M + F(M) S C N between manifolds M and S, and a metric g²¹ on N, the pullback metric f gº on is defined as:
(F'g))p(u 9 (F)((u).FFF(())
(12)
for any tangent vectors u, U E M.M.
Given two Riemannian manifolds, we can check if they are isometric by using the pullback metric.
Definition A.20. Given Riemannian manifolds (M.g MM and (N.gg) W.gg.), a smooth map F M - N is called an isometry if F is diffeomorphism such that:
F = gMr
(13)
In which case say (M, M) and (N, VN are isometric
series of results (Lee, 2003) Chapters 13,15,16 show that if two manifolds are isometric through diffeomorphism F, then F' preserves lengths of curves, distances, angles, volumes, and other geometric properties between manifolds.
Definition A.21. Given a Riemannian manifold g) we can define the following:
length or norm of a tangent vector E T,M is defined to be
واl = u),y 9p 2)
angle between two nonzero tangent vectors E TyM is the unique E [0, satisfying
cos
|0lg| 1ll
tangent vectors v, w E TyMM are said to be orthogonal (v, w)g = 0. This means either one or both vectors are zero, or the angle between them is πT/ n//2.
given smooth curve y [a, b] MM we can define the length of to be:
[g(?) = f( [1(1)), dt
and the energy of y to be:
1 b
/
E,(r)
2 f. |((1))
Proposition 13.25 of Lee (2003) shows that given curve Ty : [a, b) - M and reparametrizaaiion [c, d] - [a, that is diffeomorphism, the length of the curve 7 o (,) is equal to L,, (7).
The notion of the length of a curve given in Definition A.21 allows us to consider the Riemannian distance from to (p, q E M) denoted by dy (p, q) and defined to be the infimum of L, over all piecewise smooth curve segments from to q. shortest curve not unique since Lg(7) is reparametrization invariant, and a set of curves is locally minimizing (7).
One useful parametrization by arc-length s, which ensures that we move along the curve at a constant speed.
16

---

Identifying Metric Structures of Deep Latent Variable Models
Definition A.22 (Arc-length parametrization). A curve 7 (a, bj M said be parametrized by arc-length if the length of the curve between any two points and l2 is equal the difference in the parameter values l2 f1- Formally, y is parametrized by arc-length if:
(1) =1
for all E [a, b].
Curves that are locally minimizing (n(7) and are parametrized by arc-length are called geodesics defined in Defini- tion A.23.
Definition A.23. Given Riemannian manifold (M, g) and x, E M with x + a geodesic curve on between a and on M is formally defined as a curve y - M that locally minimizes the energy functional
1 (n)
2 h(()))
over all smooth curves 7 [a, -> M connecting two given points
((a)= = x and (bb = y. where g is the Riemannian metric tensor on M.
As discussed by, e.g., Hauberg (2024) it a standard result that a minimizer of the energy functional will necessarily be arc-length parametrized and minimize the length functional.
We can consider the Fréchet mean and variance of set of points on manifold.
Definition A.24, Let (M, dg) Riemannian metric space and let {x1... .f....20 EN } E M be points on the manifold. For any point p E M, Fréchet variance defined to be:
N
(p) Σ
& (p.
Karcher means are the points m M that locally minimize V:
Σ & (p)
772
arg min PEM
If there exists unique m M that globally minimizes V, then it is Fréchet mean.
17

---

Identifying Metric Structures of Deep Latent Variable Models
B. Proofs
Theorem B.1. (Lemma 4. in the main text)
Let and be two smooth manifolds and f E F. then f is smooth embedding and f(Z) C. D is a submanifold in D. In particular, f 2 f(2) is diffeomorphism.
Proof. By Definition A.15 f is smooth immersion it is smooth map of constant rank with injective differential (assumptions A2-A3). Using Theorem A.17 with the fact that 2 is compact set (assumption A1) gives us that is smooth embedding. Finally, Theorem A.18 gives that ((2) is submanifold of 'D and f is diffeomorphism its image.
Theorem B.2. (Lemma 4.3 in the main text) Let far fo E F and consider the generator transform Aa,b Zo Zi defined by
= fo
fa(z)
Then (z,((z) and Ar]b(=) - (z) fo fi(=) are diffeomorphisms.
Proof. The result follows from Theorem B.1 with the fact that fas fo E have the same image due to assumption A4, Theorem B.3. (Theorem 4.5 in the main text)
Let 0g (Sa, Pz,) and O, Pz.) with P9. = Po. and let (Za.gl=) and (Zo.ght) be the associated Riemannian manifolds, then the generator transform is an isometry and it holds that:
(Aa.b)" g = gf
(14)
Thus, making (Za:gr>) and (Zs.g") isometric. This makes Riemannian geometric properties such as lengths of curves, angles, volumes, areas, Ricci curvature tensor, geodesics, parallel transport, and the exponential map identifiable.
Proof. We first show that the generator transform is isometry. Let us recall that by definition the pullback we have:
g (u, v) = P. (u), dfo,.))))
for u, U. T, and where (v) denotes the differential of the map So point p Zb evaluated the vector E Zn Using this, will check Eq. 14 directly:
((Ao,))* g (v, f os(()) (d/o,s, o (v) (V), (r)())())))))))
gilel (d(fo o fo fa)al)) d(fo fot so)r(w))
(15)
gla
for y, W Z.
Since we have shown that the generator transform is an isometry, we can conclude that (Za, g==) and (Z).gr)) are isometric and thus their Riemannian metric properties are identical (O' Neill, 1997) 6 and 7]. To connect identifibility, express any of the properties that the output will be same. We will show an example of this and prove the claim for the goedesic distance function (Theorem 7 in the main text) below.
task of the form described in Section and use the isometry result above to conclude
Theorem B.4. (Theorem 4.7 in the main text)
Let 0., = (sa, P2.)) and 0g = (fb, Pz.)) with Po., = Po, and let Aa,b be the generator transform between the parameters. Furthermore, let (Ze.gfa) f and (Zo, (>) ( be the associated connected Riemannian manifolds. Then, the geodesic distance between 21 and z2 is identifiable and it holds that:
dgto (21, 22) = dots (Aa,)(21), A (z,))
(16)
for some 21, 22 E Zg be two points in the latent space that correspond some x1, X2 E M on the manifold.
18

---

Identifying Metric Structures of Deep Latent Variable Models
Proof. We need to check Definition 2.1 to show that the task of measuring distances in the latent space identifiable using the geodesic distance function.
Let X2 E 'D be the observed data points and consider the inverse of the decoders our selection function given x
z = s(f,, x1) = (x1) and s(A8,, x1) = s(6,, fo (x1) = z
(17)
in similar way we obtain z and z Define the task of measuring distances on the manifold as:
1(0a; (x1, {zī. 2 } = dgta (2f: 2z) 1(0b. (x1, {z], 2 = d (z), 25)
(18)
To check the selection function, take with A E (MM and some X1 E D. Then.
2 = 800, (x1)
f/ fo o fa (x1)
Aa, x1))
(19)
A(s(Oa: x1))
Aa,h(Ja (x1))
where we have used that A is almost everywhere equal to the generator transform Aa,s due to Xi & Bloem-Reddy To check the task function, let us recall that:
(2), inf f (1(())
(20)
where 7: [c, Za is curve in 2, connecting zf and z such that 7c)) 2 and 7(d) zg.
As a geodesic is not unique, multiple curves result in the infimum in Eq. 20. Let 7 be solution, then it is by Definition A.23 geodesic curve. By Theorem 4.5 we have that (Z., (f. and (Z,, g'o) are isometric, which means that
Aa,b maps geodesics to geodesics Neile,,, 1997), then 7o 1č, d) + Z, constructed from 7o by Aa,((7a) is a geodesic in 2g and thus a solution for d (Zf,) Aa, (z)) = d (z) 2))
C. Computing geodesics
A geodesic between and b is defined be curve (t) defined on some interval (usually [0, 1) such that y(0) = and ((1) b minimizing the length functional defined in Definition A.21. In our work, we choose parametrize a geodesic by a cubic spline (Schoenberg, 1946) and optimize the energy functional defined in Definition A.21 with respect to the free parameters using gradient methods.
C.1. Gedoesic parametrized by a cubic spline
Having settled on cubic spline as parametrization of a geodesic, we will now describe the construction of the spline and use to derive the free parameters of the resulting curve that we can use when minimizing the energy of that curve.
cubic spline is a piecewise function with pieces that are cubic polynomials. The points where the pieces meet are called the knots, h, and we want construct a continuous spline with continuous first and second derivatives. Individual components are polynomials, so we only need constrain their behavior the knots to satisfy the requirements. Suppose the knots are known, and we are using the splines to interpolate set of points. In that case, these constraints and boundary constraints will usually give system of linear equations that can be solved to find the coefficients of the polynomials. In our setting, however, we are using splines to define a path between two points, and the knots are the unknown parameters of the problem, as well as the coefficients of the polynomials. Furthermore, given that a, E R", we will look to parametrize a geodesic curve y(t) = (t)....... (t)) E R" and thus we will have n splines, one for each dimension. In the following, we will describe how such a construction works for one dimension and invite the reader to conceptually repeat this for n dimensions.
19

---

Identifying Metric Structures of Deep Latent Variable Models
Following the idea of (Hauberg, 2024), will start by connecting two points in the latent space (a, E IR) by straight line : [0, 1] + R. defined I(t) + t(b a) and then find cubic spline that will start and end in 0 to parametrize
deviation from the line. The result will be curve 7(t) ((t) S(t) that will connect the two points on the manifold.
The spline S(t) is defined as a piecewise function with cubic polynomials with coefficients ag, bs. Ct. d2 E R. each defined on an interval [A,, h441] where h; are the knots with ho = and h, set to be the endpoints.
(1()) if E [ b1] S2(t)
if E [h1 h2]
S(t) =
(21)
Stt(t)
if E [hn-1.
where each (1) is a cubic polynomial:
S,(1) - b.(1) + ca(1) + d:(1)² for L E [h
(22)
In the following, let Ç = (a1, b1. c1, .1...., an, bn Cn. dn) be a vector of all coefficients of the polynomials in our spline and E(i, // be a subvector of & containing the coefficients of the i-th and j-th polynomial.
Boundary conditions mean that we need our first polynomial start in (0, 0) and the last polynomial to end in (1,0). This gives two equations:
(0) = = 0 and (1, on + b + dn + Cn = 00
(23)
which we translate into the following matrix equation of the coefficients & and 2 x 4n matrix B:
0
BET
(24)
(0]
where
0
0
0
(25)
0
The continuity conditions are met when the values at the knots are the same for the two meeting polynomials. This can be expressed as:
(b.)) = S + (hh)) S (l+) for ,,, n
(26)
and for each knot we can write this as dot product of the coefficients E(i, i 1] and a vector cQ:
h4
h
h4 n -
(27)
such that the condition knot i becomes:
de + 11"
(28)
The conditions of first and second derivatives being continuous can be expressed in a similar way.
(((b)) (ha)) + (((hk) (ha) 0 for i 1, ,,..., (h,) (h+) (h.) - S (h:) = 0 for i 1, 1...., n
(29)
and we can write these conditions as dot products of the coefficients Eli + 1] and vectors of and of:
d
2h
3/
0
-1 -2h -3h
(30)
2000
2
6h2
0 -2 -Ghs]
(31)
such that the conditions at knot become:
deli, 11Th =0
(32)
20

---

Identifying Metric Structures of Deep Latent Variable Models
Having defined the smoothness constraints for given knot can construct matrices CO Ct C each with dimensions (n- 1 1) x 4n where each row i corresponds the respective constraint at the knot with 4 (?- - 1) zeros before the constraint and 4 (n 1 1-2 i) zeros after the constraint. E.g. for = 4 the C matrix would look as follows:
0
0
0
0
0
0
0
CO
0
where each of is a row vector as defined in Eq. 27.
Now, the final system of equations can be written as:
B
5" =0
(33)
resulting in 4n - equations for 4n unknowns. To solve this system of equations in an interpolation setting, we usually impose two additional constraints to get a square system of equations. These constraints can be that the second derivative zero at the endpoints or that the second derivatives at the first and last knots are equal. The former is known in the literature as natural spline and the latter as not-a-knot spline (Kress, 2012).
Getting back our original task of finding free parameters of the curve that we note that given that we have 4n coefficients, the actual number of free parameters is considerably smaller due to the constraints. The problem in Eq. 33 is known as the problem of finding the Null Space of the row space of matrix A. A basis for such null space, denoted by (A), can be found by computing the Singular Value Decomposition (SVD) (Klema & Laub, 1980) A. If A is of rank r, then SVD of is given by A = UEVT where and V are orthogonal matrices with dimensions ((4n 2) x 4n) each and Σ is (4n 4n) diagonal matrix with r nonzero singular values in the diagonal. The null space of A is then given by the columns of y corresponding to the zero singular values. Treating ([A) =: N as (4n X (n- r)) matrix, have arrived at set of free parameters that optimize with respect to. To recover the full set of coefficients Ç. we can use the following equation:
optimize its energy with respect to,
{ Ne
(34)
and evaluate the spline at the desired points to get the curve 7().
C.2. Optimizing the spline to find geodesic
In the previous subsection, we have reduced the infinite set of functions in which we are looking for a geodesic to another but considerably smaller, infinite set of splines. The next step is to use optimization to find the spline that minimizes the energy defined in Definition A.21. Calculating the energy requires computing an integral, which is, in practice, approximated by a sum over a discretized interval.
In the following treatment we assume that D Rh and let fo Z - R" be decoder parametrized by 0 and 7: [0, Z be spline in the latent space, then the approximation of the energy of is given by:
1 E(7) 2 J h (1)
2 /./0
oy fo (xtt)
(35)
2 (e(())()) - f E(())
where T is sequence of n, points in the interval [o, 1]. Combining this with the discussion in the previous section, we can now define the optimization problem as simply:
min
B(ww)
(36)
21

---

Identifying Metric Structures of Deep Latent Variable Models
where و de note the parameters of the k different splines given the dimensionality of the latent space 2 E I and remind the reader that (t) = (nu,, (t)....., Twm (t)) 2.
Using optimization to learn the manifold will result in different approximations depending on the initialization of the parameters and the optimization algorithm used. Considering the problem light the first line of Eq. 35, we can see that the Riemannian metric becomes the stochastic term. In this sense, the manifold is stochastic, and the resulting distances between points will be affected by this stochasticity.
Following (Syrota et al., 2024), having access to ensemble of decoders allows in principle, to make the optimization problem in Eq. 36 aware of the uncertainty involved. The methodology effectively uses Monte Carlo methods to compute the energy with respect to the uncorrelated posterior over parameters. This posterior approximated by an ensemble. The following equation the optimization problem we solve and makes the idea explicit:
fa (nw((t)) fōr (7w (ti-)))
min
(37)
2A
where A, -4 1 is the step size in the discretization of the interval [0, 1] and is assumed to be constant. The decoders fg. and fāx, are sampled uniformly and independently from the ensemble.
22

---

Identifying Metric Structures of Deep Latent Variable Models
D. Experiments
M-flow on MNIST and CIFARIO
The decoder is modeled by 10 coupling flow layers introduced by (Dinh et al., 2015) where the conditioner is 2-layer MLP with 50 hidden units and ReLu activation function, and transformer the RQS spline (Durkan al., 2019) in the interval [-3,3] with 8 knots (22 knots for CIFARI0). The decoder has 5.5 million parameter for MNIST and 55 million for the CIFAR 10 model.
The encoder consists of 4 Residual Blocks (He et al., 2015) with Convolutional transformations each time multiplying the number of channels by 5 and the Relu activation function. The last layer is the linear layer with hyperbolic tangent function multiplied by 3 at the end to constrain the output to [-3,3] and accommodate the coupling flow above. The encoder has 135 thousand parameters for both MNIST and CIFAR 10 models.
Adam (Kingma Ba, 2017) was used for all the experiments.
VAE on FMNIST and CELEBA
The encoder consists of five x 4 convolutional layers, the first two with 128 channels, followed by the next three with 256 channels, with the first and last having a stride of one and the others stride of two; two linear layers, one with 256 umits and the second with 64 units (latent dimensions) for the VAE approximate posterior mean and 64 for the VAE approximate posterior variance.
The decoder starts with two fully-coneccted layers, one with 256 units and one with 16384 units; five 4 x 4 resize convolutional layers (Odena et al., 2016), with the same stride configuration as the encoder, but with 256, 128, 64, and 32 channels, respectively. All layers the encoder and decoder have an ELU activation function. Total number of parameters: 6.5 million.
The VAE is trained with an additional loss term coming from the Perceptual loss (Johnson al., 2016).
Geodesic computation
The geodesic computation was done on models that had an ensemble of decoders following the procedure of (Syrota et al., 2024) and hyperparameters in Table 2.
Parameter
Value
Initialization of free parameters
(straight line)
Number of polynomials the spline
Discretization in time (energy)
256
Discretization in (final length)
256
Optimizer
Adam (Kingma Ba, 2017)
Max. steps
4096
Learning rate
0.01
Early stopping patience (steps)
100
Early stopping delta
1.0
Table 2: Shared geodesics training parameters for all models.
23


Reviews:
[{"review_id": "rn2CwB32ut", "reviewer": "Reviewer_Rhp6", "summary": "This paper studies the geometry of latent spaces of latent variable models like VAEs, normalizing flows, diffusion models etc. Primarily, the authors highlight that many seemingly simple factors like the latent coordinates or their pairwise euclidean distances etc. of latent variable models are provably not identifiable. This means that under a probablistic setting: despite having different parameterizations- they do not lead to the same distributions. The motivation of the generative model is to discover meaningful intrinsic properties of data that should not be dependent on randomness that is inherent due to training, noise etc. \n\nThe authors explore this in the context of differential geometry of the latent space of generative models. The main presmise is that unlike the latent coordinates or even euclidean distances between them -  geodesic distances computed using a pullback metric from the observation space satisfies identifiability. This paper both rigorously proves this and emperically demonstrates the hypothesis with experiments from MNIST and and CELEBA.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for valuable feedback and favoring acceptance. \n\n>There is scope to be much more comprehensive in the experiments - for e.g. I would be very glad to see a table similar to Figure 7 for the transcriptomic data example from Figure 1 and that would solidify the main message of the paper across different models and types of data.\n\nWhile the focus of our work is theoretical (see reply to reviewer ev8z for more details), we acknowledge the reviewers’ requests for additional experiments and will extend the paper with an Appendix including extra results and details on implementations and model choices.\n\nIn particular, addressing this point we run experiments of FMNIST and CIFAR10. Following the approach in the main paper, we split them into an experiment satisfying the injectivity constraint (CIFAR10) and an experiment that does not satisfy the constraint (FMNIST). The histograms and an updated table are available [at this link](https://tinyurl.com/yxzwks2j). The findings are similar to those reported in the submitted paper, which supports the presented theory.\nFurthermore, we share the sentiment that the transcriptomic data example is underexplored and plan to add a distance matrix for the geodesic distances to Figure 1. \n\n>I am tempted to make the conclusion that generative models when trained well, tend to produce representations that preserve \"intrinsic\" distances on the data-manifold. \n\nIndeed, the main result of the paper, Theorem 4.5, shows that under any well-trained (injective) generative model, the indeterminacy transformations of the true latent space will automatically respect the true geometry of the data. I.e. it proves that any geometrical information extracted from the model is identifiable. \n\n>For example - If I leave out the decoder 'f' completely and simply use my training dataset with a k-nearest neighbor graph and then compute a Dijkstra-like shortest path distance on this graph - I suspect it would correlate quite well with the construction of the pullback metric and computation of the geodesic distance from (35) which very much depends on the chosen model 'f'. \n\nWe appreciate and share your intuition. [This paper](https://arxiv.org/abs/1812.08284) considers geodesics under such an approach. However, we should emphasize that this is heuristic and is not strictly tied to our theoretical results on identifiability. Practically, we expect the approach to work well, though.\n\n>It would be nice to have some experiments where the variability in the geodesic is also visualized somehow (like Fig 5 but with a band instead of just one curve) - especially in comparison to this model-independent geodesic distance\n\nWe agree that different optimisations can lead to different geodesics between the same two points, and acknowledging that a geodesic in itself is not unique. However, in this paper we address the variability of the distance measure across retraining of the models themselves. In our experiments, this means that we compute the geodesic distance between the same two points across 30 different models with 30 different latent spaces. Fig.5 shows just one geodesic in one latent space. Therefore plotting the band would be concerned with a different kind of variability. \n\n> Is Table 1 reporting the values for the geodesic or Euclidean distance? I am unable to parse the message here\n\nBuilding on the above, for each pair of points we have 30 measurements of distances according to both Euclidean and geodesic measures. To demonstrate that the geodesic distance is more stable, we compute the coefficient of variation (CV) of both distance measures for each point pair. These are then plotted in Fig. 7 and Table 1 reports the Student’s t-test for the CVs with one sided null hypothesis that geodesics are more stable, i.e. has a smaller CV. The message of Table 1 is that geodesic distances exhibit significantly less variation than Euclidean distances.\n\nWe acknowledge that the table caption should be improved. We will make these details more explicit and will further add an appendix detailing the experiments.\n\n> Is the familiarity in the trajectories reported in Figure 6 always the case? I was expecting that the trajectory of the Euclidean geodesic to give images with comparatively more \"abrupt\" changes in each step in comparison to the geodesic which should give a more seamless transformation. It is somewhat visible already - although not strikingly.\n\nYour expectation is correct. We often see that geodesics provide more ‘smooth’ interpolations while Euclidean interpolations are more ‘abrupt’. This happens because geodesics move with constant speed in data space. This trend is, however, more evident, e.g., in MNIST than in CelebA. We’re happy to include such MNIST examples in the appendix if they are deemed interesting.\n\nWe hope that our clarifications and additional experiments strengthen your view of the paper, and we thank the reviewer for the thoughtful feedback.", "claims_and_evidence": "Yes. The central point is proved and also demonstrated emperially. However - I do find the experiments lacking generality in the class of generative models investigated.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "There is scope to be much more comprehensive in the experiments - for e.g. I would be very glad to see a table similar to Figure 7 for the transcriptomic data example from Figure 1 and that would solidify the main message of the paper across different models and types of data.", "relation_to_broader_scientific_literature": "To the best of my knowledge - this paper discusses an important issue that even though is not unique in all literature - provides a novel comprehensive analysis: theoretically and empirically regarding parameterization invariance of generative models.", "essential_references_not_discussed": "Its fine", "other_strengths_and_weaknesses": "Overall - I think this is a nice paper with a comprehensive conceptual and theoretical treatise on developing parameterization invariant representations. However, the experiments do lack generality and some more convincing demonstrations of the core message would go a long way. Therefore, I am very much on the border leaning slightly positively because of a well-compiled submission and an interesting read.", "other_comments_or_suggestions": "I am tempted to make the conclusion that generative models when trained well, tend to produce representations that preserve \"intrinsic\" distances on the data-manifold. For example - If I leave out the decoder 'f' completely and simply use my training dataset with a k-nearest neighbor graph and then compute a Dijkstra-like shortest path distance on this graph - I suspect it would correlate quite well with the construction of the pullback metric and computation of the geodesic distance from (35) which very much depends on the chosen model 'f'. It would be nice to have some experiments where the variability in the geodesic is also visualized somehow (like Fig 5 but with a band instead of just one curve) - especially in comparison to this model-independent geodesic distance.", "questions_for_authors": "- Is Table 1 reporting the values for the geodesic or Euclidean distance? I am unable to parse the message here\n- Is the familiarity in the trajectories reported in Figure 6 always the case? I was expecting that the trajectory of the Euclidean geodesic to give images with comparatively more \"abrupt\" changes in each step in comparison to the geodesic which should give a more seamless transformation. It is somewhat visible already - although not strikingly.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w1h5bP4Zrs", "reviewer": "Reviewer_TdYi", "summary": "In this paper, the authors address the challenge of statistical identifiability in deep latent variable models, which are used to extract condensed representations of data. Traditional methods attempt to improve identifiability by imposing constraints like labeled data or limiting model expressivity. Instead, the authors shift the focus from identifying individual latent variables to identifying meaningful relationships between them—such as distances, angles, and volumes.\nThe authors prove that these geometric relationships can be statistically identified under minimal assumptions, without additional labeled data. This result is significant for fields like scientific discovery, where reliable data interpretation is crucial. In the experiments, the authors test their assumption over 2 different datasets, MNIST and Celeb-A. They performed Student's T-test to show that the geodesic distances have much less variance than Euclidean distance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's valuable feedback and support for acceptance.\n\n>I believe there can be more benchmarks included in this paper. Other commonly used image datasets, such as Fashion-MNIST, SVHN, Cifar-10, should also be computationally cheap to run.\n\nWhile the focus of our work is theoretical (see reply to reviewer ev8z for more details), we acknowledge the reviewers’ requests for additional experiments.\n\nIn particular, addressing this review we run experiments of FMNIST and CIFAR10. Following the approach in the main paper, we split them into an experiment satisfying the injectivity constraint (CIFAR10) and an experiment that does not satisfy the constraint (FMNIST). The histograms and an updated table are [at this link](https://tinyurl.com/yxzwks2j) and we provide the table here as well. \n|                | MNIST  | CELEBA  | FMNIST | CIFAR10 |\n|-------------- |--------|--------|--------|--------|\n| **t-statistic** | -8.64  | -22.33 | -16.75 | -42.83 |\n| **p-value**    | 1.00   | 1.00   | 1.00   | 1.00   |\n\n*Table: One-sided Student's t-test for the variability of geodesic versus Euclidean distances.*\n\nThe findings are similar to those reported in the submitted paper, which supports the presented theory. \n\nWe will further extend the paper with an appendix including extra results and details on implementations and model choices.\n\n>1. Why do the authors only use 3 classes of MNIST? Is this cherry-picked?\n\nAs mentioned in the submitted paper, the choice of 3 classes for MNIST was to simplify plotting. We point to the extra experiments above and code in the submitted supplementary material (CelebA) to document absence of any cherry-picking.\n\n>2. I don't see why we should expect that the digit class 0, 5, and 7 are naturally close to each other. I hope the authors can explain it in more details.\n\nWe do not expect any classes to be naturally close to each other. The digits 0,5 and 7 were picked randomly and their placement in Fig. 5 is merely a consequence of optimization of the model fitting.\n\nWe hope to have addressed your concerns and sincerely thank you again for your review.", "claims_and_evidence": "I believe all the claims are well supported.", "methods_and_evaluation_criteria": "I believe there can be more benchmarks included in this paper. Other commonly used image datasets, such as Fashion-MNIST, SVHN, Cifar-10, should also be computationally cheap to run.", "theoretical_claims": "I believe the theoretical claims are sound.", "experimental_designs_or_analyses": "I find the claim in the result analysis confusing. For example,\n1. Why do the authors only use 3 classes of MNIST? Is this cherry-picked?\n2. I don't see why we should expect that the digit class 0, 5, and 7 are naturally close to each other. I hope the authors can explain it in more details.", "relation_to_broader_scientific_literature": "A key contribution of the paper is linking identifiability to Riemannian geometry, establishing a novel theoretical framework. This connection allows practitioners to leverage established Riemannian tools (e.g., Riemannian averages, covariances, and principal components) to analyze latent structures in a statistically sound manner.", "essential_references_not_discussed": "I think the literature is reviewed well.", "other_strengths_and_weaknesses": "The goal of this work is well motivated and the paper is well strucutured in general.", "other_comments_or_suggestions": "I don't have any other comments.", "questions_for_authors": "I have listed my questions in previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "There is no ethical concern."}}, {"review_id": "IJPwPYeyMR", "reviewer": "Reviewer_ev8z", "summary": "This paper addresses the problem of learning identifiable representations from a novel perspective, focusing on the distances between representations rather than their coordinates. The authors begin by discussing the challenge of identifiability in latent variable models, emphasizing that maximum likelihood estimation (MLE) alone does not guarantee identifiability. They highlight that training the same model twice can result in different learned representations. To address this, the paper introduces a new notion of identifiability: given two models, A and B, the geodesic distance between latent variables $z_1$ and $z_2$ is considered identifiable if it remains consistent across both models. To validate this idea, the authors train multiple VAEs on different datasets and compare Euclidean and geodesic distances between 100 randomly selected test sample pairs. Their results show that geodesic distances exhibit lower variance, supporting their proposed notion of identifiability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for thoughtful feedback, as well as their support for acceptance.\n\n>However, I find that the experiments do not strongly validate the hypothesis that geodesic distances are truly identifiable. The main result, particularly Figure 7\n\nWe emphasize that the main result of the paper is Theorem 4.5. It shows that under any (injective) generative model, the indeterminacy transformations of the true latent space will automatically respect the true geometry of the data. I.e. it proves that any geometrical information extracted from that model is identifiable. We achieve this without placing (notable) restrictions on the model, architecture, or training methods. \n\nWe motivate our paper and focus our experiments within the point of view of geodesic distances which is an example of geometrical information used in practice. This leads to Theorem 4.7 proving that these are identifiable.\n\n>there is considerable overlap between the two histograms. [..] I would argue that this falls short of demonstrating that they are identifiable in a definitive sense\n\nWe want to clarify that identifiability is a theoretical question and that our present theorems are the definite evidence.\n\nThe experiments demonstrate that the asymptotic property in Theorem 4.7 is practically achievable in standard models using off-the-shelf methods on finite data. \n\nHowever, we understand the concern about overlapping histograms, but emphasize that such is to be expected and is fully in line with the theory:\n- Proposition 4.8 shows that a (scaled) Euclidean distance can be identifiable if the model behaves in a Euclidean way in a region.\n- Riemannian geometry is locally Euclidean, implying that local Euclidean distances can be expected to be robust when points are close. \nConsequently, neighboring points can have robust Euclidean distances (low CoV), implying overlapping histograms.\nFinite data further implies uncertainty compared to theoretical treatment and makes the estimation of the manifold stochastic. As the reviewer points out, optimization of geodesics is noisy and may lead to a distorted picture (no efforts were made to counteract this on specific datasets or models).\n\nWhile the focus of our work is theoretical, we acknowledge the reviewers’ requests for additional experiments, and have added results for FMNIST and CIFAR10. See [link](https://tinyurl.com/yxzwks2j) for plots and tables. FMNIST results are similar to previous results, while CIFAR10 shows the clear separation of histograms that you have requested.\n\n>there is a strong link between this work and the literature on lie group latent spaces [5-8]\n\nWe do not claim to be the first to explore transformations of latent space; however, we are the first to apply them in the context of identifiability using Riemannian geometry.\n\nWhile previously used for different purposes, latent space transformations have proven valuable in various areas, including disentangled and equivariant learning, highlighting their mathematical and conceptual connections.\nThe mentioned literature assumes a disentangled latent space where transformations decompose into individual factors of variation and the goal is to find representations that respect this structure (roughly speaking, equivariance of $A_{a,b}$ (Def 4.2)). Instead of enforcing specific properties, our theory analyzes the natural properties of $A_{a,b}$, and we find that just by learning a generative model, $A_{a,b}$ will automatically respect the latent Riemannian geometry (Theorem 4.5).  Thus, our theory is valid regardless of whether a disentangled latent space exists in the sense of [5].\n\n>previous works on computing and analyzing geodesics must be cited\n\nWe already cite [4] and will include [1-3] as appropriate.\n\n>The claim \"[...] most data is equipped with units of measurement[...].\" does not hold in most vision and nlp tasks \n\nWe acknowledge that there are cases where picking a suitable metric in data space is not trivial, but emphasize that this metric should only be meaningful infinitesimally. E.g. Euclidean distances are generally unsuited for images, but infinitesimally they are perfectly reasonable. Our theory also applies when pulling back ‘perceptual distances’, e.g. using features from pre-trained neural networks, see e.g. [this paper](https://tinyurl.com/3jvx27r3).\n\n>using Euclidian distance in x-space, we implicitly assume that the dataset is dense\n\nWe disagree with this statement as we are not measuring 'isomap-style' geodesics. We measure geodesics along the manifold spanned by the model which does not require dense data to be identifiable.\n\n>The main weaknesses of the paper are: (1) Not discussing the connection between this work and all the disentanglement work (2) The results sadly are not very promising.\n\nTo conclude:\n1. see the discussion above,\n2. our main results are theoretical, and the experiments are fully consistent with the theory. Our released code provides a path for turning theory into practice.", "claims_and_evidence": "See Weaknesses", "methods_and_evaluation_criteria": "See Weaknesses", "theoretical_claims": "The theoretical framework, to the best of my ability, checks out.", "experimental_designs_or_analyses": "See Weaknesses", "relation_to_broader_scientific_literature": "- I think the paper misses an important section of literature: Disentangled representations + lie groups, namely [5] and all the papers that were built on top of that (e.g. [6-8]) as well as many other equivariant neural networks. Currently, the paper reads in a way that is the first paper that thought about representations in terms of transformations. This claim needs to be toned down quite a bit I would say. As these papers show the group element $g$ $g. x_1 = x_2$  indeed corresponds to the geodesics. \n\n- Furthermore, while I understand that the authors do not make a big claim on proposing a new way to compute geodesics. However previous works on computing and analyzing geodesics must be cited: (3.g. [1-4])", "essential_references_not_discussed": "See Relation To Broader Scientific Literature*\n\n[1] Chadebec, Clément, and Stéphanie Allassonnière. \"A geometric perspective on variational autoencoders.\" Advances in Neural Information Processing Systems 35 (2022): 19618-19630.\n\n[2] Chen, Nutan, et al. \"Fast approximate geodesics for deep generative models.\" Artificial Neural Networks and Machine Learning–ICANN 2019: Deep Learning: 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17–19, 2019, Proceedings, Part II 28. Springer International Publishing, 2019.\n\n[3] Chen, Nutan, et al. \"Metrics for deep generative models.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2018.\n\n[4] Arvanitidis, Georgios, Lars Kai Hansen, and Søren Hauberg. \"Latent space oddity: on the curvature of deep generative models.\" arXiv preprint arXiv:1710.11379 (2017).\n\n[5] Higgins, Irina, et al. \"Towards a definition of disentangled representations.\" arXiv preprint arXiv:1812.02230 (2018).\n\n[6] Zhu, Xinqi, Chang Xu, and Dacheng Tao. \"Commutative lie group vae for disentanglement learning.\" International Conference on Machine Learning. PMLR, 2021.\n\n[7] Wang, Tan, et al. \"Self-supervised learning disentangled group representation as feature.\" Advances in Neural Information Processing Systems 34 (2021): 18225-18240.\n\n[8] Yang, Tao, et al. \"Towards building a group-based unsupervised representation disentanglement framework.\" arXiv preprint arXiv:2102.10303 (2021).", "other_strengths_and_weaknesses": "**Strengths**:\n\n- The paper reads really well. And it offers a good introduction to the identifiability problem\n\n- The key contributions of the paper namely Theorem 4.7 is a good theoretical contribution and worth highlighting.\n\n- Regardless of the results: the experiment design is very systematic and justified. \n\n**Weaknesses**\n\n- The theoretical framework, to the best of my ability, checks out. However, I find that the experiments do not strongly validate the hypothesis that geodesic distances are truly identifiable. The main result, particularly Figure 7, is somewhat underwhelming, as there is considerable overlap between the two histograms. While this does suggest that geodesic distances are more identifiable than Euclidean ones, I would argue that this falls short of demonstrating that they are identifiable in a definitive sense. Ofcourse, there are some notable challenges in measuring geodesic distances accurately. Specifically: (1) 1. The optimization procedure used to compute geodesic distances is not optimal, as the authors themselves acknowledge. Euclidean distance in the data space is not an ideal metric for these datasets. A useful addition to the paper would be a toy experiment where the data manifold is known, allowing for precise measurement of the true geodesic distances. \n\n- As pointed out in the prior work section, while this is the 1st work to the best of my knowledge that points out the connection between identifiability and transformation, there is a strong link between this work and the literature on lie group latent spaces [5-8]. The authors need to discuss some of these works and tone down the claim that this is the first work that focused on transformations. \n\n- While the authors point this out in the first paragraph of Section 7, it remains a strong problem. The claim \"We argue that most data is equipped with units of measurement, which greatly simplifies the task of picking a suitable metric in the observation space.\" does not hold in most vision and nlp tasks for example. Moreover, if we're using Euclidian distance in x-space, we implicitly assume that the dataset is dense, which is not a very reliable assumption.       \n\n\nOverall, I think the connection between identifiability and distances is a perfectly valid contribution worthy of publishing and of interest to the representation learning community. The main weaknesses of the paper in my opinion, are: (1) Not discussing the connection between this work and all the disentanglement work that was built on [4] using groups (\"transformations\") (2) The results sadly are not very promising.", "other_comments_or_suggestions": "- I want to applaud the authors for writing Section 7.", "questions_for_authors": "- In Figure 1: I'm wondering if you can show the the matrices using the geodesics distances as well instead of Euclidean? How would the matrices for different runs look like?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates identifiability in latent variable models, focusing on identifiability of geometric features such as distances in the latent space. The core of the paper are theoretical results, on the identifiability of a Riemannian metric in the latent space of a deep generative model, and of related Riemannian geometry properties. Reviewers would have liked to see more experimental validation of the learned geodesics, specifically as there is still some unexpected variation in Figure 7, but all reviewers are in favour of accepting the paper, in particular given the potential importance of the theoretical results.
Decision: Accept (poster)