Connecting Neural Models Latent Geometries with Relative Geodesic Representations
Authors: Hanlin Yu
Berfin Inal
Georgios Arvanitidis
Søren Hauberg
Francesco Locatello
Marco Fumero
Venue: NeurIPS 2025 poster ()
Keywords: Relative representations
representation alignment
latent space geometry
geodesics
latent communication
Primary area: deep_learning

Abstract: Neural models learn representations of high-dimensional data on low-dimensional manifolds. Multiple factors, including stochasticities in the training process, model architectures, and additional inductive biases, may induce different representations, even when learning the same task on the same data. However, it has recently been shown that when a latent structure is shared between distinct latent spaces, relative distances between representations can be preserved, up to distortions. Building on this idea, we demonstrate that exploiting the differential-geometric structure of latent spaces of neural models, it is possible to capture *precisely* the transformations between representational spaces trained on similar data distributions. Specifically, we assume that distinct neural models parametrize approximately the same underlying manifold, and introduce a representation based on the *pullback metric* that captures the intrinsic structure of the latent space, while scaling efficiently to large models. We validate experimentally our method on model stitching and retrieval tasks, covering autoencoders and vision foundation discriminative models, across diverse architectures, datasets, pretraining schemes and modalities. Code is available at the following [link](https://github.com/marc0git/RelativeGeodesics).

Connecting Neural Models Latent Geometries with Relative Geodesic Representations
Hanlin Yu University of Helsinki
Berfin Inal University of Amsterdam
Georgios Arvanitidis
Søren Hauberg DTU
DTU
Francesco Locatello
Marco Fumero!
IST Austria
IST Austria
Abstract
Neural models learn representations of high-dimensional data on low-dimensional manifolds. Multiple factors, including stochasticities in the training process, model architectures, and additional inductive biases, may induce different representations, even when learning the same task on the same data. However, it has recently been shown that when a latent structure shared between distinct latent spaces, relative distances between representations can be preserved, up to distortions. Building on this idea, we demonstrate that exploiting the differential- -eeomtrree structure latent spaces of neural models, it is possible to capture precisely the transformations between representational spaces trained on similar data distributions. Specifically, assume that distinct neural models parametrize approximately the same un- derlying manifold, and introduce a representation based on the pullback metric that captures the intrinsic structure of the latent space, while scaling efficiently to large models. We validate experimentally our method on model stitching and retrieval tasks, covering autoencoders and vision foundation discriminative models, across diverse architectures, datasets, pretraining schemes and modalities. Code is available httth:tt///// http//ggttttttttee/ee wttpt//..ww..arooggggaaaaaaaaaaa
Introduction
D
D2
Neural models learn meaningful representations of high-dimensional data generalizing to many tasks, spanning different data modalities and do- mains. Recent research reveals that these mod- els often develop similar internal representations given similar inputs [Li al.) 2015, Moschella et al. 2023 Fumero et al.) 2024 Kornblith et al.; 2019]. phenomenon that was observed in bi- ological networks [Laakso and Cottrell 2000 Haxby et al. 2001]. Remarkably, even when models have different architectures, their inter- nal representations can frequently be aligned through simple, e.g. orthogonal, transformation Maaorca et al. 2024 Lähner and Moeller 20044 Moayeri et al.
T
Zi
E1
Ez
X
Figure 1: Neural models trained on similar data learn parametrizations of the same mani- fold. NNs learn parametrizations D2) of the same underlying manifold y up isometries T. Pulling back the metric from y makes relative geodesic representations invariant transforma- tions T between latent spaces E, and Z,.
2023 This suggests certain consistency in how neural nets encode informa- tion, emphasizing the importance of studying the
'Corresponding emails: marco. www.roffumrristttcmmmm ac. at, hanlin. www.ee..eeeeeeiii....
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

internal representations and the transformations that relate them, to the extent to hypothesize whether neural nets are converging toward unique representation of reality (Huh al. [2024..
One strategy to understand how different models are related is to identify representations that are invariant to transformations between distinct models' representational spaces. A simple and effective recipe is that of relative representations [Moschella et al.) 2023]. where samples are represented as a function of a fixed set of latent representations. The similarity function employed is cosine similarity, hinting at the fact that representations across distinct models are subject to angle preserving transformations. However, the choice of similarity function should not be limited only capturing invariances of one class of transformations. As shown in Cannistraci et al. (2224) Fumero et al. (02T]. other choices can be good as well, and there not a clear best choice among different transformations for capturing transformation across distinct latent spaces. We posit that when it is possible to relate distinct neural models' representational spaces, neural models are learning distinct parametrizations of the same underlying manifold (see Figure [. In this paper, we employ geodesic distance in the latent space for relative representations. This approach ensures that the relative space remains approximately invariant to the isometries and reparametrization of the data's 's manifold, as characterized by a Riemannian structure. Our contributions can be summarized as follows:
We observe that distinct neural models learn parametrization of the same underlying manifold when trained on similar data.
We propose a new representation that captures the isometric transformation between data manifolds learned by distinct models, by leveraging the pullback metric.
We propose to employ a scalable approximation of the geodesic energy to compute intrinsic distances that preserve the ranks of true distances.
We show how to get meaningful pullback metrics from discriminative models, such classifiers and sulf-supervised models.
We test relative geodesics on retrieval and stitching tasks autoencoders and vision foundation models, across different models, training schemes, and modalities, outperforming prior methods.
2 Related Work
Representation alignment. Numerous studies have shown that neural networks trained under different initializations, architectures, or objectives learn highly similar internal representations
Moschella et al. 2023 Bonheme and Grzes 2022 Kornblith et al. 2019 Cannistraci et al., 2024| Li
al. 2015 Bengio et al. 2014 Maiorca et al. [024 Huh et al. 2024 Guth al.
2024 Chang et al.
2022 Conneau et al.) 2018 T.sssulii et al. 2020 Nejatbakhsh et al.
[0224] This correspondence
becomes stronger wide and large networks [Barannikov al. [022]] Morcos al. 2018 Somepalli
ctt al. [022]. Leveraging these aligned embeddings. a simple linear transformation often suffices to map one network's latent space onto another's, s. enabling techniques such as model stitching, where components from different networks can be interchanged with minimal loss in performance [Fumero al. 2024 Bansal et al. 2021 Csiszárik et al.) 202T). In practice, aligning latent spaces using a linear transformation achieves comparable downstream task performance (Moayeri et al. 2023
Merullo et al. 2023] Maiorea et al., 2024 Lähner and Moeller 2024].
Latent space geometry. Early work on the geometry of deep latent representations focused on autoencoders, where the decoder's mapping from latent data space induces natural pullback metric under the assumption that the ambient space is Fuclidean [Shao et al. 2018 Tosi et al. [014]]
Arvanitidis et al. 2018]. The Riemannian viewpoint allows one compute geodesic paths and meaningful distances that respect the manifold structure of the learned embedding. Subsequent research has introduced computationally efficient approximations, such as energy-based proxies, and extended these ideas to estimate local curvature for improved interpolation and sampling [Chen
al. [2019 Chadebec and Allassonnière 2022, [oaiza--aamn Ganem et al., (2024] AAranitidis et al.) 2021
2022a In the context of discriminative models, one can obtain a Riemannian metric primarily using
two approaches Grosse 2022], either by pulling back the Fisher Information Matrix (Amari) 2016
Arvanitidis et al. 2022b or by assuming a Fuclidean geometry on the output space and pulling back
the 7.2 metric. Interestingly, one can obtain some identifiabiliyy guarantees by taking geometry into
consideration [Syrota et al.) 2025

---

3 Method
3.1 Notation and background
Neural networks (NNs) are parametric functions Fe, composed of encoding map and a decoding map, represented as Fg Da, o Eg,. The encoder Es, X' + Z generates a latent representation Eg, (2z), where E X' is mapped from the input domain X' to the latent space Z. The decoder De, is responsible for performing the task at hand, such as reconstruction or classification. For simplicity, we omit the parameter dependence (0) our notation moving forward. For any single module E (or equivalently D), we use Ex to denote that the module E was trained on the domain A'. In the next sections, we will provide the necessary background to introduce our method.
Latent space communication. Given a pair of domains (X, X"), a pair of neural models trained them (F) F2x) and partial correspondence between the domains I Ax Axx' where Ax C X and Ax" C X², the problem of latent space communication is the one of finding a full correspondence A (x)) + (X)" between the domains, from I. In simplified setting, for example two models trained with different initialization or architectures on the same data, X' X" and the correspondence is the identity. When X' f X' the problem recovers the multimodal setting.
Relative representations. The relative representations framework [Moschella et al. 2023] provides straightforward approach to represent each sample in the latent space according to its similarity to a set of fixed training samples, denoted as anchors. Representing samples in the latent space as a function of the anchors corresponds to transitioning from an absolute coordinate frame into relative one defined by the anchors and the similarity function. Given a domain X' an encoding function Ex X + 2, set of anchors Ax C X, and similarity or distance function d 2 x Z - R, the relative representation for sample æ E X is:
AR(z;:x;:)
+ Ex(a)) a.EAx
where x = (x(x)). and + denotes row-wise concatenation. In the original method [Moschella et al. 2023). d corresponds cosine similarity. This choice induces representation invariant to angle-preserving transformations. In this work, our focus to leverage the intrinsic geometry of latent spaces to employ a metric that captures isometric transformations between data manifolds.
Latent space geometry. For the latent space of a neural network, generally hard to reason about its Riemannian structure. However, it is often easier to assign Riemannian structure to the output space. As such, one can define a pullback metric from the output space to the latent space, which is standard operation in Riemannian geometry (see Ch.2.4 4 of Do Carmo and Flaherty Francis (1922)
Formally, the decoder D Z y' takes as input a latent representation a Riemannian metric defined on as Gy))), one can obtain the Riemannian metric at as:
and outputs y. Given
Oy Oz
Qy 0=
Gz(z)
Gy(v)
JD(z) [)D())]GGy())dDbt> (y()))D(().
where (2) is the Jacobian of at z. The metric tensor Gy is useful to compute quantities such as lengths, angles, and areas on M. Given a smooth curve 7 [a, + M, its are length is defined as:
L(>) ((7)= - [" V /u(t) (y(T())))))tt)e
where v(I) -()). A slight variation of the above functional gives the geodesic energy of y Arvanitidis et al. 2018 Shao et al.] 2018
1 l" 8()) 1 /. v(t) Gy((())() dt.
Both can be discretized and approximated in practice using finite difference approaches (Yang et al. 2018 Shao et al. 2018). Geodesics minimize both the length and the energy, where for optimization the latter usually preferred for numerical stability (Hauberg. 2025). These quantities have the property of being invariant to certain reparametrizations, formalized in the following proposition:
3

---

Proposition 3.1. Let [0, M be smooth curve on Riemannian manifold (M, G), and let (M', G') be a reparameterization of the manifold and ip [0.1] [0. a smooth, strictly increasing reparametrization of Setting (' ((p(T)) the Riemannian length and energy of " are invariant under reparameterizatiion of the manifold:
El>] 2 u
1
dry' lla dr
1 /
do''
llar dt, dr
4h7 I.
lI dr = S(
A de
do' di
Ilay dt.
Furthermore, the Riemannian are length of y invariant under reparametrizations: y' on M:
L = / dr dr
1 11 ae lla dt = L(7].
lla dr =
We provide the proof in Appendix A.1.1
3.2
Relative geodesics representations
Algorithm 1 Relative Geodesic Representations
Require: Sample E X, anchors Axx, encoder E, decoder D, distance dx induced by metric Gy, steps N, step size Åt, mode E (energy, distance)
Ensure: RRR900 (2; Ax)
1: 2 - E(x), RR2900 []
2: for E Ax do
3:
Za f- E(a),
d+0
4:
for j 1 to N do
5:
Y; + (1 { ) N
Y/-1 + (1-2 1-1 ) N 2
6:
7:
v - D(7,) ) - -(9)-1)
8:
G G+ (D(())))
9:
8 v Gv
10:
de d At (energy + 2, distance = Vz)
end for
12: Append d to Rigeo
end for
14:
return RiRgeo
From differential geometry perspective, the problem of latent space communication can be inter- preted as finding a transformation between the data manifolds M1 approximated by two neural models Fx F The relative representation framework captures this transformation implicitly if equipped with the right metric: we posit that a natural candidate for this metric is the geodesic distance defined on Mi, M2, respectively. This choice makes the relative representations invariant to isometric transformation 7' of the manifolds M2, However, for high-dimensional problems, the high cost of computing the geodesic (corresponding minimizing Eq. 2 makes this impractical [Shao et al.) 2018 Chen et al.] 2019]. Furthermore, one can argue against directly using the latent geometry induced by deterministic models from theoretical perspective [Hauberg 2019], as may result in undesirable properties, for example the geodesics going outside of the data manifold.
We therefore approximate the geodesic quantities by directly considering the energy (or the length) of the straight line (in the Euclidean sense) connecting representations in the latent space:
R.Reco (z; Ax)
E(Ha(z, (x(a))))
a,EAx
where (21, = (1 (1-a)2 az2 is the convex combination between the points 21, Z2. The approximation gives natural upper bound the geodesic distance: for 7 can be shown to relate to the are length of a curve defined in Eq. and the energy in Eq. 2g.22in using the following bounds:

---

Approximate energies
Geodesic energies
Approximate energies
Geodesic energies
(a) MNIST
(b) CIFAR-10
Figure 2: Pairwise latent-space energy matrices for (a) MNIST and (b) CIFAR-10. each subfigure, the left heatmap shows the straight-line energy approximation and the right shows the geodesic en- ergies of the ground truth geodesic curve. The Spearman rank correlations between the two measures are P = 0.99 for MNIST and 1.00 for CIFAR-10, demonstrating near-perfect agreements.
d(z0. 21)² (2) S 2E(
The proof is in Appendix Moreover the approximation is far more efficient compute, without requiring minimization Equation 2 and is accurate, as empirically verified in Figure
Discretization. When the step size is sufficiently small, the energy and are length in the latent space, as defined Equations [ [ can be approximated by their counterparts in the output space using discretized finite difference schemes [Shao et al. 2018)
N
N
E(7)
Σ
E
G(())((4))
(4)
2
i=1
t=
L(7) Σ
d
v(t)) ((L))())))
(5)
where At= N. with N being the number of discretization steps. For Euclidean geometry, the geodesic are lengths are given closed form as the geodesics are straight lines. Unlike the energy, the curve length is invariant under reparametrizations (proposition 3.Tp. As such we focus on the curve length in our experiments. The resulting algorithm is summarized in Algorithm with specific choice of Gy one avoid approximating the distance between D(7),, 75-1) explicitly using Gy by directly calculating the distance or energy between Y and 7;-1 on )
[ In practice,
Approximate geodesic energies. Our choice comes with three advantages: (i) efficiency: avoiding minimization of Eq computation for every sample reduces to a single forward pass for every discretization step 7 and for each anchor, resulting in overall complexity of O A) forward passes of the decoder, where the number of anchors is the number discretization steps. (ii) Directly using the are length ensures invariance to reparametrizations of the manifold, matching our assumptions. (iā) As we only need reasonably accurate estimates of the are lengths rather than the geodesic trajectory, the approach is accurate. Specifically, to assess how close the straight line energy approximation (2) is to the true geodesic energies, we first encoded 100 samples (10 per class, sorted by label) from MNIST [Deng] 2012 and CIFAR-10 using simple convolutional autoencoder (architecture detailed Appendix A.3.2). We then computed pairwise geodesic energy matrices these latent representations using both methods, and the results displayed in Fig. 2 Visually, both energy matrices exhibit the same block- block-diagonal structure, mainly due to belonging to the same class, and clustering patterns. Numerically, their Spearman rank correlation exceeds 0.99 with only 8 discretization points (see Appendix A.3.5 for correlation results across different numbers of discretization steps and for implementation details).
3.3 Choice of pullback metric
The properties of the relative geodesic representations are determined by (i) the choice of the output space, (ii) the choice of the metric to pullback from the output space and (iii) the pretraining objective (c.g. reconstruction classification) on which the decoder trained.
5

---

Generative models. For models trained on a reconstruction loss such as autoencoders, or on generative objectives, such as variational autoencoders Kingma and Welling 2013), pulling back metrics such as I,2 distance have been shown to effectively reflect the underlying geometry of the latent space [Tosi et al. 2014 Arvanitidis et al. 2018 Hauberg 2019).
Discriminative models. For discriminative models, such as classifiers or instance based discriminative models [Ibrahim et al. 2024]. is not immediate how to assign a Riemannian structure the space of latent representations. From the perspective of information geometry, perhaps the most natural choice the Fisher information matrix (Amari 2016). in which case the metric in the output space can be obtained as the one with categorical likelihood. However, neural networks typically experience Neural Collapse [Kothapalli] [023], possibly rendering the resulting geometry troublesome, We empirically inspect this approach in Appendix [A.4.II] In this work we consider two principled approaches for discriminative models based on classification decoder heads and instance discrimination heads.
Pulling back from classifiers. Perhaps the most natural idea is, as discussed in Section [3.1 to construct a pullback metric based on the model's outputs, by simply pulling back the Euclidean L2 metric from the logit space of the classifier. Given an arbitrary encoder model, we train a classification head upon the latent representations (extracted e.g. from the last layer) and pulling back the Euclidean L2 metric from the output logits of the head. The resulting relative geodesics representation will inherit properties of both the decoder head (up to class information) and the pretrained encoder.
Pulling back from instance discrimination decoders. Diet [lbaaii et al. 2024| is sulf-super training method which has been shown to learn representations with strong generalization to down- stream tasks, and yield identifiability guarantees [Reizinger et al., 2025). Specifically, in the infinite data limit representations from the Diet objective align the cluster centers of von-Mises Fisher (vMF) distributions, which lie on unit sphere. The loss based on simple instance discrimination is:
exp (w f(x)) E, exp (w) f(x))
D = Exn
log
where W is a linear projection and f is a nonlinear map. Further, assigning the same instance label to data augmentations was shown beneficial to improve invariance. Originally proposed to train the entire neural network [lbrahim et al.] 2024), we instead use it to learn a decoder D on top of the pretrained neural network latent representations, by setting D f o W. To construct relative geodesic representations, we propose to pullback the spherical metric from the penultimate layer of the diet decoder, before W. Further discussions on Diet can be found in Appendix A.22I
Both classifier and instance discriminator approaches discussed above use proper pullback metrics and fall under our proposed framework of relative geodesic representations: these representations inherit semantic information, up to class or instance level, from the decoder, while retaining structure of the pretrained encoder. Notably, when the encoder is pretrained, the relative geodesics representations are computationally efficient, since the decoder remains lightweight and the approximate geodesic energy computation is cheap. As we demonstrate in the following experimental sections, this approach yields meaningful and identifiable representations that are consistent across models.
4 Experiments
In the following we will evaluate the performance of relative geodesic representations on latent com- munication problems across models with different initializations, architectures, sizes and modalities.
Tasks description. We evaluate our approach on two representative instantiations of the latent communication problem: retrieval and neural stitching. In a retrieval setting we aim solve the latent communication problem up to the instance level. Given pairs of model encoders (E)r, E²²) and access to their latent representations, we seek to recover a full correspondence A starting from a partial one Т. For neural stitching the goal is to solve the latent communication problem up to the task-label level. Classical stitching approaches train an adapter V between intermediate components of distinct neural networks so that D2 o V E remains functional on downstream task (e.g.. classification). Section 4.2 we operate in the zero-shot stitching regime [Moschella et al., 2023), where no adapter is trained explicitly. Instead, we solve implicitly for the transformations between representations by mapping them into relative representation spaces. This enables stitching pairs of models without any fine-tuning or additional supervision.

---

In Section 4.1 we evaluate relative geodesic representations on generative models, focusing on au- toencoders. This analysis examines performance across networks trained with different initializations and datasets. In Section 4.2 we extend the evaluation to discriminative foundation models, assessing performance at scale across diverse architectures, pretraining objectives (e.g., self-supervised and classification), datasets, and modalities.
4.1
Experimental evaluation on autoencoders
In the following sections, we evaluate relative geodesic representations on the latent communication problem across autoencoder models trained with different initializations, architectures and datasets.
4.1.1
Aligning independently trained neural representational spaces
1444 ozcreon mede
N4A orede Peirer
VAR enteo on oibe
Rditive Costre Retativ Seeduax
Relatve Codlrd Relatee
Pelative Codra
Condens
Onticrrall
Mamber Af
Number Inthoor
Mumber AfAnctore
130
Figure 3: Aligning latent spaces of culooenoders:: MRR score function of the number of anchors on pairs of autoencoders trained with different initializations on the MNIST (left), FashionMNIS (center), CIFAR10 (right) datasets, respectively. In green, we plot the performance Moschell et al.) [2023]; in red and orange the linear and orthogonal baselines respectively; in blue, our method. The shaded area indicates standard deviation across 5 different random sets of anchors. Relative geodesic consistently outperforms baselines, obtaining peak performance.
Setting. For the following experiment, we trained pairs of convolutional autoencoders (Fi, F22) with different initializations on MNIST [Deng, 2012], FashionMNIST [Xiao et al. 2017). CIFAR10 Krizhevsky) 2009] datasets. The architecture of the convolutional autoencoder is detailed in Ap- pendix A...2 Alter training, we extracted 10k samples from the test set, and mapped them to the latent spaces of the two models, to representations Z₁ E, (X) Z₂ Ex(X) respectively. Starting from a small set of anchors in correspondence I Ax + Ay. the objective is to evaluate how well possible to recover the full correspondence A between the representations Z1, Zg from the relative representations. As a baseline, we compare with relative representations using cosine similarity MMoschela et al. 2023). and with fitting a linear orthogonal mapping using r.
Analysis of results. Fig. 3 plots the performance terms of MRR on MNIST, FashionMMIST and CIFAR10 datasets. To obtain the score, first compute similarity matrices between relative
RR(Z, )² RR(Z.),
representations of the two spaces as D(Z,, Z2) where Di. = [RR R Then we compute the Mean Reciprocal Rank (MRR, see Appendix A.3.II I top of the similarity matrix. In the figure, we plot MRR as a function a random set anchors, where the shaded areas indicate standard deviations over 5 different sets of random anchors with the same cardinality. Our method consistently performs better than Moschella et al. 2023), saturating the score with few anchors on all the domains, despite the different degrees of complexity of the latent spaces. In addition, our method shows significantly less variance, being more robust to the choice of the anchor set.
Takeaway. Relative geodesic representation near-perfectly captures transformations between repre- sentational spaces of models initialized differently, sample efficiency and robustness.
4.1.2 Stitching autoencoder models
Setting. We consider the same pairs of autoencoders trained on MNIST, FashionMNIST CIFAR10 datasets of Section 4.1.1 Starting from a set of five random anchors, we estimate transformation between the model representational spaces Z1, Zz. this experiment, to keep differently from Moschella al. 2023], in which zero-shot stitching achieved by training once decoder
7

---

9
9
7
7
9
3
3
8
3
3
4
4
4
4
4
S
S
8 S
8
8
O
5
8
Figure 4: Stitching on Autoencoders: We visualize qualitative reconstructions of samples, stitching autoencoders of models trained with different initializations on MNIST (left), FashionMNIST (center), CIFAR10 (right). The first two columns show reconstructions from the original models; middle three columns represent baselines |Maorraa et al., [024 Moschella et al. [023]; I; the rightmost column is our method. Relative geodesic yields the best stitching results using just 5 anchors.
Table 1: Average MRR cosine results for differrnnt methods across different datasets. Relative repre- sentations pulling back from diet decoder (RelGeo (Diet)) consistently provides better retrievals.
Method
CIFAR-10
CIFAR-100
ImageNet-Ik
CUB
SVHN
Rel (Coaine) Moschella al. 2023
0.129
0.135
0.166
0.162
0.221 0.178
0.135 0.148
0.068 0.08
RelGea (1.2)
0.047 + 0.013
0.112 + 0.031
0.412 + 0.09
0.28 + 0.129
0.025 + 0.012
(al
0.387 + 0.145
0.445 + 0.142
0.566 0.111
0.523 + 0.177
0.314 + 0.188
module with relative representations and then exchanging different encoder modules, here we achieve stitching without training any decoder. We compute relative representations with respect to the set of anchors, and compute similarity matrix DZZ,, Z2). Then we compute the vector с = arg max, (D) representing a correspondence between the two representation matrices Z1, Zz. and use to fit a linear transformation I to approximate the transformation between the two domains. We perform stitching by performing the following operation for a sample x E X: = D2 T o (rr).
Analysis of results. We visualize the results of reconstructions of random samples in Fig. 4 comparing against Mosschela et al. [2023], Lähner and Moeller (2024), , Maiorca al.) [2024). For each dataset, each column represents respectively: (i) the original autoencoding mapping for a sample x of model Fi, (E( (ii) D2 (x)),, (iii) the mapping Da (E (z)), (iv) the mapping Dy(Tanchors E1 (x)) where Tanchors is estimated on the five available anchors, (v) the mapping Dy(Teosine E1 (I)) where Teosine is estimated among all 10k samples with the correspondence e obtaining in the relative space of Moschella et al. [2023], (vi) Our result (2(Treigeo (x)), where T Trogge estimated from the correspondence obtained in the relative geodesic space. As shown in Fig. 4 while the baselines do not reach a good enough reconstruction quality, reconstructions with our method are almost perfect in accordance with the results in Fig. 3
Takeaway, Relative geodesics enable stitching of neural modules trained with different initializations.
4.2 Experiments on vision foundation models
In this section, we evaluate relative geodesic representations' performances retrieval and model stitching tasks on vision foundation discriminative models across models pretrained with different objectives, architectures, sizes and modalities.
4.2.1 Matching representational spaces of discriminative foundation models
In this section, we test the compatibilities of representations of vision foundation models with different architectures, such residual networks [He et al. 2016 and vision transformers (Dosovitskiy et al.) 2021]. and with different pretraining objectives including classification and self-supervised learning.
Setting. We perform experiments on retrieval tasks on pretrained vision foundation models, inves- tigating how well we can match representations together with different backbones subject to the decoding tasks, on 5 datasets, varying in complexity and size: CIFAR10. CIFAR100 (Krizhevsky] 2009], SVHN Yuuval Netzer et al., (0TT), CUB (Wah et al. 2023), and ImageNet-11 (Russakovsky For ImageNet- 1k, we used 1000 anchors, while for other datasets we used 500. As
et al. 2015
8

---

backbones we consider ResNet-50 [He et al. 2016], Vision Transformers (ViT) Dosovisskiy et al. 202T] with both patch 16-224 and patch 32-384, and DINOv2 [Oquab et al.) 2024 We compare the original formulation of relative representations with cosine similarity [Moschella et al. 2023 denoted as Rel (Cosine), relative geodesic representations pulling back from Fuclidean logits denoted as RelGeo (T.2), and pulling back the spherical metric using Diet decoder denoted as RelGeo (Diet).
Analysis of results. Table [ shows results from different methods averaged across all possible pairs of models on the considered datasets. Additionally, Fig. shows the results on CUB. While RelGeo (L2) may result worse MRR numbers, RelGeo (Diet) provides consistently improved retrieval performance. In Appendix A..41 we report full results for the datasets.
Takeaway. Relative geodesic representations pulling back from instance discrimination decoders identifiable across vision foundation models, improving retrieval performances.
Rel(Ces
ReiGeo(L21
Rel (Diiet)
Rosteot.0 5g
V0:32
DaNoy2
NasNwt-50 50
WT
32
DINOV2
Aaster 50
WT kg
VI7 32
DINON2
Sesbeid
0.52
0.22
0.44
0.22
RenNeed0
0.61
0.31
0.33
0.34
Re-Nel-10
0.54
0.27
0.28
0.28
VT-16-
0.41
0.76
VIT-16-
0.49
0.79
0.67
VIP-6-
0.58
MTT
0:43
0.58
0.51
VIT
0.78
DINOV2
0.64
DINOx2
DINOx
0.84
Rel(Casine)
ReIGeo(L2)
Rel Geo(eet)))
Rusteb 56
W7:26
NarNeb-g 50
VAT 16
WT: 32
ResNeb-S00 50
V7 16
DINCUU
Reshiet
1.00
0.04
0.08
0.02
Reshict500
1.00
0.20
0.21
0.16
ResNett50
1.00
0.44
0.46
0.32
0.04
1.00
0.45
0.14
0.20
1.00
0.55
0.27
0.44
1.50
0.59
0.48
0.21
1.00
1.00
0.55
Figure 5: CUB Accuracies (top) and symmetricized MRR cosine (bottom). RelGeo (Diet) and especially RelGeo (I.2) provide strong stitching accuracies, while RelGeo (Diet) maintains strong instance identifiability.
4.2.2 Zero-shot stitching of vision foundation models
Table Average stitching performances across different settings. RelGeo (L2) often outperforms Rel (Cosine), while RelGeo (Diet) remains competitive.
Method
CIFAR-10
CIFAR-100
ImageNet-Ik
CUB
SVHN
Rel (Cosine) RelGeo (1.2) (1.2)
Moschella et
2023
0.907 + 0.09
0.775 : 0.132
0.549 0.152
0.531 + 0.188
0.384 0.115
0.955 0.03
0.874 + 0.055
0.501 + 0.159
0.595 + 0.163
0.59 + 0.054
MelGeo(Diet)
0.915 0.074
0.775 0.115
0.479 +: 0.17
0.559 + 0.171
0.416 + 0.079
Model stitching was introduced in Lenc and Vedaldi 2015 to analyze neural network representational spaces, by training a linear layer to connect different layers and evaluating performance, Here we sidestep the need for trainable stitching layers and consider the zero-shot model stitching task defined in Moschella et al. 2023) to effectively test how components of vision foundation models can be reused. To do this, we leverage the space of relative geodesic representations
shared compatible space. For the ith model E,, we train one decoder D; on the relative representations induced by it, then evaluate the performance of using D; to decode the representations of model E,, where E, may be a different model. This assesses how much two representation spaces can be merged with respect to the task defined by the decoder D, e.g., classification head.
Setting. We perform experiments on pretrained vision foundation models from Hugging Face Transformers [Wolf et al. 2020), investigating how well we can match representations together for

---

classification with different backbones with classification heads, on the same datasets and models as considered in Section 4.2.1 similarly comparing Rel (Cosine), RelGeo (L2) and RelGeo (Diet).
Analysis of results. The results of the different methods across the different datasets are shown in T where we average over all possible model pairs. We further show the accuracies of the models on the CUB dataset in Fig. 5 Both RelGeo (L2) and RelGeo (Diet) provide strong stitching accuracies, with RelGeo (L2) reflecting the benefits of pulling back class specific information. RelGeo (Diet) still results in good accuracies while having very strong MRR metrics, as shown in [
Takeaway. Relative geodesic representations yield good accuracies and good MRRs, avoiding downgrading of performance when performing model stitching while retaining sample identifiability.
4.2.3 Matching different modalities
In this section we evaluate relative geodesic representations in the multimodal setting.
Setting. We study the retrieval task terms of vision foundation models and the text encoders of CLIP [Radford et al. 202T] with both patch 16 and patch 32, using Flick30kk dataset (Young) al., 2014). Keeping the text encoder of CLIP fixed, we swap the vision encoder with the ones of ResNet-55, DINOv2 and ViT, including dif- ferent patch and model sizes. Due to the lack of class labels, RelGeo ( (L2) is not applicable, and we compare RelGeo (Diet) with Rel (Cosine), While we observed that using data augmentations is beneficial for RelGeo (Diet), due to the lack of a principled approach to construct data aug-
0.12
0.09
0.08
0.05
AlliOaine
0.04
2.09
0.02
0.02
RasNe833
VI126
VIT32
DRO02
Matching multimodal models. Symmetricized MRR cosine on Flickr30k. RelGeo(Diet) substantially improves upon Rel(Cosine) in aligning multimodal models.
Figure 6:
mentations on texts corresponding to image augmentations, we do not employ augmentations.
Analysis of results. The results in terms of symmetricized MRR cosine metric with CLIP with patch 16 are shown in Figure We observe that RelGeo(Diet) yields significantly improved stitching performances upon Rel(Cos, In Appendix A.4.9 we show the full pairwise matrices MRR, comprising of the unimodal performances, inter vision models, and text models.
Takeaway. Relative geodesic representations show promising results for obtaining identifiable representations in multimodal scenarios.
5 Conclusions and discussion
We have introduced the framework of relative geodesic representation starting from the assumption that distinct neural models trained on similar data distributions learn to approximate the same underlying latent manifold. As result, geodesic distances based on their representations are invariant to transformations between different representational spaces. We show that the geodesic energy and are length of straight lines provide an efficient, low-cost metric for bridging these spaces, allowing us to measure similarity and align representations across different architectures, training objectives, and out
training procedures, while outperforming previous methods.
Limitations and future work. The accuracy of approximating geodesics using straight-line length (or energy) can deteriorate in regions of high curvature in the latent space, typically corresponding to areas far from the support of the training data. Moreover, this could require increasingly smaller step sizes, hurting the efficiency performance of the method. This suggests exploring nonlinear paths, and adaptive step sizes, e.g., by estimating the support of the data building KNN graphs in the latent space and forcing the path to not deviate too much from them. By employing the pullback metric from given output space, the relative geodesic representation has the interesting property of restricting the alignment problem to the information relevant to the decoding task. This could be useful to (i) further explore no training multi-modi alignment [Norelli et al. 2023]. where is of interest to capture not only the shared information across modalities, but also the modality- specific information; (īi) to better understand the relation between the representation similarity and decodability (Harvey al. 2024] and the interaction between tasks and learned representations
Fumerr al. 2023].
10

---

Acknowledgments and Disclosure of Funding
We thank Gregor Krzmanc, German Magai, Vital Fernandez for insightful discussions in the early stages of the project. HY was supported by the Research Council of Finland Flagship programme: Finnish Center for Artificial Intelligence FCAL HY wishes to acknowledge CSC IT Center for Science, Finland, for computational resources. GA was supported by the DFF Sapere Aude Starting Grant "GADL". SH was supported by a research grant (42062) from VILLUM FONDEN and partly funded by the Novo Nordisk Foundation through the Center for Basic Research in Life Science (NNF200C0062606), SH received funding from the European Research Council (ERC) under the European Union's Horizon Programme (grant agreement 101 125003). MF is supported by the MSCA IST-Bridge fellowship which has received funding from the European Union's 's Horizon 2020 research and innovation program under the Marie Skfodowska-Cune grant agreement No 101034413.
References
S.-i. Amari. Information Geometry and Its Applications. Springer Tokyo, edition, 2016.
G. Arvanitidis, L. K. Hansen, and S. Hauberg. Latent Space Oddity: on the Curvature of Deep Generative Models. In International Conference on Learning Representations, 2018. URL https://openreviiet
G. Arvanitidis, S. Hauberg, and B. Schölkopf. Geometrically Enriched Latent Spaces. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2021.
G. Arvanitidis, Georgiev, and Schölkopf. A prior-based approximate latent Riemannian metric. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2022a.
G. Arvanitidis, M. González-Duque, Pouplin, D. Kalatzis, and S. Hauberg. Pulling back informa- tion geometry. In G. Camps- Valls, F. J. R. Ruiz, and I. Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages
PMLR, Mar. 2022b.
Y. Bansal, Nakkiran, and B. Barak. Revisiting model stitching to compare neural represen- tations. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances Neural Information Processing Systems, volume 34, pages 225-236. Curran As- sociates, Inc., 2021. http:/tttss///ccccpeiiiieee Paaaer.pdfff... fflee//eee/Sss000fe07000:099eeee
neurips..... cc/paper..ppp..... files/ www.ff/aaeee///eeeeaaeaa pappr/2021.1
S. Barannikov, I. Trofimov, N. Balabin, and E. Burnaev. Representation topology divergence: A method for comparing neural network representations. In Proceedings the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1607-1626. PMLR, 17-23 Jul 2022.
Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives, 2014. BRLLacppe//aaxvvooor//////Do 5538
L. Bonheme and M. Grzes. How do variational autoencodrrr learn? insights from representational similarity, 2022. URL
08399
B. C. Brown, A. I.. Caterini, B. I.. Ross, J. C. Cresswell, and G. Loaiza-Ganem. Verifying the Union of Manifolds Hypothesis for Image Data. In The Eleventh International Conference on Learning Representations, 2023, URL
I. Cannistraci, I.. Moschella, M. Fumero, V. Maiorea, and E. Rodolà, From bricks bridges: Product of invariances to enhance latent space communication. ICLR, 2024.
C. Chadebec and S. Allassonnière. geometric perspective on variational autoencoders, 2022. URI. https://arxiv.. wttp://..rg/a/b////e/eeee/e
07370
A. Chang, Z. Tu, and B. K. Bergen. The geometry of multilingual language model representations, 2022. URL arxiv org/ abs / 2205. 10964

---

N. Chen, F. Ferroni, A. Klushyn, A. Paraschos, J. Bayer, and P. van der Smagt. Fast approximate geodesics for deep generative models. In Artificial Neural Networks and Machine Learning- ICANN 2019: Deep Learning: 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17-19, 2019, Proceedings, Part II 28, pages 554-666. Springer, 2019.
A. Conneau, G. Lample, M. Ranzato, L. Denoyer, and H. Jégou. Word translation without parallel data, 2018. hRtphts:://axxi http:p////arri.ooor////////11 04087
A. Csiszárik, P. Kórösi-Szabó, Åkos K. Matszangosz, G. Papp, and D. Varga. Similarity and matching of neural network representations, 2021. URL https: littps://arrs arxiv. www.iioog//.....oooooooooooooolo wwt/oog//////ae/e/ aa/aaaaa/11111 14633
L. Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2916)114 2012.
N. S. Detlefsen, A. Pouplin, C. W. Feldager, C. Geng, D. Kalatzis, H. Hauschultz,
M. Gonzälez-Duque, F. Warburg, M. Miani, and S. Hauberg. Stochman. GitHub. Note:
https::/gittub..oom/// www.aaaooea/aaaaaaaaaaaaaa wttp://chiinn.aaoionoaaaeeeeoooo
2021.
M. P. Do Carmo and Flaherty Francis. Riemannian geometry, volume 2. Springer, 1992.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning hett:/oorurrideeccccCFTTTT
Representations, 2021. URL
M. Fumero, L. Cosmo, S. Melzi, and E. Rodolà. Learning disentangled representations via product manifold projection. In International conference on machine learning, pages 3530-3540. PMLR, 2021.
M. Fumero, Wenzel, L. Zancato, A. Achille, E. Rodolà, S. Soatto, B. Schölkopf, and F. Locatello. Leveraging sparse and shared feature activations for disentangled representation learning. Advances in Neural Information Processing Systems, 36:27682- 27698, 2023.
M. Fumero, M. Pegoraro, V. Maiorca, F. Locatello, and E. Rodolà. Latent functional maps: spectral framework for representation alignment. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neu- ral Information Processing Systems, volume 37, pages 66178-66203 Curran Associates, hewp://cceediinnnneeeeeeeeeeeeee neurips.....e 2h0p::/6554110000000000055000002 Cow.eeeeeerooooeeeee....o pdf
Inc., 2024. URL
R. Grosse. Chapter 3: Metrics,,,, 2022. URL URL http://www....oo/ www.ttooottooooo edu/ redbu//rrrrosscouusses
csc2541_ 0t1p541_222//reaiing//0....... htt_1/000eeeeiii___ee pdf
F. Guth, B. Ménard, G. Rochette, and S. Mallat, A rainbow in deep network black boxes, 2024. URL httts://arxiv.com wwwps/.oorrbbbs////00/ 18512
S. F. Harvey, Lipshutz, and A. H. Williams. What representational similarity measures imply about decodable information. arXiv preprint 2rX0v:2411.08197, 2024.
S. Hauberg. Only Bayes should learn a manifold (on the estimation of differential geometric structure from data), 2019. _eprint: 1806.04994.
S. Hauberg. Differential geometry for generative modeling. DTU, 2025.
J. V. Haxby, M. I. Gobbini, M. L. Furey, A. Ishai, J. L. Schouten, and P. Pictrini. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293(5539): 2425-2430, 2001.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
M. Huh, B. Cheung, T. Wang, and Isola. The platonic representation hypothesis, 2024. URI attps://arxiv.. org/ abs/2405. 07987
12

---

A. Hyvärinen, I. Khemakhem, and H. Morioka. Nonlinear independent component analysis for principled disentanglement in unsupervised deep learning. Patterns, 4(10):100844, 2023. ISSN 2666-3899.
M. Ibrahim, D. Klindt, and R. Balestriero. Occam's 's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations? 2024. URL https://arxiv wwtp:rogg//a//aeeeee wtggabbb/440..177// _eprint: 2406.10743.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv org/abs/14122 6980
D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In International conference machine learning, pages 3519-3529. PMLR, 2019.
V. Kothapalli. Neural Collapse: A Review on Modelling Principles and Generalization. Transactions on Machine Learning Research, 2023.
A. Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, pages 32-33, 2009. URL httpt:tpp////. //tp::://wwwwcc cs. toronto. www.oot..dd//rriii/aaaaiiga///o. heapi///eeammin--eeeteeeee 2009 TR pdf
A. Laakso and G. Cottrell. Content and cluster analysis: Assessing representational similarity in neural systems. Philosophical Psychology, 13:47 76, 2000.
Z. Lähner and M. Moeller. On the direct alignment of latent spaces. In Proceedings of UniReps: the First Workshop on Unifying Representations in Neural Models, volume 243 of Proceedings of Machine Learning Research, pages 158-169. PMLR, 2024. URL httpt::/ppocceeddiiieieo .mr
http///2///1aae224..aa. html
K. Lenc and A. Vedaldi. Understanding image representations by measuring their equivariance and equivalence, 2015. URL //axxi/ www./ogg////aaeeeeeeoooeee 1111. 5908
Q Lhoest, A. Villanova del Moral, P. von Platen, T. Wolf, M. $aško, Y. Jernite, A. Thakur, L. Tun- stall, S. Patil, M. Drame, Chaumond, J. Plu, Davison, S. Brandeis, Sanh, Le Scao, P. Schmid, S. Gugger, N. Raw, S. Lesage, A. Lozhkov, M. Carrigan, T. Matussière, L. von Werra, L. Debut, S. Bekman, and C. Delangue. Datascts: A Community Library for Natural Language Processing, In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184. Association for Computational Linguistics, Nov. 2021.
K. Canwen Xu, N. Patry. S. Liu, A. McMillan-Maute
Y. Li, J. Yosinski, J. Clune, H. Lipson, and J. Hopcroft. Convergent learning: Do different neural networks learn the same representations? arXiv preprint 101.0:151,.07543, 2015.
G. Loaiza-Ganem, B. L. Ross, R. Hosseinzadeh, A. L. Caterini, and J. C. Cresswell. Deep generative models through the lens of the manifold hypothesis: A survey and new connections, 2024. URL htttp:://axxx: org abs/ /404. 02954
V. Maiorca, L. Moschella, A. Norelli, M. Fumero, F. Locatello, and E. Rodolà. Latent space translation via semantic alignment. Advances in Neural Information Processing Systems, 36, 2024.
J. Merullo, L. Castricato, C. Fickhoff, and E. Pavlick. Linearly mapping from image to text space, 2023. URL https://arxiv, org/ 22009 15162
M. Moayeri, K. Rezaei, M. Sanjabi, and S. Feizi. Text-to-concept (and back) via cross alignment. In International Conference on Machine Learning, pages 25037-25060. PMLR, 2023.
A. S. Morcos, M. Raghu, and S. Bengio. Insights on representationall similarity neural networks with canonical correlation, 2018. URL
05759
L. Moschella, V. Maiorca, M. Fumero, A. Norelli, F. Locatello, and E, Rodolà. Relative representa- tions enable zero-shot latent space communication. In The Eleventh International Conference on Learning Representations, 2023. /RU/nttps//ooppereiiee... /ttp://tr/////oooomeeeoo/o//o//r -nwiii
13

---

A. Nejatbakhsh, V. Geadah, A. H. Williams, and D. Lipshutz. Comparing noisy neural population dynamics using optimal transport distances. arXiv preprint arXiv:2412.14421, 2024.
B. M. G. Nielsen, E. Marconato, A. Dittadi, and L. Gresele. When Does Closeness Distribution Imply Representational Similarity? An Identifiability Perspective, 2025. URL https://axxiv aataaaa/22500 03784 _eprint: 2506.03784.
A. Norelli, M. Fumero, V. Maiorca, L. Moschella, E. Rodolà, and F. Locatello, Asif: Coupled data turns unimodal models multimodal without training, 2023. URL wwl.ittts//// /tatpp://arxiv... www.gorgga.......
01738
M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. HAZIZA, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Mísra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL [RRLsttps///ppereviiet wwwp:rreiiiie.nn//rrrrnn. id a6
R. Pascanu, C. Lyle, I.-V Modoranu, N. E. Borras, D. Alistarh, P. Velickovic, S. Chandar, S. De, and Martens. Optimizers Qualitatively Alter Solutions And We Should Leverage This, 2025. URL https://arxis:
Begg/bbs/5500 12224 _eprint: 2507.12224.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkín, Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PMLR, 2021.
P. Reizinger, A. Bizeul, A. Juhos, J. E. Vogt, R. Balestriero, W. Brendel, and D. Klindt. Cross-Entropy Is All You Need To Invert the Data Generating Process, In The Thirteenth International Conference on Learning Representations, 2025. URL /tpp://////oe
G. Roeder, L. Metz, and D. Kingma. On Linear Identifiability of Learned Representations. In Pro- ceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9030-9039, PMLR, July 2021.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recog- nition Challenge. International Journal of Computer Vision (IJCV), 115(3):211 252,, 2015. doi: 10. 1200070/11220--0--001116---
H. Shao, A. Kumar, and T. Fletcher. The Riemannian Geometry of Deep Generative Models. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 428 4288, 2018. doi: 10. 1099/VPPR.2001 20..0007..
G. Somepalli, L. Fowl, A. Bansal, Yeh-Chiang, Y. Dar, R. Baraniuk, M. Goldblum, and T. Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective, 2022. htt.:httt///axrtte org/ 2203 08124
S. Syrota, Y. Zainchkovskyy. J. Xi, B. Bloem-Reddy, and S. Hauberg. Identifying metric structures of deep latent variable models. In A. Singh, M. Fazel, D. Hsu. S. Lacoste-Julien, Berkenkamp. T. Maharaj. K. Wagstaff, and J. Zhu, editors, Proceedings of the 42nd international conference on machine learning, volume 267 of Proceedings of machine learning research, pages 58065-58083 PMLR, July 2025. URL
mlr. wttpsiinw...l...rrs/eeeeeaaeaae/ atress/v267/syrota255 html
A. Tosi, S. Hauberg, A. Vellido, and N. D. Lawrence. Metrics for Probabilistic Geometries. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI' 14, pages 800 808, Arlington, Virginia, USA, 2014. AUAI Press. event-place: Quebec City, Quebee, Canada.
A. Tsitsulin, M. Munkhoeva, D. Mottin, P. Karras, A. Bronstein, I. Oscledets, and E. Müller, The shape of data: Intrinsic distance for data distributions, 2020. URL httpsst//aaxiiiii org/abs/ 1905. 11141
C. Wah, S. Branson, P. Welinder, Perona, and Belongie. The caltech-uesd birds-200-22. dataset, Aug, 2023.
14

---

T. Wolf. L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: State- of-the-Art Natural Language Processing. In ACI., pages 38 45, Association for Computational Linguisties, Oct. 2020.
H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
T. Yang. G. Arvanitidis, D. Fu, X. Li, and S. Hauberg. Geodesic clustering in deep generative models. In arXiv preprint, 2018.
P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denota- tions: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014. doi: 10.1 (0.1162/tacl_ a 00166. URL
https://aclatthology Seapenhhooooooooog//11. 0og/ 1006/
Place: Cambridge, MA Publisher: MIT Press.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. In Dataset, 2011.
15

---

A Appendix
A.1 Proof of theoretical results
A.1.1 Proof of Proposition 3.1
Proof. We first prove the first half, i.e. the invariance of Riemannian curve length and energy across reparameterizations the manifold. This can be proven by observing that the inner product at point along the curve is invariant across such reparameterizations:
|| = G(=) .' G' (''')aa''
du at) da' dn
G² G'('')
da' dæ x! de
|| alar
As such, the length the energy of the same curves on different manifolds are integrals of the same quantities, hence are equal.
We then prove the second half, i.c. the invariance of Riemannian curve length across reparameteriza-
tions of the curve. Based on Equation 4.7 from Hauberg 2025], we have
dr 1/1 1 G
s(t) dt
lla di L/ dr lla
dr s0() dr
di
t
[.(7].
A.1.2 Proof of Equation 3
Proof. We first prove d(zo, 21)² L2()), then prove (2( 2E(7).
For the first part, according to the definition of geodesic distance, we have d(zo: 21) S L(7) and, as such, d(z0, 21)² (7))
The second part involves (7) 28(7).
which can be proven using the inequauaaa inequality, di and vú See Equation 7.14 in Hauberg 2025 where we denote u. or
-V
I 12 dt =V V.. 11 1
T2((9)
d5 dt (u, v) |u |||||||||
de
=2E(-))
A.2
Additional explanations
A.2.1
Details on Diet
Proposed self-supervised learning method [lbrahim et al. 2024]. Diet was also shown to yicld interesting identifiable guarantees [Reizinger et al.) [025], laying the theoretical foundation for RelGeo (Diet), where we employ the resulting geometry.
One can consider such scenario [Reizinger et al.) [2025]: some latent variables are drawn from vMF distribution, and pushed forward through continuous and injective generator function g to obtain the data æ. Remarkably, given only ae without the knowledge of g, is possible to (to some degree) recover the latent variables z through parameterizing model and optimizing the instance discrimination loss as given in Equation [ Specifically, suppose there is finite set of vectors n, on unit sphere, each representing class, and finite set of instances. One instance belongs to
16

---

exactly one class, and every class is employed by some instance. Additionally, the instance labels are chosen uniformly, and the latent variables x are drawn from a vMF distribution centered around the corresponding cluster vector with concentration parameter K.
Then, after the model is trained using the loss function in Equation 6 when both f and not unit-normalized, f g is linear. This can be proven rigorously by expanding upon the theoretical framework of non-linear ICA [Hyvärinen et al. 2023 As such, we propose utilize o g to form the representations, For further technical details on the assumptions and additional results, we refer interested readers to Reizinger et al. 2025
Assuming spherical geometry, the distance between two points æ and can be computed as
d(æ, y) arccos
|||||| |
In the above formula, points that do not precisely lie on the unit sphere are effectively projected onto it. Interestingly this bears a strong resemblance the cosine distances used in the original paper on relative representations [Moschella et al. 2023).
A.2.2 Why it works
Prior work representational alignment has shown that representations from different models can often be approximately aligned using simple transformations, c.g. linear, orthogonal or locally linear maps. Eyen when models are trained independently. with differrent architectures, modalities, or datasets that nonetheless share an underlying structure - they tend to learn similar representations, suggesting convergence towards shared encoding of entities (Huh et al. 2024). From theoretical standpoint, identifiability results [Roeder et al., [222]] imply that if two discriminative models learn the same likelihood function, their internal representations must be equivalent up linear transformation. However, this ideal scenario rarely holds exactly in practice: training dynamics, nuisance factors, and unmodeled variability can all introduce distortion. In our case, it may be too strong to assume that two models learn different parameterizations of an identical manifold. Instead, we adopt a weaker assumption, that they do so up to some bounded distortion. Recent theoretical work has begun to explore relaxations of strict identifiability to account for such bounded distortions [Nielsen et al.) 2025]- Integrating these relaxations into our Riemannian framework presents a promising direction for future work.
In general, the few theoretical results available, e.g. [Roeder al. 2021 often rely on unrealistic assumptions, e.g. proofs in axiomatic settings, infinite-data regimes, or the requirement that two models learn exactly the same likelihood function. In our view, meaningful first step toward bringing theory and practice is to relax these assumptions, as initiated in [Nielsen et al. 20255] and begin to model more realistic scenarios, e.g. including the dynamics introduced by model training.
We believe that Riemannian methods can play a key role in this direction to try to capture local alignments beyond linear global transformations of the space as considered in RReder et al. (2021) and possibly accounting for distortions measured in the linear space in practice. Nevertheless, we remark that neural networks could find qualitatively different solutions (Pascanu al.) (025) and that the union of manifolds hypothesis might be more appropriate for modeling image data [Brown et al.
2023
A.3
Additional details
A.3.1 Mean Reciprocal Rank
Mean Reciprocal Rank (MRR) a commonly used metric to evaluate the performance of retrieval systems, and has been used to evaluate the capabilities of representations for instance discrimination MMschella et al. 2023]. It measures the effectiveness of a system by calculating the rank of the first relevant item in the search results for each query.
To compute MRR, we consider the following steps:
1. For each query, rank the list of retrieved items based on their relevance to the query.
2. Determine the rank position of the first relevant item in the list. If the first relevant item for query i is found at rank position r,, then the reciprocal rank for that query is 1.
17

---

3. Calculate the mean of the reciprocal ranks over all queries. If there are Q queries, the MRR is given by:
MRR = 1 Q Q
-54
Here, T; rank position of the first relevant item for i-th query. a query has no relevant items in the retrieved list, its reciprocal rank is considered to be zero.
MRR provides single metric that reflects the average performance of the retrieval system, with higher MRR values indicating better performance.
Similar to stitching accuracies, MRR generally asymmetric. However, can also made symmetric. Specifically, MRR is calculated based distance matrix D, one can make the distance matrix symmetric by setting D = (D + D). In Section 4.2.1 we reported the symmetric version. Otherwise report both the original version and the symmetric version, and discriminate between these two by explicitly indicating when it is symmetric.
A.3.2
Architectural details
We provide in Table B the architectural details of the convolutional autoencoders employed in experiments in Figures and
Table 3: Architecture of the convolutional autoencoders.
Encoder
3 x 3 conv. 32 stride 2-ReLu
3 x 3 conv. 64 stride 2-ReLu Flatten
(64 * k)
h Linear
Latents
Decoder
h x (64 + l k) Linear Unflatten
x conv. stride 2-ReLu
3 x 3 conv. 32 stride R-RLLu Sigmoid
For the classifier experiment, in order to obtain geometric representations we need decoder. The architecture is shown in Table For RelGco(Diet), the last linear layer is configured with bias=False in accordance with the original algorithm.
For evaluating the performances of the representations, we train classification head with the same architecture as used by Moschella et al. (2023) as given in Table
Table 5: Architecture of the decoders for eval- uations.
Table 4: Architecture of the simple decoders.
Classification head
Final classification head
impul dim LayerNorm imput dim 500 Linear-Tand 500 x num_clcoo num_classes Linear
input_div dim LayerNorm
input_dim _imm_ddim_m_ x input_ddi _diim_iimmi_ Lincar-Tami Tanh
InstanceNormld
inpul dimn x reurn classes Linear
A.3.3 RelGeo(Diet) augmentations
As noted by |bbahim et al. [2024], it is beneficial to employ data augmentations when using Diet to perform self-supervised training of neural networks. We largely follow their approach, and considered different levels of data augmentations. Following Ibrahim et al. (2024] consider different levels
18

---

of data augmentations indexed by scalar strength, which are summarized below using Py PyTorch pseudocode; strengths of a higher level employs the augmentations of lower levels well.
O: No augmentations;
1: RundommResizeedCCop(hhhchhh width)), Random/orizontalPiin)::
2: RandomAppiyyCCClo//tttte004 0.4, 0.4, 0.2)), p=0.3); RandomGrayscclle(0222)
3: RandomAApiyyGruusiinBlur(( 3), (1.0, 2.0)), p=0.2), RandomEFrasing(0.55)
A.3.4 Compute resources
Experiments regarding the geodesic approximation conducted using NVIDIA A100 GPU and 12 CPU cores. Run time varies depending on the discretization steps, number of anchors and the used dataset.
The autoencoder stitching and retrieval experiments were conducted on a single NVIDIA RTX 3080TI GPU. Experiments involving vision foundation models were run on compute cluster, each job using a single NVIDIA A100 GPU and 10 CPU cores, with runtimes of several hours. Preliminary experiments required additional resources, and in total estimate having used several hundred GPU hours.
Further ablation studies on the running times can be found in Section A.4.10
A.3.5 Geodesic approximation
Here, we provide the experimental details of the results presented in Fig. 2 and Fig. geodesic energies, used small autoencoder, whose architecture is presented in Table 6
assess the
Autoencoder training We trained a lightweight convolutional autoencoder (see Table on both MNIST CIFAR-I0 to obtain the latent representations used our experiments. For MNIST, first convolutional layer was adjusted accept single input channel; for CIFAR-10 it used three channels. Each model was trained for 30 epochs using the Adam optimizer [Kingma and Ba 2017 with batch size of 64. We set the learning rate to 0,001, and fixed random seed of 42 to ensure reproducibility.
Energy computation After training, we selected 10 samples per class (100 total) in label order from each dataset and encoded them to produce their latent encodings. True geodesics are computed using Stochman library [Detlefsen et al., 2021], which has Apache-2.0 license, which wraps the decoder into pullback manifold, intializes parameterized spline path between codes, and then optimizes its parameters to minimize the Riemannian energy. Geodesic energies are computed as in Eq. Pairwise energies are computed and visualized in Figures anddd] 7 demonstrating the close agreement between the two measures under identical encoding and discretization settings. In Fig.
2 latent dimensions for MNIST and CIFAR are 64 and 128 respectively. while Fig. 7 latent dimension is 2 for both datasets.
Table ConvAutoeneoder architecture (latent dim d).
Encoder
Activation
Conv2d(1, 32, kernel 3, stride=2, pad=l)
ReLU
Cony2d(32, 64, kernel 3, stride=2, pad=1)
ReLU
Flatten
Lincar(644747.
Decoder
Activation
Linear(d, 64*7*7)
ReLU
(nflatten(64.7.77
Conv Transpose2d 32, kernel = 3, stride=2, pad=1, out pad=la Conv (322, kernel 3, stride=2, pad=1, out_pad=l)
ReLU
Sigmoid
19

---

Approximate energies
Geodesic energies
Approximate energies
Geodesic energies
(a) MNIST
(b) CIFAR-II 10
Figure 7: Pairwise latent-space energy matrices for (a) MNIST and (b) CIFAR-10, 10, with latent dimensionality 2. each subfigure, the left heatmap shows the straight-line energy proxy and the right shows the full Riemannian geodesic energies. The Spearman rank correlation between the two measures 0.99 for MNIST and = 1.00 for CIFAR-10. 10, demonstrating pear-perfect agreement.
lanue Dnstrace
11000
(a) MNIST
Facieazx
10002
4.:00
344713
250033
200C4
1100
34873
20851
37803
(b) CIFAR-10
Figure 8: Impact of varying discretization levels on similarity and energy metrics for MNIST and (b) CIFAR-10 10 datasets. Each subplot shows how Spearman's 's Pearson's 's r, and Euclidean distance change the number of discretization levels increases.
A.3.6 Autoencoder stitching and retrieval
We provide the experimental details of the results presented in Figure and Figure 4 All models employed followed the architecture depicted in Table with a latent dimensionality of 128.
We trained the lightweight convolutional autoencoder (see Table 6 on MNIST, CIFAR-IO. FashionM- NIST with different seeds, to obtain the latent representations used in our experiments. For MNIST and MaSSionMNIST the first convolutional layer was adjusted to accept a single input channel; for CIFAR- 10 used three channels. Each model was trained for 50 epochs, reaching convergence, using the Adam optimizer (Kingma and Ba 2017 with batch size of 64. We set the learning rate to 0.001.
A.3.7 Vision foundation models
We use the pretrained models as provided by Huggingface Transformers (Wolf et al. 2020). which
has Apache-2.0 license, and the datasets as provided by HuggingFace Datasets [Lhoest et al. [022]] which also has Apache-2.0 license. The license information of the datasets are: CIFAR-10 10: unknown; CIFAR-100: unknown; CUB; unknown; ImageNet-Ik, ImageNet agreement; SVHN: non-commercial use only.
Unless otherwise stated, directly the original test set of the dataset the test set, while using 0.9 of the original train set as the train set and the remaining as the validation set. Both the anchors and the Diet data points are selected from the validation
For CIFAR-10M 100, we use the coarse labels. For SVHN, the objective is to predict the cropped digits. For CUB dataset,
the version available at wettttt://// /ttgg/gg////gggaaaaeeee. co/datasess/

---

Table 7: Aggregated results of MRR CDist Sym.
Method
CIFAR-10
CIFAR-100
ImageNet-1k
SVHN
Rel (Coaine) RalGeo (1.2
[Msschella 2023
0.098 0.133
0.122 0.161
0.103 4: 0.146
0.046 0.055
0.046 = 0.081
0.046 + 0.013
0.105 + 0.031
0.179 + 0.173
0.187 0.141
0.04 + 0.021
RelGeo (Diet)
0.252 0.189
0.278 + 0.211
0.402 0.148
0.433 = 0.212
0.306 + 0.188
Table 8: Aggregated results of MRR Cosine.
Method
CIFAR-10
CIFAR-100
ImageNet-1k
CUB
SVHN
Rel (Casine) Moschella al. 2023
0.08 0.077
0.122
0.109
0.21 0.149
0.089 0.094
0.035 + 0.034
RelGeo (1.2)
0.019 + 0.005
0.046 : 0.016
0.236 1: 0.08
0.156 + 0.089
0.013 = 0.005
(Diet)
0.117
0.358
0.184
2at___2000221111111 2012200111DS Given the relatively small training set, select 2000 points as the validation set. When reporting aggregated MRR metrics in the tables, we always exclude the diagonal entries as these are generally (close to) 1. For ImageNet- 1k, we use the validation set and split it into the final train, val and test sets. Further details can be found in the provided code.
Pender-project/CUUe CCCCCCUBUOOUU0
For all cases where need train classification heads, apart from the ones with Diet the heads trained for 10 epochs, while the ones with Diet are trained for 50 epochs. The heads used to obtain the gometric information are trained using learning rate 5e 4 and batch size 64, while the heads used for stitching was trained using learning rate 1e - 4 and batch size 32. We always use the Adam optimizer [Kigmma and Ba 2017).
When reporting stitching results, we train three classification heads and average the accuracies as the final results.
A.4 Additional results on Vision Foundation models
We provide additional results vision foundation models. For ablation studies, focus the performances of the models on CUB dataset. We refer to accuracy as Accuracy, symmetricized MRR based on cosine MRR Cosine Sym, symmetricized MRR based on edist MRR CDist Sym, MRR based on cosine as MRR Cosine and MRR based on edist as MRR CDist.
A.4.1 Full results
We provide the heatmaps on the different datasets in Figure Figure 10 Figure
Figure 11
Figure 12 12n and
A.4.2
Other evaluation metrics
We provide the results of other evaluation metrics in Table Table S
and Table 9
A.4.3
Alternative aggregation
Here we consider an alternative way to aggregate the results, i.e. grouping by the models. The results are reported in Table I0 Table [T] Table [2] Table 13 and Table [4| In general, the observation remains: RelGeo(L2) yields good accuracies and RelGeo(Dict) yields good MRRs.
Table 9: Aggregated results of MRR CDist.
Method
CIFAR-10
CIFAR-100
ImageNet-1k
CUB
SVHN
Rel (Cosina) [Msschela al. 2023
0.051 0.072
0.071 0.107
0.075 + 0.105
0.023 0.02
0.02 0.032
RelGea (1.2)
0.019 1 0.005
0.04 + 0.015
0.106 1: 0.11
0.108 1: 0.092
0.012 : 0.005
RelGeo (Diet)
0.127 0.118
0.151 + 0.138
0.298 0.141
0.269 0.195
0.123 0.103
21

---

Rel(Cosine)
ReiGeo(L2)
Rel (Diiet)
Realitt55
MF:16
M1232
VIT-18
M7--2
DRNOY2
Resher-50
W7-16
V17-22
DINOa
Seshet
6.71
RcsNet-50
0.91
ResNett-0
0.17
6.76
VI1-1-
0.85
0.57
0.66
0.94
ViI-16
0.97
0.97
0.97
0.97
Vil-66
0.95
0.97
0.95
0.95
VTT32-
0.94
0.94
0.6
0.03
VIT32-
0.96
0.96
0.97
0.96
VIT.2-
0.94
0.55
0.96
0.94
DINOV2
0.96
0.95
0.5
0.08
DINOV2
0.96
0.98
0.98
0.98
DINO22
0.95
0.95
0.96
0.95
Rel(Casine)
ReIGeo(L2)
Rel Geo(Diot))
Nosthel 30
WP:18
V7-32
Nor
DRNOy2
Maxter-
W7 kg
V77
DHNOV2
1.00
0.02
0.02
0.01
1.00
0.04
0.05
0.03
Iew/Net-50-
1.00
0.24
0.30
0.25
VIT-1-
0.02
1.00
0.39
0.18
VIT-16
0.04
1.00
0.07
9.06
VIF-66
0.24
1.00
0.66
0.43
VI732
0.02
0.39
1.00
0:34
VIT32
0.05
D.07
1.00
2.04
VIT-22
0.30
0.56
1.00
0.44
DINOV2
0.01
0.18
0.14
1.00
DINOx2
0.03
0.05
0.04
1.00
DINOx2
0:25
0.41
0.44
1.00
Rel(Cae
ReIGio(L2)
Rel Geo(Diet)
Rosteet.0
V17-1
V7-
VIT 11
V17
72-32
Rester
V17 -1
N7 1g
DHNcky2
Reskett5D
1.00
0.01
0.11
0.01
Restiet-50
1.00
0.04
0.05
9.03
ResMet-50
1.00
0.08
0.05
0.15
VTT16-
0.01
1.00
0.38
0.32
VIT
0.04
1.00
0.07
2.04
VIT
D.00
1.00
0.63
0.30
VIT.22
0.01
0.18
1.00
0.07
VIT.22
0.05
0.07
1.00
3.04
VIT-32
0.00
0.67
1.00
0.27
DINDV2
0.01
0.12
0.07
1.00
DINOv2
0.03
0.04
0.04
1.00
DINOx2
0:15
0.30
0.27
1.00
Rel(Casine)
ReiGao(12)
Rel Geo(Diet))
Rustred 50
VIT-22
DRIOv2
No0-50
VTT-16
VYT
H7-32
MenMer-00
V77 18
VIT. 22
DINO02
Reshet 50
1.00
0.01
0.01
0.0L
Rcshect0
1.00
0.02
0.02
0.01
RooNet 50
1.00
0.12
0.14
0.12
VIT-16
2.92
1.00
821
0.07
VIT:16
0.01
1.00
0.03
9.02
VIl-16
D:07
1.00
0.59
0.21
VST-32
0.03
0.22
1.00
0.0G
Vil-32
9.02
0.03
1.00
0.02
Vil-32
0.09
0.41
1.00
0.22
DINOV2
0.04
0.17
0.12
1.00
DINO22
0.01
0.02
0.02
1.00
DINO22
0.21
0.21
1.00
Rel(Cosine)
ReiGeo(L2)
Rel Geo(Diet)
Ars
Rester-50
V1:26
VIT-16
M7-32
W7-16
W7-32
Reshet
1.00
0.01
B.11
0.01
50
1.00
0.02
0.22
5.02
ResMitt 50
1.00
0.25
0.04
0.09
VIT-18
0.01
1.00
0.19
6.05
VIT-18
0.07
1.00
0.21
3.00
Vil-16
0.11
1.00
0.37
0.12
0.01
0.3
1.00
0.03
VIT-32
0.02
0.03
1.00
9.02
VIT-32
0.02
0.38
1.00
0.11
0INOV2
0.01
0.04
0.02
1.00
DINOx2
0.01
0.02
0.02
1.00
DINOx2
0.04
0.15
0.12
1.00
Figure 9: Results on CIFAR-10. From top to bottom: Accuracy, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist
Table 10: Alternatively aggregated results of Accuracy.
Method
ResNet-50
VIT-16
VIT-32
DINOv2
Rel (Cosine) Moschella et al. 2023
0.507 +
0.669 + 0.229
0.664 0.218
0.678 0.24
RelGeo (1.2)
0.646 + 0.209
0.709 + 0.208
0.72 + 0.194
0.737 + 0.209
RelGeo (Diet)
0.529 + 0.194
0.658 + 0.229
0.661 +: 0.219
0.668 = 0.237
22

---

Rel(Cosine)
ReiGeo(L2)
Rel (Dieet)
M2-32
VIT-18
M7--2
ORNOV2
Resher-pp
W7-16
V77-22
DINOk2
Seshet
0.77
0.51
RcsNet-50
0.82
5.76
ResNett-0
VI1-1-
0.79
0.51
0.85
0.82
Vil-16
0.50
0.92
0.90
0.89
Vil-26
0.80
0.50
0.83
0.81
VIT-2-
0.76
0.82
0.8
0.77
VII-12
0.88
0.88
0.90
0.58
VI--2
0.77
0.51
0.88
0.79
DINDV2
0.83
0.85
0.84
0.34
DINO
0.92
0.92
0.92
0.94
DINOV2
0.81
0.52
0.63
0.93
Rel(Cosi
ReIGeo(L2)
Rel Geo(Diet)
Nostrot 30
W7:18
V7-32
Mor 50
W7 32
DkwOy2
Maxteer
W7 kk
V9
.2
DHNOV2
1.00
0.03
0.03
0.02
1.00
0.09
0.21
0.08
Rtw/Net-50
1.00
0.28
0.33
0.35
VIT-18
0.03
1.00
0.47
0.24
VIT-1
0.09
1.00
0.18
9.11
VIF-36
0.28
1.00
0.69
0.51
VI732
0.03
0.47
1.00
0.21
VIT32-
0.11
0:18
1.00
5.30
VIT-22
0:33
0.59
1.00
0.51
DINOV2
0.02
0.21
0.21
1.00
DINOx2
0.08
0.11
0.30
1.00
DINOx2
0:35
0.51
0.51
1.00
Rel(Casine)
ReIGeo(L2)
Rel (eo((et))
Mostlet.00 50
WF-16
Mor
WT 16
-2
MesNet-S0
M7 1g
V7. 1g
DWNcg2
Reskett5D
1.00
0.01
0.01
0.01
Restiet-50
1.00
0.09
0.22
0.07
ResMet-50 -
1.00
0.57
0.07
0.15
VIT16-
0.02
1.00
0.47
0.15
VIT
0.09
1.00
0.37
0.09
VIT
0.07
1.00
0.66
0.38
VIT.22
0.01
0.67
1.40
0.30
VIT.32
0.12
0:17
1.00
3.04
VIT-32
0.07
0.64
1.00
0.M4
DINDV2
0.01
0.15
0.10
1.00
DINOv2
0.07
0.09
0.09
1.00
DINOx2
0:15
0.38
0.34
1.00
Rel(Casine)
ReiGao(12)
Rel Geo(Diet))
Rosthen
Wn:118
VI-32
Noh
VTT.
MenNen-50
V7:
1.00
0.01
0.01
0.0L
Restbet-50-
1.00
0.04
0.05
0.02
RewNee-50-
1.00
0.15
0.18
0.20
VST-16
2.07
1.00
0.30
0.09
VIT:16
0.05
1.00
0.08
9.05
Vil-16
D.09
1.00
0.45
0.25
VST-32
0.06
0.34
1.00
0.00
ViT-32
0.04
0.08
1.00
2.04
Vil-12
0:22
0.16
1.00
0.29
DINOV2
0.08
0.2
0.21
1.00
DINO22
0.03
0.04
0.04
1.00
DINO22
0.12
0.28
0.28
1.00
Rel(Cosine)
ReiGeo(L2)
Rel Geo(Diet)
V1:26
VIT-16
V7-32
DinOv2
W7-16
M7-30
DINGA
5o
1.00
0.01
0.01
0.01
50
1.00
0.03
0.24
5.02
ResMitt 50
1.00
0.25
0.05
0.11
VIT.66
0.01
1.00
827
0.07
VIT
0.03
1.00
0.07
3.01
VIT.16
0.22
1.00
6.41
0.16
VST-32
0.01
0.34
1.00
0.05
VIT-22
0.04
0.07
1.00
0.03
VIl-32
0.02
0.43
1.00
0.14
DINDv2
0.01
0.06
0.03
1.00
DINO
0.03
0.04
0.03
1.00
DINOx2
0.04
0.20
0.17
1.00
Figure 10: Results on CIFAR-100. From top to bottom: Accuracy, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist
Table 11: Alternatively aggregated results of MRR Cosine Sym.
Method
ResNet-50
VIT-16
VIT-32
DINOv2
Rel (Cosine) Moschella al. 2023
0.032 - 0.023
0.212 + 0.173
0.208 + 0.175
0.124 + 0.104
RelGeo (L.2)
0.143 0.132
0.197
0.186
0.205 + 0.189
0.154 + 0.137
RelGeo (Diet)
0.336 + 0.143
0.506 + 0.2
0.526 4: 0.187
0.42 - 0.166
23

---

Rel(Casine)
ReiGeo(L2)
Rel (Dieet)
Realist55
M1-32
VIT-18
M7--2
DRNON2
Resher-50
W7-16
V17-22
DINOa
So
6.11
RcsNet500
ResNett-0
0:35
VI1-19
0.55
0.78
0.60
0.41
Vil-16
0.42
0.77
0.48
0.37
Vil-26
0.41
0.78
0.48
0.32
VTT.2-
0.53
0.55
0.77
0.36
VIT32-
0.43
0.47
0.77
0.38
VIT 32
0.42
0.46
0.76
0.32
DINOV2
0.57
0.51
0.54
0.79
DINOV2
0.42
0.29
0.41
0.78
DINO22
0.43
0.39
0.41
0.77
Rel(Casine)
ReiGao(L2)
Rel Gen(let))))
Nosthel 30
WP:18
V7-32
Nor
DlwOy2
Maxter-
V7 kk
DHNOV2
1.00
0.09
0.05
0.03
Bestlet--50 -
1.00
0:35
0.40
0.38
Rew/Net-50
1.00
0.54
0.59
0.92
VIT-1-
0.09
1.00
052
0.30
VIT-166
0.35
1.00
0.61
9.36
VIF-66
0:54
1.00
0.50
0.45
VI732
0.05
0.52
1.00
0.53
VIT32
0.40
2.61
1.00
9.97
VIT322
0:3M
0.50
1.00
0.46
DINOV2
0.01
0.30
0.33
1.00
DINOV2
0:36
0:36
0.37
1.00
DINOx2
0.52
0.48
0.45
1.00
Rel(Cae
ReIGio(L2)
Rel Geo(Diet)
Rosteet.0
V1-16
V7.
VIT 11
V17
-2-32
Rester
V17 11
N7 1g
DiNCk
Reskett5D
1.00
0.01
0.11
0.01
Resliet-50
1.00
0.02
0.02
9.92
ResMet-50
1.00
0.39
0.44
0.91
VTT16-
0.01
1.00
8.42
0.07
VIT
0.02
1.00
D.42
2.28
VIT
0:34
1.00
0.78
0.44
VIT.22
0.01
0.42
1.00
0.10
VIT.22
0.02
0.47
1.00
5.28
VIT-32
0.44
0.78
1.00
0.42
DINDV2
0.01
0.07
0.10
1.00
DINOv2
0.02
0.28
0.28
1.00
DINOx2
0.31
0.44
0.42
1.00
Rel(Casine)
ReiGao(12)
Rel Geo(Diet)
Rustred 50
VIT-22
DRIOv2
No0-50
W7-26
VTT
77-32
DRRO22
MenNer-50
V17 16
VIT
DWNO02
Reshet 50
1.00
0.02
0.01
0.0L
Rcshect0
1.00
0.20
0.22
0.23
RooNet 50
1.00
0.34
0.37
0.36
VIT-16
1.00
0.31
0.11
VIT:16
0.17
1.00
0.39
9.20
Vil-16
9.29
1.00
0.50
0.92
VST-32
0.16
0.41
1.00
0.13
ViT-32
0.21
0.43
1.00
0.22
Vil-32
0.34
0.64
1.00
0.32
DINOV2
0.31
0.42
0.36
1.00
DINO22
0.19
0.19
0.19
1.00
DINO22
0.26
0.22
1.00
Rel(Cosine)
ReiGeo(L2)
Rel Geo(Die)))
V1:26
Ars
VIT-16
M7-32
Rester-50
W7-16
W7-32
1.00
0.01
B.11
0.01
50
1.00
0.01
0.01
2.01
ResMitt 50
1.00
0.28
0.50
0:25
VIT-18
0.01
1.00
825
6.12
VIT-18
0.01
1.00
0.24
5:15
VIl-16
0:14
1.00
0.56
0.11
0.01
0.32
1.00
0.15
VIT-32
0.01
0.32
1.00
0.16
VIT-32
0.22
0.61
1.00
0.30
DINOV2
0.01
0.02
0.03
1.00
DINOx2
0.01
0.15
0.13
1.00
DINOx2
0.12
0.22
0.19
1.00
Figure 11: Results on ImageNet-Ik, 1k. From top to bottom: Accuracy, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist
Table 12: Alternatively aggregated results of MRR CDist Sym.
Method
5es
VIT-16
VIT-32
DINOv2
Rel (Cosine) Moschella et al. 2023
0.009 + 0.005
0.141 + 0.156
0.134 0.158
0.019 0.017
RelGeo (1.2)
0.052 + 0.033
0.144 + 0.146
0.147 + 0.146
0.103 0.085
RelGeo (Diet)
0.204 0.13 0.13
0.432 - 0.235
0.437 0.232
0.313 +:2 0.108

---

Rel(Cosine)
ReiGeo(L2)
Rel (Dieet)
Realist.g 56
W1-16
V1--2
VIT 11
M7-32
DRNON2
Resher 30
1g
32
DINOgd
0.52
6.22
Reslbet-50
0:11
3.M4
RewNet-50
0.28
0.28
V7-16-
0.41
0.56
VIT-16
9.49
0.79
0.67
9.59
VIF-6-
0.44
0.77
0.65
0.58
VTT32-
0.43
0.67
0.77
0.58
VIT32-
0.51
0.70
0.50
0.63
VIT 32
0.47
0.68
0.78
0.61
DINOV2
0.64
0.60
0.40
6.83
DINOV2
0.54
0.67
0.67
5.87
DINO22
0.51
0.63
0.65
0.84
Rel(Casine)
ReIGao(L2)
Rel Gen(lit)))
Nosteet 50
WF:18
V7-32
NorMesh 50
Maxtenn
V7 kg
DINOu
1.00
0.04
0.08
0.02
1.00
0.20
0.21
0.16
Rew/Net-50 -
1.00
0.44
0.46
0.32
VIT-1-
0.04
1.00
0.45
0.14
VIT-166
0.20
1.00
0.35
9.27
VIF-66
0.44
1.00
0.59
0.45
VST-22
0.05
0.45
1.00
0.00
VIT-22
0.21
0.55
1.00
9.90
Vil-22
9.46
0.59
1.00
0.55
DINDv2
0.07
0.14
0.09
1.00
DINO22
0.16
0.27
0.30
1.00
DINO22
D.32
0.48
0.55
1.00
Rel(Cosinel
ReiGea(L2)
Rell Geo(Diet)
Nestrol.50
V1T:16
MT-32
Not
VIT-16
VIT-32
Restett -0
VT-15
NT.
DiNcky2
Reskett5D
1.00
0.01
0.22
0.01
Restiet-50
1.00
9.07
0.08
0.11
ResMet-50
1.00
0.36
0.40
0.17
VIT16
0.01
1.00
0.16
0.06
VIT16
0.07
1.00
0.40
5:19
VIT
0:36
1.00
0.67
0.38
VIT.22
0.02
0.16
1.00
0.00
VIT.32
0.08
0.48
1.00
5.20
VIT-32
9.40
0.87
1.00
0.42
DINDV2
0.01
0.05
0.02
1.00
DINOv2
0.11
0.19
0.20
1.00
DINOx2
0.17
0.38
0.42
1.00
Rel(Casine)
ReiGao(12)
Rel Geo(Diet)
Resteet-5
V77-28
M1-32
DRNOv2
Neh-00
VIT 26
WT.
32
DNNOv2
Rester-00
N7. 16
M7. .7.33
DINN022
Reshet 50
1.00
0.01
8,22
0.01
Rcshect0
1.00
0.04
0.04
3:08
RosNet 50
1.00
0.22
0.24
0.16
VST-16
0.07
1.00
0.35
0.05
VIT:16
0.09
1.00
0.34
5.15
VIl-16
0.22
1.00
0.73
0.32
VST-32
0.10
0.20
1.00
0.03
Vil-32
0.10
0.34
1.00
0.17
Vil-32
0.23
0.70
1.00
0.38
DINOV2
0.04
0.19
0.08
1.00
DINO22
0.08
0.16
0.17
1.00
DINO22
0.15
0.26
0.32
1.00
Rel(Cos
ReiGeo(L2)
Rel Geo(Diet)
VIT-1
Om
Reater-30 30
N7. 28
MT.
DINO
Reshwi60
1.00
0.01
B.11
0.01
RasNet-00
1.00
0.04
0.24
2.06
Iew-Nll-30
1.00
0.18
0.20
0.09
VIT-18
0.01
1.00
0,55
0.05
VIT-18
0.03
1.00
0.31
3.04
Vil-16
0:18
1.00
0.71
0.21
0.01
0.06
1.00
0.02
VIT-32
0.03
0.29
1.00
9.30
VIT-32
0.20
0.67
1.00
0.25
DINOV2
0.91
DINOx2
0.04
DINOx2
1.00
Figure 12: Results on CUB. From top to bottom: Accuracy, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist
Table Alternatively aggregated results MRR Cosine.
Method
ResNet-50
ViT-16
VIT-32
DINOv2
Rel (Cosine) Moschella et al. 2023
0.011 + 0.005
0.138 + 0.11
0.133 0.112
0.147 0.128
RelGeo (L.))
0.074 + 0.077
0.107 + 0.118
0.116 0.126
0.079 0.074
RelGeo (Diet)
0.182 0.107
0.299 0.184
0.316 + 0.182
0.201 + 0.076
25

---

Rel(Cosine)
RelGeo(L2)
Rel (Diiet)
Realist.g 56
WF-16
M1--2
VIT 11
M7-32
DRNON2
Rester 30
DRNOa
32
0:10
Restbet-50
Resdeet-0
V7-16-
0.28
0.55
0.32
VIT-16
9.48
0.59
0.53
0.53
VIF-66
0.34
0.51
0.39
0.36
VTT
0.30
0.37
0.58
0.34
VIT.2-
0.55
0.59
0.63
0.56
VIT 32
0:25
0.40
0.54
0.39
DINOV2
0.21
0.29
0.29
0.55
DINOv2
0.54
0.59
0.58
5.64
DINO22
0:33
0.33
0.32
0.53
Rel(Casine)
ReIGao(L2)
Rel Gen(lit)))
Nosteet 56
WF:18
V7-32
NorMehh 50
Maxtenn
V7 kg
DINOu
1.00
0.01
0.01
0.02
1.00
0.02
0.02
0.02
Rew/Net-50
1.00
0.22
0.15
0.15
VIT-1-
0.01
1.00
0.24
0.06
VIT-16
0.02
1.00
0.05
9.02
VIF-66
0.12
1.00
0.63
0.41
VST-22
0.02
0.24
1.00
0.06
VIT-22
0.02
2.05
1.00
2.92
Vil-22
9:15
0.53
1.00
0.45
DINDv2
0.07
0.05
0.06
1.00
DINO22
0.02
0.02
0.02
1.00
DINO22
0:25
0.41
0.43
1.00
Rel(Cosinel
ReiGea(L2)
Reil Geo(Diet)
Nestrol-50
V1T:16
MT-32
Not
VIT-16
VIT-32
Restett -0
VT-15
NT.
DiNcky2
Reskett5D
1.00
0.01
0.11
0.01
Restiet-50
1.00
0.02
0.23
9.92
ResMet--0
1.00
0.21
0.14
0.14
VIT16
0.01
1.00
023
0.01
VIT16
0.02
1.00
0.08
2.05
VIT
D:21
1.00
0.62
0.41
VIT.22
0.01
821
1.00
0.00
VIT.32
0.01
0.08
1.00
5.05
VIT-32
0:14
0.62
1.00
0.42
DINDV2
0.01
0.01
0.02
1.00
DINOv2
0.02
0.05
0.05
1.00
DINOx2
0.14
0.41
0.42
1.00
Rel(Casine)
ReiGao(12)
Rel Geo(Diet)
Resteet-50
V17-28
M1-32
DRNOv2
Net-00
VIT
26
VT.
32
DNNDv2
Rester-00
N7. 16
MT. .2
DHNOV22
Reshet 50
1.00
0.01
0.11
0.01
Rcshect0
1.00
D.61
0.01
3.01
RooNet 50
1.00
0.34
0.04
0.05
VIT-16
0.01
1.00
0.09
0.06
VIT:16
0.01
1.00
0.02
9.01
VIl-16
0.03
1.00
0.34
0.16
VST-32
0.02
0.10
1.00
0.07
ViT-32
0.01
0.02
1.00
0.01
Vil-32
0.04
0.33
1.00
0.16
DINOV2
0.01
0.02
1.00
DINO22
0.01
0.01
0.02
1.00
DINO22
0.17
0.18
1.00
Rel(Cosine)
ReiGeo(L2)
Rel Geo(Diet)
VIT-16
Om
Reater-30 30
W7. k1
My.
DINO
Reshwi60
1.00
0.01
B.11
0.01
RasNet-50
1.00
0.01
0.01
2.01
Iew-Nll-30
1.00
0.23
0.04
0.04
VIT-18
0.01
1.00
0.10
0.01
VIT-1W
0.01
1.00
0.22
3.01
Vil-16
0.22
1.00
0.31
0.16
0.01
0.08
1.00
0.01
VIT-32
0.01
0.02
1.00
9.01
VIT-32
0.03
0.32
1.00
0.16
DINOV2
DINOx2
0.01
DINOx2
1.00
Figure 13: Results on SVHN. From top to bottom: Accuracy, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist
Table 14: Alternatively aggregated results of MRR CDist.
Method
ResNet-50
VIT-16
VIT-32
DINOv2
Rel (Cosine) Moschella et al. 2023
0.007 + 0.001
0.079 + 0.086
0.089
0.112
0.019 0.015
RelGeo (L.))
0.023 + 0.016
0.075 + 0.096
0.078
0.099
0.051 0.05
RelGeo (Diet)
0.121 0.094
0.253 + 0.193
0.258 + 0.194
0.143 + 0.065
26

---

A.4.4 Number of anchors
We investigate the impact of the number of anchors. The results are shown in Figure 14 Figure 15 The general conclusion that ReIGeo(L2) is good in terms of accuracies, RelGeo(Diet) is good in terms of MRRs persist with varying number of anchors.
A.4.5 Number of Diet points
We analyze the impact of the number of Diet points. The results are shown in Figure 16 The performances of Releeo(Diee) improve the number of diet points become larger.
A.4.6
Number of discretization steps
We analyze the impact of the number of discretization steps on ReIGieo(L.2) and RelGeo(Diiet) and provide the results in Figure[77] and Figure[T8] The performances do not vary much depending on the discretization steps, though using multiple steps seems to help.
A.4.7 Diet augmentation strengths
We analyze the impact of different data augmentation strengths on RelGeo(Diee). The results are shown in Figure 19 Similar to the observations in terms of self-supervised learning [lbrahim et al.. 2024). RelGeo(Dieg) (Diet) benefits from stronger data augmentations.
A.4.8 Anchor selection scheme
As discussed in Moschella et al. [2023], there are different ways of choosing the anchors. In the main paper, we consider the case where the anchors are selected uniformly at random, referred to as uniform. There are other choices as well, e.g. using farthest point sampling, referred as fps, and using anchors the data point close to the centroids of K-means clustering, referred to as kmeans.
Here we additionally report the results for fps and Since we need to align multiple models, in practice we use the selection mechanism to select a fixed number of anchors based on the representations of each model, and combine them while employing random subsampling obtain the final anchors of given number.
The experimental results for fps and kmeans are shown in Figure 20 and Figure respectively.
A.4.9 Multimodal
In the main paper we reported results based on MRR Cosine Sym. Below we show the full experi- mental results in Figure 22 and report aggregated results in Table 15
Table 15: Average MRR results for different methods across different datasets. Relative representa- tions pulling back from diet decoder (RelGeo (Diet)) consistently provides better retrievals.
Method
MRR Cosine Sym
MRR CDist Sym
MRR Cosine
MRR CDist
Rel (Cosine) RelGeo (Diet)
Moschella et al. 2023
0 + 0.395
0.293 + 0.389
0.283 + 0.382
0.252 - 0.373
0.413 0.353
0.384 = 0.359
0.317 0.372
0.302 0.378
A.4.10 Running times
We report running times for autoencoder experiments with an NVIDIA 3080 GPU and vision foundational model experiments with an NVIDIA A100 GPU.
Times to train the models For autoencoder experiments, training the autoencoders is not expensive, as it merely takes around 20 minutes on an RTX3080Ti for 50 epochs.
For vision foundation models, the employed pretrained backbones are typically computationally costly to train. For instance, Oquab et al. (2024) reported that training DINOv2 ViT-L/14 on ImagetNet-22k using 96 A A100-80GB GPUs takes approximately 3.3 days. In contrast, training the decoders much faster. We report the running times to train the decoders in Table I6 where for each setting
27

---

Kel(Cesine)-50
Re
RelGeo|Diete
Rastelt 5p
VIT:16
V17-2
Not 50
VIT-18
V17
DHwdy2
Postett 5p
V7-.kg
Vm7-22
DINd22
Masiieosn
RelGao(LZ)-
(ellGeDDitt-iD00 -100
V07:18
VI7-32
VI7:46
V17-18
V17-32
DeNo
DINDv2
DINOu2
DINOv2
Rell(osine))-20
RelGeo(12))220
Rostroh-50
V1T:18
V7--22
DH4Dv2
No
VIT-1E
V1T-22
V7--8
Rushet50
Ressutt50
Wi--2
VITu22
VIT-12
DINOv2
DINOV2
DINOv2
Reli(Coiie)
RelGeo(L2)-500
RelGeo(Diet)-500
Restot--6
M1h-2
NosNot-30
V/T-16
k1h-22
DiNov2
Reatet-50
K7-16
K7:32
DINOO2
50
50
Wil-22
DINOW2
DINO42
DWWOx2
PellCosine)-1000
RelGee/(Det)-1000
Restrot
M7:18
V7-32
DB40v2
Rex
K7-22
Pes
Nestwt50
Hestet-550
Hus/ett-50
VIT-L8
VIT-32
VIT-22
VIT-32
DINOw2
DINOv2
DIWOv2
Figure 14: Accuracies on CUB with varying number of anchors. From top bottom: 50, 100, 200, 500, 1000 anchors.
28

---

RelGeo|Diete DINd12
Rastee 5p
V1--2
NorNor-50 50
VIT-1e
V17-32
DHNOy2
Postet 5p
Vm7u2
Masineesn
RelGeo(L2)-
V07:1
VI7:32
DMM042
VI7:16
V17-18
V17-32
DeNo
DINDv2
DINOv2
DINOv2
Nell(Coine))-00
RelGeo(12)-200
V17:18
V7--22
DH4Du2
RvsNor-50
VIT-1E
V1T-22
V7--88
DINOu2
Rushet50
Rss
RessNtt50
Wi--66
VITu22
VIT-12
DINOv2
DINOV2
DINOv2
Co00nee)))
RelGeo(L2)-500
RelGeo(Diet)-500
M1h-22
DHMDv2
NosNot-30
V/T-16
k7h-2
DiNov2
Reatet-50
K7-46
K7:32
DINOO2
50
Wil-22
DINOW2
DINOx2
DWWOx2
NelICosine)-1000
Me/GeolL2)-1000
RelGeeo(Dett-1000
Restrot
M7:18
V7-22
DB40v2
Rex
W7-22
PesNet-30
Neshwt53
Hestett550
Hus/Ntt-5d
VIT-32
VIT-32
VIT-32
DINOV2
DINOv2
DIWOv2
1.00
Figure 15: MRR Cosine Sym on CUB with varying number of anchors. From top bottom: 50, 100, 200, 500, 1000 anchors.
29

---

0000
VIT-1
VIT-22
DRMbv2
NvsNoh:0
VIT-18
VI7-12
DRROV2
V1T-38
V17.22
DINOq
Reshet50
Rishkt50
Rcs
VTT-2
VITx22
Vil-22
DINOv2
DINOV2
DINOv2
points
points
points
V1:38
M1-32
DHADV2
No
Vm:-16
V1--2
DHNOY2
K7-16
V7-32
DHNON2
VT7-2
VIT.22
VIT-32
DINOV2
DINOV2
DIWOx2
Figure 16: Results of RelGeo(Diet) CUB with varying number of diet points. Top: accuracies; bottom: MRR Cosine Sym.
30
Rotbet
VIF-22
RarNeb-S0
V77.12
DRR0y2
Mes
V7
VIT-32
DINO22
WiF-22
DINDv2
DINO.2
BINOv2
steps
30
Rostrot55
VIT118
VT-32
DMMD
AvsNot:50
VIT-18
VTT-22
DRYOy2
V7--8
V7T-22
ONNOU2
Vil-26
VIT-32
VITx02
VIT-12
DINOw2
DINOV2
DINOv2
Figure 17: Results of ReIGeo(L2) on CUB with varying number of discretization steps. Top: accuracies: bottom: MRR Cosine Sym.
30

---

30
Rutheet35
VIT-8
VIT-22
DRMDN2
NvsNoh:0
VIT-18
VIT-12
VI7-38
V17.22
DINOq
Reshet50
Rishkt50
Rcs
VITx22
Vil-12
DINOv2
DINOV2
DINOv2
step
steps
sleps
V7:38
M1-32
DH4DV2
Vm--16
V1h32
DWNOy2
Restatt00
K7-16
V7-32
DHNOV2
VT7-2
VIT.22
VIT-32
DIMOW2
DINOV2
DIWOx2
Figure 18: Results of RelGeo(Diet) on CUB with varying number of discretization steps. Top: accuracies; bottom: MRR Cosine Sym.
Strength DINOu2
Rotbet
RorNeb-S0
Mes
V1F-22
I97.12
V7
VIT-32
WiF-22
DINDv2
DINO.2
DINOv2
Strength
Strength
Strength
Rostrot55
VTT118
VT-32
DMMD
AvsNot:50
VIT-1E
V1T-22
DRYOy2
V7-18
ONNOu2
Vil-26
VIT-32
VITx02
VIT-12
DINOV2
DINOV2
DINOv2
Figure 19: Results of RelGeo(Diet) on CUB with varying diet augmentation strengths. Results with strength can be seen above. Top: accuracies; bottom: MRR Cosine Sym.
31

---

Cosine)
ReiGeo(L2)
GDe(e
Rasteet 5p
VIT-22
NorNor-50
VIT-1e
V1T-32
DiNOy2
Postet 5p
VI7.18
V17-22
DINOV2
Ral(Cas
RelGeo(L2)
Rel Geo(Dett
V07:18
VIT-32
DMM042
Ryr
V17:16
V17-32
Pen
VI7-18
V17-32
DiNou2
DINDv2
DINOu2
DINOv2
Rel(Cosine)
RelGeo(122
Geo(Diet)
Rostroh55
V17:18
V7--22
DHMDv2
AvsNor-50
VIT-1E
V1T-22
DW
V7--88
DMNOu2
Reshet50
RessNtt50
Wi--2
VIT-22
VITu22
VIT-12
DINOv2
DINOV2
DINOv2
Restot-55
V1h-22
DHMDV2
NosNot-50
V/T-16
k1h-22
DiNOv2
Reatut-00
K7--6
V7:32
DINOV
50
50
Wil-22
DINOW2
DINO42
DWWOx2
ReIGeo(L2)
Geo(Diet)
Restrot
M7:18
WP-32
D84Dv2
Rex
PesNet-3t
V7-22
DWNO
Nestwt53
Hestett-50
Hus/Ntt-5d
VIl-26
VIT-32
VIT-22
VIT-32
DINOV2
DINOv2
DIWOx2
Figure 20: Results using the fps scheme to select the anchors. From top bottom: accuracies, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist.
32

---

Cosine)
ReiGeo(L2)
GDe(e
Rasteet 5p
VIT16
VIT-22
Daxbv2
NorNor-50
VIT-1e
V1T-32
Diwoy2
Postet 5p
VI7.18
V17-22
DINOV2
RaliCns
RelGeo(L2)
Rel Geo(Diet)
V07:18
VIT-32
DMM042
Ryr
V17:16
VI7-32
Pen
VI7-18
V17-32
DiNou2
Reshet50
DINDv2
DINOu2
DINOv2
Rel(Cosine)
RelGeo(122
Geo(Diet)
Rostroh55
V1T:18
V7--22
DH4Dv2
AvsNor-50
VIT-1E
V1T-22
DW
AwsMeR-00
V7--8
DMNOu2
Reshet50
RessNtt50
Wi--26
VIT-22
VITu22
VIT-12
DINOv2
DINOV2
DINOv2
Restot-55
V1h-22
DHMDV2
NrsNot-50
V/T-16
k1h-22
DiNOv2
Reatut-00
K7--6
V7:32
DINOV
50
50
Wil-22
DINOW2
DINO42
DWWOx2
ReIGeo(L2)
Geo(Diet)
Restrot
M7:18
WP-32
D84Dv2
Rex
PesNet-3g
V7-22
Nestwt53
Hestett-50
Hus/Ntt-5d
VIT-32
VIT-22
VIT-32
DINOV2
DINOv2
DINOx2
1.00
Figure 21: Results using the kmeans scheme to select the anchors. From top to bottom: accuracies, MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist.
33

---

ReeGee
V7-32
DMNON2
CUP-1
CUP-32 32
Rodtet-Sp 5g
DNNO22
CUP
16P-16
CUI-32
Reshet 50
1.00
0.03
0.02
0.02
3:03
0.03
Res
1.00
0.36
0.42
0.41
0.09
9.09
0.03
1.40
0.52
0.01
0.36
1.00
0.42
0.08
2.00
M1-32
DINO42
0.02
0:35
0:35
1.00
8.01
0.01
DRA0V2
9:41
0.42
0.45
1.00
0.11
0:10
CUP-16
0.03
0.01
0.02
0.94
CLI-166
0.03
0.05
0.11
2.00
0.98
Rel(Cosine)
RelGeo|Diett
Renteet 5g
W7-18
V77.32
16
RocMet-5
WF-1d
DINON2
16
VT.16
0.01
0.42
0.01
VI
0.25
1.00
V1--2
0.01
0.02
1.00
0.39
2.02
0.01
V1--22
0.28
0.64
1.00
0.75
2.08
5.00
0.39
0.39
0.36
CUP-16
0.01
0.75
CLI-366
2.08
1.00
CLPR-2
0.02
0.01
0.01
0.02
0.75
1.00
CLIP-2
2.08
0.07
0.08
0.30
0.95
1.00
Rel(Cosine)
RelGeo|Diett
Fentet-50
18
V77-22
CUP-10
CLIP
Rosdeel 500
VI-16
DHNON2
CLUR
Reshet 50
1.00
0.01
0.01
0.02
2.02
0.02
Restien-5 50
1.00
0.15
0.20
0.20
5.03
2.03
0.08
1.00
0.36
0.23
0.04
0.15
1.00
0.37
0.22
0.03
0.03
DIND2
8.13
0.28
0.24
1.00
8.06
0.06
D2VDV2
8.16
0.15
0.19
1.00
0.04
0.04
CUP 16
0.01
0:01
0.01
0.61
1.00
0.91
QLI-16
3.05
0.04
0.05
0.07
1.00
0.96
RellCosine)
RelGeo|Diett
Fest
V1-22
CUA26
32
WF-1
VT-32
DiNic22
16
0.01
0.01
0:13
0.03
VT.16
1.00
0.37
0.01
VT.36
0.10
1.00
5.01
VT.32
0.01
0.44
1.00
0.28
3.01
0.61
V1T32
5.12
0.42
1.00
6.24
2.0
0.03
0.17
1.00
0.15
011
1.00
3.84
CUP-36
0.01
ELIP-16
CUP-32 32
0.01
0.01
0.01
0.03
0.63
1.00
CLI-32
2.05
0.83
0.24
0.07
2.05
1.00
Figure 22: Stitching results for multimodal scenario. From top to bottom: MRR Cosine Sym, MRR CDist Sym, MRR Cosine, MRR CDist.
34

---

average over the different models to obtain uncertainty estimates. We remark that the exact running times depend heavily on implementation details, while currently we focus on correctness instead of the speed, and the running times could possibly be improved with better implementations.
Table 16: Times in seconds for training the decoders.
Decoder
CIFAR-10
CIFAR-100
ImageNet-IK
CUB
SVHN
Abs
10.467 0.765
10.71
8.804 + 0.581
1.393 0.615
15.37 0.843
(0)
3.462 + 0.03
3.501 = 0.045
3.556 = 0.018
3.509 = 0.028
3.506 0.044
Diet (1)
718.818 1: 216.179 179
730-446 1: 226.91
1651.49 = 211.037
1937 1: 192.894
705.929 1: 202.844
Diet (2)
735.144 +: 219.211
739.671 +: 222.36
2034.677 + 279
2428.941 019.942222 + 196.388
727-477 4: 206.929
Diet
771.299 +: 213.022
767.453 +: 222.62
26415.914 260.069
3201.588 242.339
754.965 202.021
Times to obtain the representations We first investigate the running times accuracy tradeoff of RelGeo representations on CUB dataset, where we vary the number of anchors and monitor the times to obtain the representations and the qualities of the resulting representations. We report the results on autoencoders in Table 17 and the results on vision foundation models Table 18 Table and Table 20 respectively.
We then investigate the times to evaluate the representations across different datasets and report the results in Table 21 under the same experimental settings reported in the main paper.
A.4.11 RelGeo(Fisher)
We additionally report the results of relative geodesic representations based on another choice of Rimennian metric, RelGeo(Fisher), which pulls back the Fisher-Rao metric from the classification heads' output probabilities on the different datasets. We report the aggregated results in Table [2] and the results the individual datasets in Figure 23 Figure 24 Figure 25 Figure 26 and Figure 27
RelGeo(Fisher) often results higher accuracies and lower MRRs; we hypothesize that this is due to the Neural Collapse phenomenon [Kothapalli 2023 observed in well-trained neural networks.
Table 17: Running time / accuracy tradeoff of RelGeo(L.22.
Num Anchors
Time (s) + Std
MRR
0.8480 = 0.0407
0.0168
3
0.8348 = 0.0367
0,0807
0.8377 0.0435
0.3503
8
0.9450 + 0.0285
0.7004
1.0969 0.0297
0.8384
15
1.2853 + 0.0264
0.9296
20
1.6160 - 0.0188
0.9616
25
1.8548 + 0.0218
0,9868
50
3.3311 + 0.0286
0.9981
100
6.2122 + 0.0318
0.9986
300
17.6494 + 0.0653
0,9982
500
29.1925 + 0.0806
0.9986
35

---

Accuracy
MRR Cosine
MRR CDist
Rustvet 5p
VI:16
VIP-22
VIT
VIT
DHwdy2
Postett 5p
VI7
V77 32
DINOV2
Casine
MRR CDIst
Rosten 50
V1: 16
VIT-12
Wh
Figure 23: Results of RelGie(Fiishr) on CIFAR-10.
Accuracy
MRR Cosine Sym
MAR CDist Sym
Bushot 36
WF:16
VIT-32
neiivv
RosNet 5g
WT.
k7 32-32
DlwOy2
Postez 30
WT
DINOva
MRR Cosine
MRR CDist
Restoe 50
V7 18
VTT322
DIN0y2
Aedtwtt
Vit
DRNOuQ
Reshi1-50
1.00
Reshet-50
3.00
Figure 24: Results of RelGeo(Fisher) CIFAR-100.
36

---

Accuracy
MRR Cosine
MRR CDist
Rostve 5p
Postet 5p
VI:16
VI--2
VIT
VI
Dlwdy2
VI7
V77
DINOG2
Casine
MRR CDIst
Rotten 50
V1T 16
VIT-12
Figure 25: Results of RelGee((iisher) on ImageNet- 1k.
Accuracy
MRR Cosine Sym
MAR CDist Sym
Bushot 36
WT
VIT-22
neiiv2
Roseet 5g
WT. 18
17 32
DlwOy2
Postez 30
W7
DINOya
MRR Cosine
MRR CDist
Restie 50
V1 18
VTT322
DIN0y2
Vir.
DRNOuQ
Reshe1-50
1.00
Reshet-50
1.00
Figure 26: Results of RelGeo(Fisher) on CUB.
37

---

Table 18: Running time accuracy tradeoff of Rel(Cosine).
Num Anchors
Time (8)
Accurncy
200
0,088 0.14
0.425 0.189
500
0.175 + 0.053
0.531 0.188
1000
0.062 + 0.091
0.539 + 0.179
Table 19: Running time accuracy tradeoff of RelGeo(L.2).
Num Anchors
Time (5)
Accuracy
200
8.004 + 3.304
0.5 0
500
21.297 8.774
U.515 = 0.163
1000
47.2 + 19.186
0.619 = 0.154
Table 20: Running time accuracy tradeoff of RelGeo(Diet).
Num Anchors
Time (8)
Accuracy
200
7.274 3.28
0.458 1 0.177
500
19.427 8.771
0.550 + 0.171
1000
43.108 1. 19.502
0.585 1 0.163
Table 21: Times in seconds of Rel (Cosine), RelGeo (L2) and RelGeo (Diet) for generating the representations.
Method
CIFAR-10
CIFAR- 100
ImageNet-Ik
CUB
SVHN
Rel (Cosine) Moschella et al. 2023 RelGeo (L.)
0.071 0.113
0.184 0.065
0.085 + 0.124
0.05 0.071 139.998 1: 66.509
85.414 40.61
86.079 40.652
258.788 + 70.064
RelGeo (DLet)
89.899 = 10.599
89.902 + 10.588
135.991 = 70.136
147.336 +: 66.503
Table 22: Results of RelGeo(Fisher).
Metric
CIFAR-10
CIFAR- 100
ImageNet-Ik
CUB
SVHN
Accuracy
0.959 0.026
0.894 + 0.043
0.623 0.103
0.729 = 0.133
0.625 0.033
MRR Cosine Sym
0.012 + 0.0
0.021 + 0.003
0.075
0.082
0.034
MRR CDist Sym
0.012 + 0.001
0.025 + 0.004
0.13 + 0.046
0.201 + 0.08
0.028 + 0.008
MRR Cosine
0.012 + 0.001
0.003 0.019 + 0.003
0.066 + 0.022
0.152 + 0.069
0.011
+ 0.002
MRR CDist
0.012 0.001
0.023 + 0.005
0.111 0.052
0.144 + 0.073
0.011 0.003
Accuracy
MRR Cosine Sym
MRR CDist Sym
Resteot 50
HT-38
VTT32
DaN0v2
Reshet-30
VIT- 17
WT.
32
Rester-300
MT. 18
VT. 22
DINO
Nushut50
0.69
0.65
0.66
0.67
ResNut-50
1.00
0.02
0.03
3:03
Res/Nt-50
1.00
0.02
0.02
0.02
VIT-15
0.57
0.60
0.88
0.50
VI3-18
9.02
1.00
0.05
3.04
Vil-2e
0.02
1.00
0.04
0.03
VIT-12
0.62
0.62
0.63
VIT-12
0.03
0.05
1.00
2.04
VIT-12
0.02
0.04
1.00
0.03
DINOV2
DINOV2
DINOx2
1.00
MAR Cosine
MRR CDist
Restiut 50
V7. 16
VTT322
Restewt-2 50
VIT-
DINOV2
ResNet 50
1.00
0.01
0.01
0.01
5c
1.00
0.61
0.01
3.01
VIF-18
0.01
1.00
0.02
0.02
VIl-18
0.01
1.00
0.02
3.01
VIK-32
5.01
0.02
1.00
0.02
VIT-32
0.01
5.02
1.00
0.01
DINO22
0.01
0.01
0.01
1.00
DINOx2
0.01
3.81
0.01
1.00
Figure 27: Results of RelGeo(Fisher) on SVHN.
38

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Main claims have been listed in the introduction and they have been explained in theoretical aspects (see Sec. 3 and have been supported by experimental results (Sec. 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations in our introduced approach has been discussed Sec. including the possible limitations with scalibility and computational efficiency.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
39

---

Answer: [Yes]
Justification: Theoretical background and notation are clearly explained in detail in Sec. 3.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the pseudocode for our approach in Sec. 3 as well as additional implementation details (such architectural details and hyperparameters) in Appendix AA.3] All the models and the datasets used in the experiments have been cited. Experimental settings have also been explained in Sec. 4.2 including details such number of anchors used in each experiment.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
40

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All the datasets used in the experiments have open access and they been cited accordingly in the paper. Codebase is not included in the main paper (it is available in the supplementary material), and the architectural details of the implemented models, such as classification head and decoders, are explained in Appendix [A.3.2] Other pretrained models are explained clearly (including the hyperparamer choices) and cited, partly in the main paper, and mostly in Appendix A.3
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
Sublic/guides /ttgg////ooeeuumiiieee (CodeSubmissionooliiii
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurlPS code and data submission guidelines (https http:///oooeeSmmiiiiieiioomm
wts.s/oobbmiiioooooooooooo.oo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Some experimental results are provided in the main paper, while more are provided Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our result tables report both the means and the standard deviations. Further details on the statistical properties of the performances of the methods are provided in Appendix A.4 where provide the full results and results under alternative aggregation.
Guidelines:
41

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details on the compute resources are provided in Appendix A.3.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///////aae....ooo/////////e
Answer: [Yes]
Justification:
We closely follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
42

---

Justification: Our work studies geometric concepts in distinct neural network models, thus has the potential to advance our understandings deep learning general. We remark that our work thus also suffers from the potential negative societal impacts of deep learning, as the trained models might used for dubious purposes.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release such data models. We largely focus using open source data and open source models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provide some information the main paper, and more in the Ap- pendix A.3.2
Guidelines:
43

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaassttsspperssaaaes paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: Concerning code, our code is provided in the Supplement and will be openly available upon acceptance. Concerning data, focus open source datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve research with human subjects, therefore it is not relevant.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
44

---

Answer:
[NA]
Justification:
The paper does not involve research with human subjects, therefore the
question is not relevant.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The core method of the paper does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not non non- stanstandard components. Please refer to our LLM policy for what should or should not be described.
involve LLMs as any important, original,
45


Reviews:
[{"review_id": "lzABNLSWEl", "reviewer": "Reviewer_7YCq", "summary": "This paper proposes an algorithm that connects two latent structures generated by two different models, using Riemannian geometry of the space of representations. The method aims to grant a Riemannian structure on the space of representations and use them (with anchors) to identify relative geometries of two spaces. This paper verified their algorithm by evaluating reconstruction and stitching capabilities.", "questions": "See weaknesses.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- This paper uses Riemannian metric for identifying the space of representations, and uses it for representation alignment, which looks like an advanced approach in representation alignment methods.\n\nWeaknesses\n- (line 122) In the rightmost term of the equation below the line 122, should there be the metric $G_X(x)$ in the middle of the two Jacobians?\n- (line 147) Is the $\\hat{\\gamma}$ the straight line between $z_1$ and $z_2$? What’s the b? Also, if that is a straight line (which will get one temporal parameter – probably $\\alpha$ here) it would be better to understand if you write the $\\hat{\\gamma}$ as a function of $\\alpha$.\n- Using the energy (or length) of a straight line interpolating two latent points, instead of geodesic distance, looks like a simple heuristic.I understand it’s hard to find any alternatives in a high-dimensional setting like this, but do you have any evidence (theoretical or empirical) that the energy (or length) of a straight line approximates the geodesic distance well?\n- The contents in section 3.1 and section 3.2 are pretty much disconnected (e.g. the notation is different – why use the lowercase $f$ for function in section 3.2?). It seems like this paper is proposing two distinct methods (pullback and diet), not a unified single method, representation alignment.\n- Section 3.2 is hard to read. Many statements are written without sufficient background knowledge or justification.\n  - (line 184) what does it mean that “it can identify the cluster centers of Von-Mises Fisher (VMF) distributions” – e.g. does it mean that the cluster center can be recovered from the latent variables encoded by the sample from the VMF?\n  - (line 189) why is it possible to recover the latent variable z by training an encoder f? Under what assumptions does your argument hold?\n  - (line 200) Do you mean that the latents are pushed into the unit sphere via a function f, and then the geodesic distance (=cosine distance) is computed?\n- In the first experiment (reconstruction), the results of Moschella et al. [1] look visually blurry (figure 4 of this paper, CIFAR10 dataset). However, according to the results in paper [1] (figure 4 of [1]), the reconstruction results look similar to the original ones (visually speaking). What’s the difference? \n- In the second experiment (zero-shot stitching) the Imagenet experiment has a better metric with the baseline method than the methods proposed by this paper. Since the imagenet is the largest (and most widely-benchmarked) dataset, should there be a proper explanation about it?\n\n[1] L. Moschella, V. Maiorca, M. Fumero, A. Norelli, F. Locatello, and E. Rodolà. Relative representations enable zero-shot latent space communication. ICLR, 2023, https://openreview.net/forum?id=SrC-nwieGJ.", "was_revised": true, "final_justification": "After reading the rebuttal and engaging in the discussion, I appreciate the authors’ clarifications. However, I remain concerned that this paper is suggesting two different methods, which don't share many things in common, as their key method. I maintain my score.", "rebuttal": "We thank the Reviewer for their detailed comments and feedback. Please find our responses below. We remain available for any additional questions during the discussion period.\n\n* **Typos line 122 and 147:** Thanks for noticing these typos. The convex combination should be:  $\\tilde{\\gamma}(z\\_1,z\\_2; \\alpha)= (1- \\alpha) z\\_1 + \\alpha z\\_2\\$. We fixed the typos in the manuscript.\n\n* **Evidence of fidelity of the straight line approximation:** In Figure 2 and the companion text starting in Line 161, we empirically verify that the straight line approximation is generally reasonable by reporting pairwise matrices and showing that the rank is preserved on random pairs of the CIFAR and MNIST dataset. Theoretically this provides an upper bound to the geodesic distance.\n\n* **Pullback and Diet:** We propose the general methodology of using Riemannian distances to enhance relative representations, pulling back a metric \\$G\\_{\\mathcal{X}}\\$ from the output space to the latent space. For classifier networks we employ two different choices for \\$G\\_{\\mathcal{X}}\\$ and its ambient space: pulling back the Euclidean metric from the logits space (denoted as pullback) and pulling back the spherical metric from the layer before in Diet decoders, assuming a vMF distribution in the space (denoted as Diet). We will clarify the text and improve the flow between Section 3.1 and Section 3.2.\n\n* **“Section 3.2 is hard to read”:** We skipped the technical details to keep the main text concise. We provide a concise summary below, and will provide a more detailed description in the paper. Further technical details can be found in \\[1]:\n\n  * Suppose there is a finite set of vectors \\$v\\_{c}\\$ on a unit sphere, representing each class, such that each instance belongs to exactly one class. The latent variables \\$z\\$ are drawn from a vMF distribution with concentration parameter \\$\\kappa\\$ from the corresponding \\$v\\_{c}\\$, and the data \\$x\\$ is generated using a function \\$g\\$ as \\$x=g(z)\\$. Then, with a Diet decoder, the representations before the final linear head identify the corresponding ground truth \\$z\\$ up to a transformation.\n  * The recovery of latent variable \\$z\\$ can be rigorously proven by expanding on the theoretical framework of non-linear ICA. The theory holds under reasonable assumptions on the data generation process and the embeddings and weights in Diet decoder’s final layer. For a formal treatment of the assumptions, we refer the reviewer to Assumptions 1C and Theorem 1C in \\[1].\n  * Theory suggests that a trained Diet decoder is capable of identifying the ground truth \\$z\\$ up to a transformation. As such, we directly assign a spherical geometry to \\$z\\$, and compute the distances accordingly.\n\n* **Reconstruction experiment (stitching):** In the experiment in Figure 4 we consider a setting where the number of anchors is limited, i.e., 5 for MNIST and FMNIST datasets and 10 for CIFAR, which leads to the difference: in the corresponding Figure 4 in (Moschella et al. 2023), the number of anchors was set to 500 (cfr. Appendix A.5.5 in Moschella et al. 2023). This highlights the huge increase in sample efficiency of our approach, as confirmed by our results in Figure 3.\n\n* **Performance on Imagenet (stitching):** We currently do not have a definitive explanation of this, and it remains an interesting avenue for future work.\n\n\\[1] *Cross-Entropy Is All You Need To Invert the Data Generating Process*, Reizinger et al., ICLR 2025"}, {"review_id": "DAJ34yFNnN", "reviewer": "Reviewer_ZLWQ", "summary": "This paper focuses on the similarities of learned representations of vision models on the same task/dataset, either through smaller differences like initialization or larger differences like architecture. Previous work has analyzed this from a linear relative perspective, but this paper takes a geometric analysis and treats the underlying dataset as drawn from an embedded submanifold. Using a Riemannian metric dependent on the encoders/decoders, they propose a Riemannian-metric-aware representation that allows for better stitching and retrieval across features of different models.", "questions": "1. Are there reasons why MNIST and CIFAR-10 are enough to demonstrate the need for a Riemannian approach for stitching? Are the datasets themselves not linearly alignable?\n2. Would experiments on more modern image datasets like ImageNet run into issues with the linear interpolation scheme?\n3. Are there particular benefits to a training-free approach for representation alignment? What does the computation time look like compared to other methods that would require optimization?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The work is theoretically well-motivated, building on differential geometry and the pull-back metric formalism, and presents clear derivations and an explicit algorithm (although maybe an equation reference or two would be helpful for the reader to tie back to the paper). The stitching performance increase is notable, showing there is something significant to the authors' method. A Riemannian approach to this representational similarity area is, to my knowledge, still not properly explored.\n\nThere are some experimental confusions and concerns. Figure 2 seems to imply that the representation spaces are more or less straight if linear metrics are almost the same as Riemannian metrics, even though I imagine there must be some difference given the notable increase in stitching performance. Still, there is some concern that experiments are restricted to MNIST and CIFAR-10, which are smaller image datasets and potentially more geometrically simple. As stated in limitations, the method assumes that the latent space is convex and thus linear interpolants all lie in a well-defined region for the metric, which may not always be true. In this case, it is still unclear the effectiveness of this versus different variations of a linear alignment. While this paper is a training-free algorithm, it would be nice to compare against linear methods in order to validate the unique usefulness of the geometric approach here.", "was_revised": true, "final_justification": "The provided experiments and explanation have helped with validating that a Riemannian approach is necessary in these settings and provides nontrivial insight.", "rebuttal": "We thank the reviewer for their thoughtful comments. We address their concerns below and remain available for further questions during the discussion period.\n\n\n* **Clarifications about linearity:**\nWe respectfully remark that the approximation using the straight line does not correspond to a linear metric. \n\n* **“Figure 2 seems to imply that…”:** Figure 2 and the companion text starting in Line 161 serve as a verification that the proposed straight-line approximation is a meaningful approximation of the ground truth geodesic distance, preserving the rank. We remark that the straight-line approximation corresponds to the Riemannian curve length (or the energy)  of the line (“straight” in the Euclidean sense) under the Riemannian metrics. In other words, both the approximation and the true geodesic use the same Riemannian metric, with the difference lying in the employed curve. The geodesic energy is an upper bound to the true geodesic distance. Additionally we point out that the latter is not scalable to large models, while the approximation nicely scales.  \n\n* **Comparison with linear methods:**\nWe repeat the stitching experiment of Figure 3, adding two additional baselines: \n    * Linear: fitting a linear map between the two spaces using the anchors\n    * Ortho: fitting an orthogonal map between the two spaces using the anchors, similarly to (Maiorca et al. [2024])\n\n    and report the results in the tables below. We also note that additional qualitative evidence of comparison with linear methods is in Figure 4, fourth column. \n\n**Takeaways:** Our method outperforms both the new baselines on all datasets (MNIST, FMNIST, CIFAR), while being training free. Additionally, we observe that, despite its simplicity, the dataset cannot be aligned with only a few anchors available, as confirmed by our results. \n\n\n\n\n\n\n### **MNIST**\n\n| Method/Num anchors           | 2               | 3               | 5               | 8               | 10              | 15              | 20              | 25              | 50              | 100             | 300             | 500             |\n| ------------------ | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- |\n| rel cosine         | 0.00 ± 0.00     | 0.02 ± 0.00     | 0.15 ± 0.04     | 0.52 ± 0.05     | 0.68 ± 0.05     | 0.86 ± 0.03     | 0.92 ± 0.01     | 0.94 ± 0.01     | 0.97 ± 0.01     | 0.98 ± 0.00     | 0.99 ± 0.00     | 0.99 ± 0.00     |\n| **rel geo (ours)** | **0.02 ± 0.00** | **0.17 ± 0.05** | **0.85 ± 0.04** | **0.99 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** |\n| lin                | 0.00 ± 0.00     | 0.00 ± 0.00     | 0.01 ± 0.00     | 0.02 ± 0.00     | 0.03 ± 0.00     | 0.11 ± 0.01     | 0.25 ± 0.02     | 0.43 ± 0.02     | 0.94 ± 0.01     | 1.00 ± 0.00     | 1.00 ± 0.00     | 1.00 ± 0.00     |\n| ortho              | 0.00 ± 0.00     | 0.00 ± 0.00     | 0.01 ± 0.00     | 0.01 ± 0.00     | 0.02 ± 0.00     | 0.09 ± 0.01     | 0.18 ± 0.02     | 0.32 ± 0.03     | 0.90 ± 0.01     | 1.00 ± 0.00     | 1.00 ± 0.00     | 1.00 ± 0.00     |\n\n---\n\n### **FMNIST**\n\n| Method/Num anchors                 | 2               | 3               | 5               | 8               | 10              | 15              | 20              | 25              | 50              | 100             | 300             | 500             |\n| ------------------ | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- |\n| rel cosine         | 0.01 ± 0.00     | 0.03 ± 0.01     | 0.17 ± 0.02     | 0.38 ± 0.04     | 0.51 ± 0.03     | 0.68 ± 0.03     | 0.74 ± 0.01     | 0.76 ± 0.03     | 0.83 ± 0.02     | 0.85 ± 0.01     | 0.87 ± 0.01     | 0.88 ± 0.01     |\n| **rel geo (ours)** | **0.02 ± 0.00** | **0.17 ± 0.06** | **0.61 ± 0.14** | **0.78 ± 0.11** | **0.88 ± 0.07** | **0.94 ± 0.02** | **0.96 ± 0.02** | **0.98 ± 0.01** | **0.99 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** |\n| lin                | 0.00 ± 0.00     | 0.00 ± 0.00     | 0.01 ± 0.00     | 0.01 ± 0.00     | 0.02 ± 0.00     | 0.05 ± 0.01     | 0.09 ± 0.01     | 0.14 ± 0.01     | 0.54 ± 0.02     | 0.97 ± 0.00     | 1.00 ± 0.00     | 1.00 ± 0.00     |\n| ortho              | 0.00 ± 0.00     | 0.00 ± 0.00     | 0.01 ± 0.00     | 0.01 ± 0.00     | 0.02 ± 0.00     | 0.05 ± 0.01     | 0.08 ± 0.01     | 0.13 ± 0.01     | 0.48 ± 0.02     | 0.95 ± 0.01     | 1.00 ± 0.00     | 1.00 ± 0.00     |\n\n---\n\n### **CIFAR**\n\n| Method/Num anchors                 | 2               | 3               | 5               | 8               | 10              | 15              | 20              | 25              | 50              | 100             | 300             | 500             |\n| ------------------ | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- |\n| rel cosine         | 0.00 ± 0.00     | 0.01 ± 0.00     | 0.05 ± 0.02     | 0.12 ± 0.03     | 0.11 ± 0.08     | 0.15 ± 0.10     | 0.19 ± 0.06     | 0.25 ± 0.07     | 0.37 ± 0.06     | 0.32 ± 0.03     | 0.37 ± 0.03     | 0.39 ± 0.03     |\n| **rel geo (ours)** | **0.04 ± 0.00** | **0.39 ± 0.09** | **0.99 ± 0.01** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** | **1.00 ± 0.00** |\n| lin                | 0.00 ± 0.00     | 0.00 ± 0.00     | 0.01 ± 0.00     | 0.03 ± 0.01     | 0.06 ± 0.01     | 0.19 ± 0.03     | 0.35 ± 0.02     | 0.51 ± 0.04     | 0.94 ± 0.01     | 1.00 ± 0.00     | 1.00 ± 0.00     | 1.00 ± 0.00     |\n| ortho              | 0.00 ± 0.00     | 0.00 ± 0.00     | 0.00 ± 0.00     | 0.01 ± 0.00     | 0.01 ± 0.00     | 0.03 ± 0.01     | 0.05 ± 0.02     | 0.07 ± 0.03     | 0.27 ± 0.07     | 0.74 ± 0.08     | 0.98 ± 0.00     | 0.99 ± 0.00     |\n\n* **Experiment on larger datasets:** We provided stitching experiments on vision foundation models in Section 5, which involved larger datasets, demonstrating the performance of our method and of the geodesic energy approximation scheme. We remark that computing true geodesics with models of this size is largely infeasible, due to computational and memory constraints, as well as optimization difficulties since the Jacobian might be ill defined at some points.\n\n* **Benefits to a training-free approach for representation alignment:** For stitching application, having a non parametric method is of fundamental importance for different reasons: \n    - Canonical mapping: Our method can be interpreted as mapping different spaces to a canonical one, invariant to the Riemannian metric adopted (see Figure 1 in the manuscript). This preferred to fitting maps between spaces as the number of adapters to train increases combinatorially with the number of spaces considered. This was observed as well in (Moschella 2023). \n    - Sample efficiency: Our method is particularly sample efficient, requiring few anchors (aligned data) in order to work. As opposed, approaches which involve training (e.g. fitting linear maps or neural networks) require much more aligned data to learn a good map (see for example, the experiment above). This is of fundamental importance for example in multimodal applications, where paired data is costly."}, {"review_id": "yoQFA1RZpc", "reviewer": "Reviewer_age6", "summary": "The paper aims at problems like model stitching or retrieval, in which latent distributions or latent vectors learned by one model have to be compared/stitched to the latent space of another model. The proposal of the paper is conceptually straightforward, it is to take inspiration from Riemannian geometry and to use pullback metrics on the data space, which would work if the models under consideration in stitching are learning the same underlying manifold or distribution. This assumption in practice is just a wishful thinking approximation, however the method inspired by it can still work nevertheless. The paper shows that in fact this is the case, and the above basic idea using approximate geodesic distance has good performance.", "questions": "1) How fast is the computation of formulas (4) and (5) from the paper? And what value of N do you choose in the experiments? Why do you use that choices? \n\n2) Line 199-200: you use the sphere distance for Diet, and you say later (line 205-207) that this is not fully justified (\"is a strong assumption\") but you say in line 206-207 that this \"results in meaningful representations\" where meaningful means what?. Can you point to what this means explicitly/quantitatively in what part of the paper?", "limitations": "I think this part does not apply to this paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The strength of the paper is that the experiments work, therefore the somehow bold hypothesis that a same underlying manifold characterizes to a strong degree same data seen by different models, is robust enough that the differential geometry insights on which the paper's methods are based resist and give good accuracy.\n\nThe weakness for me is that we don't quite have a clean explanation of why that works. It is known that the manifold hypothesis is not quite true in general, so some explanation seems to be wanting.", "was_revised": true, "final_justification": "Following the discussion with the authors, I've become more confident (confidence from 2 to 4) on the paper and I think that I can safely raise my score from 4 to 5.", "rebuttal": "We thank the reviewer for their feedback and suggestions. We address their questions in the following, and remain available for any further clarification in the discussion period.\n\n“**Why it works**”: Prior work on representational alignment has shown that representations from different models can often be approximately aligned using simple transformations (e.g., linear, orthogonal, or locally linear maps). Even when models are trained independently—with different architectures, modalities, or datasets that nonetheless share an underlying structure—they tend to learn similar representations, suggesting convergence toward a shared encoding of entities (Huh et al., 2024).\nFrom a theoretical standpoint, identifiability results [a] imply that if two discriminative models learn the same likelihood function, their internal representations must be equivalent up to a linear transformation. However, this ideal scenario rarely holds exactly in practice: training dynamics, nuisance factors, and unmodeled variability can all introduce distortions.\nIn our case, it may be too strong to assume that two models learn different parametrizations of an identical manifold. Instead, we adopt a weaker assumption—that they do so up to some bounded distortion. Recent theoretical work has begun to explore relaxations of strict identifiability to account for such bounded distortions [b]. Integrating these relaxations into our Riemannian framework presents a promising direction for future work.\n\n*[a] Roeder, Geoffrey, Luke Metz, and Durk Kingma. \"On linear identifiability of learned representations.\" International Conference on Machine Learning. PMLR, 2021.*\n\n*[b] Nielsen, Beatrix MG, et al. \"When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective.\" arXiv preprint arXiv:2506.03784 (2025).*\n\n**How fast is the computation of formulas (4) and (5)**: the computation of the relative geodesic representation for a single sample requires $N \\times | \\mathcal{A}_X |$  forward passes of the decoder, where $N$ is the number of steps, and $| \\mathcal{A}_X |$ is the number of anchors. Notice that these are vectorizable and can be run all in parallel on GPU, allowing for fast evaluation.  The number of steps we employ in the experiments is 15. In Figure 15, 16 in the Appendix, we provide an ablation on N on stitching experiments. \nThe computations can be accelerated through utilizing the pullback geometry. Specifically, each outer product involving the velocity with the Riemannian metric over a small timeframe can be approximated with the corresponding quantity viewed in the output space. For Pullback, this implies the Euclidean distances between the function outputs; for Diet, this implies the spherical distances between the function outputs. Furthermore, the computations along the straight line approximation can be vectorized. As such, they allow reasonably fast evaluations.\n\n**Line 199-200**: we agree that the text can be improved. We stated that it was “meaningful”, as the resulting method was demonstrated to achieve good performances in the experiments on vision foundation models as presented in Section 5, where it consistently yielded the most identifiable relative representations as measured by mean reciprocal ranks."}, {"review_id": "F9yIGjdSCz", "reviewer": "Reviewer_k3Ho", "summary": "The paper introduces Relative Geodesic Representations, a framework for aligning the latent spaces of independently trained neural networks. The authors endow each latent space with a Riemannian metric obtained by pulling back a metric defined in output space. Then, they compute approximate geodesic lengths along straight-line latent paths to construct distance-based relative feature vectors for a small set of anchor points. They demonstrate that these representations are approximately invariant to isometric transformations that link the two latent spaces. They tackle both the retrieval and identification of corresponding points, as well as zero-shot module stitching across models, encompassing auto-encoders, ResNet/ViT classifiers, and DINOv2 self-supervised encoders. Across five vision datasets, the proposed method, especially the variant that pulls back a spherical metric learned with the Diet self-supervised head, consistently improves mean-reciprocal-rank retrieval and stitched accuracy over prior cosine-based relative representations, while requiring only a handful of anchors.", "questions": "1. Experiments use random anchors; have you tried diversity-based or class-balanced sampling, and how does performance vary with anchor quality? Demonstrating robustness or providing guidance could improve clarity.\n2. Your approximation relies on a fixed number of steps N (Eq. 4–5). How sensitive are alignment and stitching metrics to N, and is there an adaptive criterion that balances accuracy and efficiency?\n3. You suggest that relative geodesics could aid multimodal alignment. What modifications (if any) are needed to handle heterogeneous output geometries, aligning a vision encoder with a text encoder, and do preliminary results support this claim?\n4. Proposition 3.1 guarantees length invariance under re-parametrisation, but the paper does not discuss how many anchors are theoretically sufficient. Is it possible to bound the number of anchors needed so that the relative geodesic embedding preserves all pairwise geodesic ranks with a specific probability?\n5. The method is invariant to isometries of the underlying manifold. Which non-isometric transformations (e.g., local dilations, shears) break the relative geodesic representation, and can you characterize their practical prevalence in modern networks?", "limitations": "The paper includes a “Limitations and future work” subsection, and the checklist discusses broader impacts.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Quality: Solid formulation grounded in differential geometry; extensive experiments on both small and large-scale foundation models with multiple backbones; competitive baselines; standard deviations reported.\n2. Clarity: Writing and contributions are generally clear, with well-designed figures.\n3. Significance: Bridges latent-space geometry and cross-model alignment, yielding a unifying perspective. Has potential practical payoff for model reuse without training data.\n4. Originality: Extends Relative Representations by replacing cosine with intrinsic geodesic distances and proposing a scalable proxy, the first to demonstrate that pull-back metrics from discriminative/self-supervised heads improve cross-model tasks.\n\nWeakness:\n1. Quality: No quantitative error analysis; computational cost (seconds / GPU-hours) is not included.\n2. Clarity: Dense notation (multiple pull-back metrics) may overwhelm non-geometric readers.\n3. Significance: The impact is currently demonstrated only for vision; the benefit for language or multimodal models remains speculative.\n4. Originality: Builds directly on prior relative-representation framework; novelty is incremental in that sense.", "was_revised": true, "final_justification": "I raise the score on originality from 2 to 3.", "rebuttal": "We thank the reviewers for their feedback and suggestions. Below we address the Reviewer concerns. We remain available for any further clarification during the discussion period.\n\n**Anchors sampling strategies**: We report the accuracies and symmetricized MRR based on Cosine on CUB dataset, comparing uniform sampling for anchors (as used in the main paper) with using farthest point sampling (fps) and kmeans, as described in Moschella et al 2023. We observe that the general conclusion persists. Below we report the stitching accuracies; the other results based on retrieval behaved similarly, and have not been included for space constraints:  we included the full results in the manuscript.\n\n### Accuracies\n#### Rel(Cosine)\n##### uniform\n|           | ResNet-50 | VIT-16  | VIT-32  | DINOv2  |\n| --------- | --------- | ------- | ------- | ------- |\n| ResNet-50 | $0.519$   | $0.216$ | $0.241$ | $0.218$ |\n| VIT-16    | $0.408$   | $0.760$ | $0.645$ | $0.559$ |\n| VIT-32    | $0.426$   | $0.671$ | $0.773$ | $0.581$ |\n| DINOv2    | $0.443$   | $0.602$ | $0.605$ | $0.833$ |\n\n##### fps\n|           | ResNet-50 | VIT-16  | VIT-32  | DINOv2  |\n| --------- | --------- | ------- | ------- | ------- |\n| ResNet-50 | $0.523$   | $0.230$ | $0.242$ | $0.219$ |\n| VIT-16    | $0.419$   | $0.764$ | $0.653$ | $0.563$ |\n| VIT-32    | $0.438$   | $0.671$ | $0.782$ | $0.592$ |\n| DINOv2    | $0.489$   | $0.597$ | $0.611$ | $0.838$ |\n\n##### kmeans\n|           | ResNet-50 | VIT-16  | VIT-32  | DINOv2  |\n| --------- | --------- | ------- | ------- | ------- |\n| ResNet-50 | $0.522$   | $0.220$ | $0.237$ | $0.234$ |\n| VIT-16    | $0.412$   | $0.764$ | $0.657$ | $0.560$ |\n| VIT-32    | $0.430$   | $0.682$ | $0.780$ | $0.592$ |\n| DINOv2    | $0.475$   | $0.597$ | $0.611$ | $0.834$ |\n\n\n#### RelGeo(Pullback)\n##### uniform\n|           | ResNet-50 | VIT-16  | VIT-32  | DINOv2  |\n| --------- | --------- | ------- | ------- | ------- |\n| ResNet-50 | $0.614$   | $0.312$ | $0.332$ | $0.335$ |\n| VIT-16    | $0.487$   | $0.793$ | $0.667$ | $0.593$ |\n| VIT-32    | $0.515$   | $0.695$ | $0.804$ | $0.626$ |\n| DINOv2    | $0.538$   | $0.670$ | $0.673$ | $0.866$ |\n\n##### fps\n|           | ResNet-50 | VIT-16  | VIT-32  | DINOv2  |\n| --------- | --------- | ------- | ------- | ------- |\n| ResNet-50 | $0.615$   | $0.299$ | $0.327$ | $0.312$ |\n| VIT-16    | $0.476$   | $0.796$ | $0.656$ | $0.599$ |\n| VIT-32    | $0.491$   | $0.690$ | $0.805$ | $0.617$ |\n| DINOv2    | $0.543$   | $0.634$ | $0.659$ | $0.868$ |\n\n##### kmeans\n|           | ResNet-50 | VIT-16  | VIT-32  | DINOv2  |\n| --------- | --------- | ------- | ------- | ------- |\n| ResNet-50 | $0.609$   | $0.329$ | $0.341$ | $0.332$ |\n| VIT-16    | $0.511$   | $0.793$ | $0.682$ | $0.614$ |\n| VIT-32    | $0.530$   | $0.689$ | $0.802$ | $0.631$ |\n| DINOv2    | $0.591$   | $0.665$ | $0.682$ | $0.865$ |\n\n\n#### RelGeo(Diet)\n##### uniform\n|           | ResNet-50 | VIT-16  | VIT-32  | DINOv2  |\n| --------- | --------- | ------- | ------- | ------- |\n| ResNet-50 | $0.539$   | $0.242$ | $0.255$ | $0.249$ |\n| VIT-16    | $0.429$   | $0.763$ | $0.630$ | $0.533$ |\n| VIT-32    | $0.432$   | $0.653$ | $0.780$ | $0.567$ |\n| DINOv2    | $0.483$   | $0.610$ | $0.619$ | $0.847$ |\n\n##### fps\n|           | ResNet-50 | VIT-16  | VIT-32  | DINOv2  |\n| --------- | --------- | ------- | ------- | ------- |\n| ResNet-50 | $0.544$   | $0.248$ | $0.260$ | $0.244$ |\n| VIT-16    | $0.407$   | $0.764$ | $0.623$ | $0.530$ |\n| VIT-32    | $0.445$   | $0.645$ | $0.782$ | $0.578$ |\n| DINOv2    | $0.501$   | $0.611$ | $0.632$ | $0.849$ |\n\n##### kmeans\n|           | ResNet-50 | VIT-16  | VIT-32  | DINOv2  |\n| --------- | --------- | ------- | ------- | ------- |\n| ResNet-50 | $0.540$   | $0.269$ | $0.281$ | $0.269$ |\n| VIT-16    | $0.435$   | $0.762$ | $0.640$ | $0.548$ |\n| VIT-32    | $0.456$   | $0.646$ | $0.775$ | $0.566$ |\n| DINOv2    | $0.511$   | $0.612$ | $0.628$ | $0.849$ |\n\n\n**“Sensitive .. to N”**: we provided an ablation study of the effect of N in Appendix A.2.6. We observe that the proposed method is not too sensitive to N.\n\n**“Multimodal”**: we report the results of Rel(Cosine) and RelGeo(Diet) on aligning vision models and the text encoder of CLIP on Flickr30k. We use 2000 Diet data points, and simply use the first text string of each data point for the text part. We report the retrieval results on 10000 test data points. As shown in the result tables, RelGeo(Diet) outperforms Rel(Cosine) also in the multimodal setting. We will incorporate the full results of the experiment in the manuscript.\n\n### Symmetricized MRR Cosine\n#### Cos\n|           | ResNet-50 | VIT-16  | VIT-32  | DINOv2  | CLIP-16 | CLIP-32 |\n| --------- | --------- | ------- | ------- | ------- | ------- | ------- |\n| ResNet-50 | $1.000$   | $0.026$ | $0.024$ | $0.025$ | $0.032$ | $0.032$ |\n| VIT-16    | $0.026$   | $1.000$ | $0.515$ | $0.347$ | $0.011$ | $0.010$ |\n| VIT-32    | $0.024$   | $0.515$ | $1.000$ | $0.350$ | $0.012$ | $0.011$ |\n| DINOv2    | $0.025$   | $0.347$ | $0.350$ | $1.000$ | $0.013$ | $0.013$ |\n| CLIP-16   | $0.032$   | $0.011$ | $0.012$ | $0.013$ | $1.000$ | $0.942$ |\n| CLIP-32   | $0.032$   | $0.010$ | $0.011$ | $0.013$ | $0.942$ | $1.000$ |\n\n#### RelGeo(Diet)\n|           | ResNet-50 | VIT-16  | VIT-32  | DINOv2  | CLIP-16 | CLIP-32 |\n| --------- | --------- | ------- | ------- | ------- | ------- | ------- |\n| ResNet-50 | $1.000$   | $0.356$ | $0.415$ | $0.415$ | $0.093$ | $0.089$ |\n| VIT-16    | $0.356$   | $1.000$ | $0.652$ | $0.425$ | $0.085$ | $0.077$ |\n| VIT-32    | $0.415$   | $0.652$ | $1.000$ | $0.446$ | $0.092$ | $0.088$ |\n| DINOv2    | $0.415$   | $0.425$ | $0.446$ | $1.000$ | $0.108$ | $0.104$ |\n| CLIP-16   | $0.093$   | $0.085$ | $0.092$ | $0.108$ | $1.000$ | $0.982$ |\n| CLIP-32   | $0.089$   | $0.077$ | $0.088$ | $0.104$ | $0.982$ | $1.000$ |\n\n**“Proposition 3.1”**: Proposition 3.1 states that each geodesic distance as used by relative geodesic representations is invariant under re-parameterisations, which holds regardless of the number of anchors.\n\n**“Non-isometric transformation”**: the Riemannian geometry framework only provides invariance under isometries. As such, it is unclear what happens under non-isometric transformations. We remark that, suppose we view each neural network as a smooth function, up to some assumptions, invariance under isometries can provide certain identifiable guarantees; see e.g. [a].\n\n_[a] Identifying metric structures of deep latent variable models, Syrota et al., ICML 2025_"}]

Author rebuttal:

Meta-review: This paper leverages Riemannian geometry to build representations from pullback metrics. The proposed ideas allow to align representations learned by different networks, e.g. by encoding a datapoint with one autoencoder, aligning the representation with the latent space of a different autoencoder, and then decoding with the second decoder. The high degree to which the authors show different latent spaces can be aligned is quite surprising, and provides strong evidence suggesting different models are really learning the same underlying manifold. I believe the community would benefit from being aware of these findings.

The empirical results are particularly strong and were praised by the reviewers. The paper received split reviews, with two reviewers recommending borderline rejection, one reviewer recommending borderline acceptance, and one reviewer strongly championing the paper. The main criticism from the reviewers who recommended rejection is that this is a dense paper, and that it will likely be hard to follow for readers who do not have a background in differential geometry. The other two reviewers found the paper to be well written and clear. I went through the paper carefully and while I agree that readers without a background in geometry might struggle to understand some aspects of the paper, I also believe this is a clear and well written paper for its intended audience.

Overall, this paper presents a well-motivated approach that obtains strong empirical results. I believe this paper is valuable to the community working on leveraging tools from differential geometry for machine learning, and that the difficulty of the material to those lacking background in geometry is not a compelling reason for rejection. I thus recommend acceptance. This being said, with the goal of making the paper more accessible to readers without a background in differential geometry, I also ask the authors to please add a more complete informal description of the method for the camera-ready version of the paper.
Decision: Accept (poster)