Latent Functional Maps: a spectral framework for representation alignment
Authors: Marco Fumero
Marco Pegoraro
Valentino Maiorca
Francesco Locatello
Emanuele Rodolà
Venue: NeurIPS 2024 poster (2024)
Keywords: Representation alignment; Spectral methods;
Primary area: deep_learning_architectures

Abstract: Neural models learn data representations that lie on low-dimensional manifolds, yet modeling the relation between these representational spaces is an ongoing challenge.
By integrating spectral geometry principles into neural modeling, we show that this problem can be better addressed in the functional domain, mitigating complexity, while enhancing interpretability and performances on downstream tasks. 
To this end, we introduce a multi-purpose framework to the representation learning community, which allows to: (i) compare different spaces in an interpretable way and measure their intrinsic similarity; (ii) find correspondences between them, both in unsupervised and weakly supervised settings, and (iii) to effectively transfer representations between distinct spaces.
We validate our framework on various applications, ranging from stitching to retrieval tasks, and on multiple modalities, demonstrating that Latent Functional Maps can serve as a swiss-army knife for representation alignment.

Latent Functional Maps: a spectral framework for representation alignment
Marco Fumero" IST Austria marco www.roiist..ccaaat
Marco Pegoraro" Sapienza, University of Rome pegoraro@di. uniromal. it
Valentino Maiorcal Sapienza, University of Rome maiorca@di. uniromal.is it
Francesco Locatello
IST Austria
low.eesco.ocaaeello@iis...o
Emanuele Rodolà Sapienza, University of Rome rodolaâdi, uniroma1. it
Abstract
Neural models learn data representations that lie on low-dimensional manifolds, yet modeling the relation between these representational spaces is ongoing chal- lenge. By integrating spectral geometry principles into neural modeling, we show that this problem be better addressed in the functional domain, mitigating com- plexity, while enhancing interpretability and performances on downstream tasks. To this end, introduce multi-purpose framework to the representation learning community, which allows to: (i) compare different spaces an interpretable way and measure their intrinsic similarity; (ii) find correspondences between them, both in unsupervised and weakly supervised settings, and (iii) to effectively transfer representations between distinct spaces. We validate our framework various applications, ranging from stitching to retrieval tasks, and on multiple modalities, demonstrating that Latent Functional Maps can serve as a swiss-army knife for representation alignment.
Introduction
Neural Networks (NNs) learn to represent high-dimensional data elements of lower-dimensional latent spaces. While much attention given to the model's output for tasks such as classification, generation, reconstruction, or denoising, understanding the internal representations and their geometry equally important. This understanding facilitates reusing representations for downstream tasks
[62] [4] and the comparison between different models [23] (3),, thereby broadening the applicability of NNs, and understanding their structure and properties. Morcover, recent studies have shown that distinct neural models often develop similar representations when exposed to similar stimuli,
a phenomenon observed in both biological [16] 25 [33 and artificial settings [38] 23 39] Notably,
when neural networks have different architectures. internal representations of distinct models
can often be aligned through linear transformation [6]] This indicates a level of consistency in how NNs process information, highlighting the importance of exploring and understanding these internal representations.
Equal Contribution
Work done while visiting ISTA
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

M
M
Gx (NN(X,d)
D
C: C:F(M) - F(M)
(ii)
(iii)
N
Gy ANNO), d)
Figure 1: Framework overview: given two spaces X, y their samples lie on two manifolds M, which can be approximated with the KNN graphs Gx.Gy. We can optimize for latent functional map C between the eigenbases of operators defined on the graphs. This map serves as a map between functions defined on the manifolds and can be leveraged for (i) comparing representational spaces, (ii) solving correspondence problems, and (iii) transferring information between the spaces.
In this paper, we shift our focus from characterizing relationships between samples in distinct latent spaces to modeling a map among funcctinn spaces defined on these representational spaces. To this end, we propose leveraging spectral geometry principles by applying the framework of functional maps [4 to the field of representation learning. Functional maps represent correspondences between function spaces on different manifolds, providing new perspective on the problem of representational aligmment. In this setting. many complex constraints can be easily expressed compactly [45] For instance, as shown in Figure [ the mapping (C) between two spaces M and N can be represented in the functional space as linear map with sparse structure.
Originally used in 3D geometry processing and more recently on graphs applications 660 we
extend this framework to handle arbitrary large dimensional latent spaces. Our proposed method, Latent Functional Map (LFM), is flexible tool that allows (i) compare distinct representational spaces, (ii) to find correspondences between them both in an unsupervised and weakly supervised way, and (iii) transfer information between them. Our contributions can be listed follows:
We introduce the framework of Latent Functional Maps as way to model the relation between distinct representational spaces of neural models.
We show that LFM allows us to find correspondences between representational spaces, both in weakly supervised and unsupervised settings, and to transfer representations across distinct models.
We showcase LFM capabilities as meaningful and interpretable similarity measure between representational spaces.
We validate our findings in retrieval and stitching tasks across different models, modalities and datasets, demonstrating that LFMs can lead to better performance and sample efficiency than other methods.
2 Related Work
Similarity between latent spaces. Comparing representations learned by neural models is of fundamental importance for diversity of tasks [63], ranging from representation analysis to latent space alignment and neural dynamics. In order to do so, a similarity measure between different spaces must be defined [22] This can range from functional similarity (matching the performance of two models) to similarity defined in representational space 23 which is where our framework falls in. A classical statistical method is Canonical Correlation Analysis (CCA) [8) known for its invariance to lincar transformations. Various adaptations of CCA aim to enhance robustness, such as Singular

---

Vector Canonical Correlation Analysis (SVCCA) 48 or to decrease sensitivity to perturbations using methods like Projection- Frojettion-Weightedd Canonical Correlation Analysis (PWCCA) [3
Closely
related to these approaches, Centered Kernel Alignment (CKA) 23 measures the similarity between latent spaces while ignoring orthogonal transformations. However, recent research CKA is sensitive to local shifts in the latent space.
10 reveals that
We propose to leverage LFMs as a tool measure the similarity, or how much two spaces differ from an isometry w.r.t. to the metric that has been used to construct the graph.
Representation alignment. Complementary to measuring the similarity of distinct representational spaces, several methods have been proposed optimize for a transformation The concept of latent space communication, introduced by [39], builds on the hypothesis that latent spaces across neural networks, varying random seed initialization to architecture or even data modality, are intrinsically compatible. This notion is supported by numerous empirical studies 9 L5 R], with the phenomenon being particularly evident in The core idea is that relations between data points (i.e., distances according to some metric) are preserved across different spaces because the high-level semantics of the data are the same and neural networks learn to encode them similarly [19]. With this "relative representation", the authors show that it is possible to stitch [29] together model components coming from different models, with little no additional training, as long
align them [.]
38 30 23
1 55
3
58
27
29
large and wide models [1
33
partial correspondence of the spaces involved is known. Indeed, [26 32 35 37 show that a simple linear transformation usually enough to effectively map one latent space into another, measuring the mapping performance via desired downstream tasks.
With LFMs, we change the perspective from merely relating samples of distinct latent spaces to relating function spaces defined on the manifold that the samples approximate, showing that processing information in this dual space is convenient as it boosts performance while also being interpretable.
Functional Maps. The representation we propose is directly derived from the functional maps framework for smooth manifolds introduced in the seminal work by [44). This pioneering study proposed a compact and easily manipulable mapping between 3D shapes. Subsequent research has aimed enhancing this framework. For instance, [41] introduced regularization techniques to improve the informativeness of the maps, while (34) developed refinement methods to achieve more accurate mappings. The functional map framework has been extended well outside the 3d domain, for example, in [60] and [17,, who applied the functional framework to model correspondences between graphs, and in [46. who demonstrated its utility in graph learning tasks. In particular, they have shown that the functional map representation retains its advantageous properties even when the Laplace basis is computed on a graph.
Inspired by these advancements, our work leverages the functional representation of latent spaces of neural models. We demonstrate how this representation be easily manipulated highlight similarities and facilitate the transfer of information between different spaces, thereby extending the applicability of the functional maps framework to the domain of neural latent spaces.
3
Method
3.1
Background
This section provides the basic notions to understand the framework of functional maps applicd to manifolds. We refer to (45) for comprehensive overview.
Consider two manifolds M and N equipped with a basis M (respectively V Any squared integrable function f : M - I can be represented as a linear combination of basis functions & f F a. a MMM Given bijective correspondence T M A between points on these manifolds, for any realevalued function f M + R, one can construct a corresponding function g ::N= + R such that y = fo T-1 other words, the correspondence between two function spaces Tp FR - F(N, RR. Such a mapping is linear 44 and can be represented as matrix C such that for any function f represented as vector of coefficients a, we have T (F(a) Ca.
defines a mapping
3

---

The functional representation is particularly well-suited for map inference (i.e., constrained optimiza- tion problems). When the underlying map T (and by extension the matrix C) is unknown, many natural constraints on the map hecome linear constraints in its functional representation. In practice, the simplest method for recovering an unknown functional map to solve the following optimization problem:
arg min| CA - +
where and B are sets of corresponding functions, denoted as descriptors, expressed in the bases on M and N. respectively, and p(C)) represents additional constraints deriving from the properties of the matrix C 45). When the manifolds M and N are approximately isometric and the descriptors are weel-preserved by the (unknown) map, this procedure provides a good approximation of the underlying map. In cases where the correspondence T is encoded as a permutation matrix P. the functional map can be retrieved as C = و P M where P and yy are the bases of the functional spaces F(M, and F(V, IR), respectively, and denotes the Moore-Penross pseudo-inverse.
3.2 Latent Functional Maps
3.2.1
Setting
We consider deep neural networks / == S1 o f ...fo where each layer Ji is associated to a representational space X; corresponding to the image of fi- In the following we're we're gonna drop the subseript i, when the layer considered is clear form the context. We assume that elements x E X sampled from latent manifold .M. Considering pairs of spaces (X,J), and corresponding manifolds M.N our objective is to characterize the relation between them by mapping the space of functions F(M) to F(N). Our framework can be summarized in three steps, which we are going to describe in the following sections: (i) how to represent M from a sample estimate X, by building a graph in the latent space (ii) how to encode any known preserved quantity between the two spaces by defining set of descriptor function for each domain (iii) optimizing for the latent functional map between M and M. An illustrative description of our framework is depicted in Figure
Building the graph. To leverage the geometry of the underlying manifold, model the latent space of a neural network building& a symmetric k-nearest neighbor (k-NN) graph [31] Given a set of samples X [I1,--. ... -.
J. J. we construct undirected weighted graph G (X, E, W) with
nodes X. edges E, and weight matrix W. The weight matrix is totally characterized by the choice of a distance function d: X x X + Suitabe choices for include the L2 metric or the angular distance. Edges E are defined as E = {(I,, E X x X 24 <-k 2y or æ ~ 24 where I; ~ Ty indicates that zy is among the k nearest neighbors of ai. The weight matrix W E R25"" assigns weight w(x;, being some monotonic function. Next, we define the associated weighted graph Laplacian operator LG = I- D D--I/2 - 1/2,, where is the diagonal degree matrix with entries D.,, = Σド= W LG is positive semi-definite, self-adjoint operator (57)), therefore, it admits eigendecomposition Le = AG T.. where AG is diagonal matrix containing the eigenvalues, and & is matrix whose columns are the corresponding cigenvectors. The eigenvectors form an orthonormal basis for the space of functions defined on the graph nodes (i.c., & D6 1). We give comprehensive details on the choice of l: in Appendix A.
ox(((x,,,,)) to each edge (.;, 2r)) E E, and W., 0 otherwise, with
Choice of descriptors. To define the alignment problem between pair of spaces X, will introduce the notion of descriptor funcctooor We define as descriptor function any real valued function f: G + R defined on the nodes of the graph. Informally deseriptors should encode the information which is shared between X' and Y, either explicitly or implicitly. We categorize descriptor fiunctions into supervised, weakly supervised and unsupervised descriptors. For the former we assumed to have access to partial correspondence between the domains X' Y, defined as bijective mapping Ts : Ax + Ay where Ax C X, Ay V. Then descriptors takes the form of distance functions dx : X' x Ax + Rt, dy ) x (A(x + R+. As an example considering the geodesic distance (shortest path distance on the k-NN graph) from each node of the graph to the nodes in the anchor set, is an instance of supervised descriptor. In the case of weakly supervised descriptor, we assume to have access to an equivalence relation defined Tw Ax Ay h {0, 1}. Example of these are multi-to-multt mappings, like mappings between labels. Unsupervised descriptors are quantities that

---

depends on the topology and the geometry of the graph itself and they don't t rely on any pre-given correspondence or equivalence relation. An example of this the Heat Kernel Signature [53] node descriptor, based on heat diffusion over the manifold. We give examples of descriptors and ablation in Appendix C.2
Computing LFMs. We now describe the optimization problem to compute latent functional map between X and y. We model each space using subset of training samples X (Ii,..... ,In and Y = [V,..... . and build the k-NN graphs Gx and Gy from these samples, respectively. For each graph, we compute the graph Laplacian LG and derive the first & eigenvalues Ag and eigenvectors G [ø1..... Φb). which serve as the basis for the function space defined on the latent spaces.
Given the set of descriptors Fxx = "/""x....,,a [Gx] and FGy = " }. we consider the optimization problem defined in Equation [ and incorporate two regularizers which enforce commutativity for the Laplacian and the descriptors, respectively, with the map C:
arg min Main||Ca |C
Far l apc(C) BCr(C)
where - are the spectral coefficients of the functions Fc. PC and Pj are the Laplacian and descriptor operator commutativity regularizers respectively, akin 44. We give full details on the regularizers in Appendix Once we have solved the optimization problem defined in Equation
2
we refine the resulting functional map C using the spectral upsampling algorithm proposed by as detailed in Appendix A.3
3.3
LFMs as a similarity measure
Once computed, the functional map C can serve measure of similarity between spaces. The reason is that for isometric transformations between manifolds, the functional map is volume preserving (see Thm 1.1 in 44 and this is manifested in orthogonal C. By defining the inner product between functions h2, by
F(M) as (/1,hz) SM N1 (3r (1))(()))) it holds that (h1, h2) (h1 h2) when the map preserves the local area, where h denotes the functional representation of h. In other words, when the transformation between the two manifolds is an isometry. the matrix C C will be diagonal. By measuring the ratio between the norm of the off diagonall elements of C C and the
norm of the full matrix, we can define a measure of similarity sim(X, Y) = | CII In ||C CI
Appendix A.4 prove that this similiarity measure is proper distance the space of functional maps, allowing to compare measurements from collections of spaces. Furthermore, this quantity is interpretable; the first eigenvector of CTC can act as a signal to localize the area of the target manifold where the map has higher distortion 43 as we show in the experiment in Figure 3a
3.4 Transfering information with LFM
The map C computed between two latent spaces can be utilized in various ways to transfer information from one space to the other. In this paper, we focus on two methods: (i) Expressing arbitrary points in the latent space as distance function on the graph and transferring them through the functional domain; (ii) Obtaining a point-to-point correspondence between the representational spaces from the LFM (see the first step in Appendix A.3). starting from none to few known pairs, and leverage off-the-shelf methods to learn a transformation between the spaces. Additional strategies could be explored in future work.
Space of Functional Coefficients. The space of functional (or spectral) coefficients offers an alternative representation for points in the latent space X. Using the equation fc &o any function fa E F(G, R) can be uniquely represented by its functional coefficients fa. We leverage this property represent any point x E X' as distance function fa F(G, R) from the set of points XG. which correspond to the nodes of the graph G. The functional map C between two latent spaces X' and ) aligns their functional representations, enabling the transfer of any function from the first space to the second. This functional alignment can be used similarly to the method proposed by [39 establish shared space where the representational spaces X' and
aligned.
Finding correspondences. As explained Section 3, the functional map C represents the bijection T' in a functional form. [4 demonstrated that this bijection can be retrieved as point-to-poin point
5

---

CCA (41.4 %)
CKA (99.6 %)
LFM (99.8 %
Figure 2: Similarity across layers Similarity matrices between internal layer representations of CIFAR10 comparing our LFM-bbsedd similarity with the CCA and CKA baselines, averaged across 10 models, For each method, we report the accuracy scores for matching the corresponding layer by maximal similarity.
map by finding the nearest neighbor for each row Pay C in &xx This process can be efficiently implemented and scaled up using kd-tree structures approximation strategies [21 20) Starting from an empty set or few known correspondences (anchors) between the two spaces X and V we extend the correspondence to the entire set of nodes X and Y. This extended set of anchors then be used to fit an arbitrary transformation between the latent spaces, for example an orthogonal mapping [32]. In our experiments, we demonstrate that by using very small number of anchors (typically < 50), we can retrieve optimal transformations that facilitate near-perfect stitching and retrieval tasks.
4 Experiments
In this section, present series of experiments designed evaluate the effectiveness of the Latent Functional Map (LFM) framework. We explore its application across various tasks, including similarity evaluation, stitching, retrieval performance, and robustness perturbations in latent space. By comparing LFM to existing methods under different conditions. we aim to demonstrate its versatility and superior performance aligning and analyzing neural network representations. Additional ablations and qualitative visualizations on the choice of distance metric to construct the graph and descriptors selection are reported in the Appendix
4.1 Analysis
We demonstrate the benefits of using latent functional maps for comparing distinct representational spaces, using the similarity metric sim Y) defined in Section 3.3
Experimental setting In order to validate experimentally LFMs can serve as good measure of similarity between distinct representational spaces, train 10 CNN models (the architecture is depicted in Appendix on the CIFAR-10 dataset (24),
the same sanity check as 23). We
changing the initialization seed. We compare their inner representations each layer, excluding the logits and plot them as a similarity matrix, comparing with Central Kernel Alignment (CKA)
measure (33 and Canonical Correlation Analysis (CCA) [8 48 We then measure the accuracy of identifying corresponding layers across models and report the results comparing with CKA and CCA as baselines. For CCA, apply average pooling the spatial dimensions the embeddings the internal layers, making it more stable numerically and boosting the results for this baseline compared what was observed in 23
Result analysis Figure 2 shows that our LFM- similarity measure behaves correctly as CKA does. Furthermore, the similarities are less spread around the diagonal, favoring slightly higher accuracy score identifying the corresponding layers across models.
While CKA (Centered Kernel Alignment) is widely used similarity metric in deep learning, recent research by [10 has shown that it can produce unexpected or cunter-intuitive results in certain situations. Specifically, CKA is sensitive to transformations that preserve the linear separability of

---

Sinlarks
6
Dssaance
(a) CKA LFM similarity
(b) Localized area distortion
Figure 3: Robustness of LFM similarity Left: Similarity scores as function of perturbation strength: while the CKA baseline degrades, our LFM similarity scores are robust to perturbations that preserve linear separability of the space. Right: Visualization of area distortion of the map by projecting the first singular component of the LFM in the perturbed space: the distortion localizes on the samples of the perturbed class, making LFM similarity interpretable.
two spaces, such as local translations. Our proposed similarity measure is robust to these changes and demonstrates greater stability compared to CKA.
Experimental setting We compute the latent representations from the pooling layer just before the classification head for the CIFAR10 train and test sets. Following the setup in [10], we train Support Vector Machine (SVM) classifier on the latent representations of the training samples to find the optimal separating hyperplane between samples of one class and others. We then perturb the samples by translating them a direction orthogonal to the hyperplane, ensuring the space remains linearly separable. We measure the CKA and LFM similarities functions of the perturbation vector norm, as shown in Figure 3a| In the accompanying plot on the right, we visualize the area distortion of the map by projecting the first singular component of the LFM C into the perturbed space and plotting it on 2d TSNE [56] projection of the space.
Result Analysis We start by observing that when the latent space is perturbed in way that still preserves its linear separability, it should be considered identical from a classification perspective, as this does semantically affect the classification task. Figure 33 shows that while CKA degrades as a function of perturbation intensity, the LFM similarity remains stable to high scores. To understand function of the samples by projecting the
this difference, we can visualize the area distortion first singular component of C onto the perturbed space. In Figure 3b) we use t-SNE [56] to project the perturbed samples and the distortion function into 2D. The visualization reveals that distortion is localized to the samples corresponding to the perturbed class.
4.2 Zero-shot stitching
In latent communication tasks, common challenge is combining an encoder that embeds data with decoder specialized in a downstream task, such as classification or reconstruction -tioos process is often referred as stitching. Previous works, like (29 and 2 have used trainable linear projections. known as stitching layers, to enable the swapping of different network components. However, [39 introduced the concept of zero-shot stitching, where neural components can be marged without any additional training procedure. It's important to note that while (39) still required to trained decoder module once processing relative representations before stitching could be performed, method is fully zero-shot, with no need for additional training. In the following experiments, stitching is conducted without any training or fine-tuning of the encoder or decoder, adhering strictly to zero-shot fashion.
We consider four pre-trained image encoders (see Appendix B.2 for details) and stitch their latent spaces to perform classification using Support Vector Machine (SVM) five
Experimental Setting

---

(a) CIFARIO0 coarse
(b) CIFARI00 fine
(c) ImageNet
(d) CUB
(c) MNIST
AgNews
Figure 4: Stitching results. Accuracy performance in stitching between image encoders trained 6 datasets, comparing orthogonal transformation (Ortho) and LFM+Ortho at varying anchor counts. Also shown is LFM+Ortho (Labels), which uses the dataset labels instead of anchors. Results presented with mean accuracy values reported on each box.
D1
D2
Absol
Ontho
LFM
Bi D1
D2
Absol.
Ortho
LFM
EI D1
12 D2
Abool.
Ortho
LFM
5
5
5
5
5
6
6
O
7
7
7
7
1
(a) MNIST
(b) FashionMNIST
(c) CIFAR-10
Figure 5: Qualitative results stitching. We combine the encoder decoder modules of two convolutional autoencoder (El DI, E2 D2) using three different approaches: no transformation (Absol.), the orthogonal transformation from (32 (Ortho), the orthogonal transformation augmented with LFM (LFM).
different datasets: CIFARI00 24 with coarse and fine-grained labelling, ImageNet 11] Caltech- UCSD Birds-200-2011 (CUB) 59
MNIST [12], AgNews (65]. These datasets chosen to encompass both large-scale, complex datasets like ImageNet and variety of modalities, including text and images. To evaluate the effectiveness of integrating the functional map, we extend the correspondences to determine an orthogonal transformation [32] between the latent spaces. For each encoder, we compute a graph of 3,000 points with 300 neighbors per node. We optimize the problem in Equation 2 using the first 50 eigenvectors of the graph Laplacian and consider two different descriptors: the distance functions defined from the anchors (LFM+Ortho) and the labels (LFM+Ortho (Labels)). For each dataset class, the latter provides an indicator function with if point belongs to the class and 0 otherwise. This descriptor does not require any anchor as input, representing pioneering example of stitching requiring additional information beyond the dataset.
Result Analysis Figure shows the accuracy results for all possible combinations of encoder stitching. The latent functional map (LFM+Ortho) consistently outperforms other methods across all datasets, particularly with low number of anchors. Even without anchors, using label descriptors (LFMM+rrtho (Labels)) achieves strong performance across different labeling schemes. The orthogo- nal transformation computed directly from the anchors (Ortho) only matches LFM's performance

---

Fastlext
Word2Vee
Aligned
-0.2
-0.0
Winsoo
00
6.1
-92
(b) Latent space
FastText
Aligned
6.5
0,00
Orth
LF Spots
-005
213
-8:5 -030 ab -1.2
(a) MRR score at increasing number of anchors
(c) Functional space
Figure 6: Retrieval of word embeddings. We compare the retrieval performance of the functional map framework with state-of-the-an models as the number anchors increases. The left panel shows the Mean Reciprocal Rank (MRR) across different numbers of anchors. The right panels depict the first two components of PCA for a subsample of the latent space (b) and the functional space (c), both before and after alignment using the functional map.
when more than 50 anchors are used, at which point LFM's effectiveness is constrained by the number of cigenvectors. Comparing the CIFARI00 coarse and fine-grained labeling, the fine-grained task is more complex due to the larger number of classes, making the estimation of the map more challenging. This complexity is especially evident when aligning self-supervised vision models with classification-basedd ones (e.g., DINO vs. VIT), where the training strategies differs.
This experiment shows that the latent functional map highly effective when few anchors are available (< 50). It significantly enhances performance in zero-shot stitching tasks, outperforming direct orthogonal transformations at low or no anchor counts. This suggests that the latent functional map method provides robust means of aligning latent spaces with minimal correspondence data, making it a valuable tool for tasks requiring the integration independently trained models.
4.2.1 Qualitative example
In Figure 5 we present qualitative experiments on the MNIST, FashionMNIST (64) and CIFAR- 10 datasets, focusing visualizing the reconstructions of stitched autoencoders. For these experiments, we trained convolutional autoencoders using two different seeds and stitched their encoder and decoder modules together. As a baseline, we used no transformation (Absol.). where the latent representation from the first encoder is directly input into the second decoder. We then compared this baseline with the orthogonal transformation from (32] (Ortho) and its LFM-augmented version (LFM). For both our method and 32 we used 10, and correspondences for the MNIST, MNSSi and CIFAR-10 datasets, respectively. Our method consistently produced superior reconstruction quality compared to both the baseline and the method from [32
4.3
Retrieval
We extend our analysis to the retrieval task, where look for the most similar embedding in the aligned latent space.
Experimental Setting We consider two different English word embeddings, FastText and Word2 [36]. Following the approach of (39], we extract embeddings of 20K words from their shared vocabulary using pre-trained models. We use 2K random corresponding samples construct the k-NN graphs and evaluate the retrieval performance on the remaining 18K word embeddings. We test two settings in our experiments:
1. Aligning functional coefficients (LFM Space).

---

2. Computing an orthogonal transformation using the correspondences obtained by the func- tional map (LFM+Ortho).
For this experiment, we construct K-NN graphs with neighborhood size of 300 and compute the functional map using the first 50 eigenvectors. We evaluate the methods' performance using the Mean Reciprocal Rank (MRR), as detailed in Appendix [.4] Our functional map methods are compared with the method proposed by [39 (Relatives) and the orthogonal transformation method proposed by [32 (Ortho). We choose to fit the same orthogonal transformation on top of the correspondence found by LFM to make the comparison the fairest possible, although in principle any off-the-shelf methods could be used to estimate the transformation once the new correspondence found. We illustrate this versatility in Table of the Appendix, where LFM is combined with other methods in the same task.
Result Analysis Figure 6 shows the performance of these methods the number of anchors increases. Numerical results are detailed Ta the Appendix. The functional map significantly improves performance with just 5 anchors, achieving an MRR of over 0.8. As the number of anchors increases, the performance of competing methods improves but still falls short FMAP+Transfor at 300 anchors, which reaches an MRR of 0.99. Interestingly, the performance of the functional map methods does not improve beyond 5 anchors, suggesting that this number of anchors is sufficient to achieve optimal functional map between the spaces. These findings are further supported by Table 3 in the Appendix, where LFM consistently achieves superior MRR scores, particularly with fewer anchors, outperforming other baselines such [29]. In Appendix C.3) we analyze how the results improve as the number of eigenvectors used to compute the functional map increases. Notably, the MRR score drastically increases to over 0.6 with more than 25 eigenvectors.
These results confirm that the latent functional map is a valuable tool in settings with little knowledge about correspondences. It significantly enhances retrieval performance with minimal number of anchors, making it an efficient approach for aligning latent spaces. Moreover, its performance can be improved using higher number of eigenvectors. In particular, our framework comprises at the same time (i) an interpretable similarity measure, (ii) a way to find correspondences, and (iii) solving for an explicit mapping between different spaces in single framework, differently from [39 and 33
which attempt to solve just the latter implicitly.
Conclusions
In this paper, we explored the application of latent functional maps (LFM) to model and analyze the relationships between different latent spaces. Our approach leverages the principles of spectral geom- etry, providing robust framework for comparing and aligning representations across various models and settings. By extending the functional map framework to high-dimensional latent spaces, we offer versatile method for comparing distinct representational spaces, finding correspondences between them in both unsupervised and weakly supervised settings (few or no anchors) and transferring information across different models.
Limitations and future work The performance of LFM can be sensitive to the number of eigenvec- tors used, as observed in some of our experiments (see Appendix C.3). Finding the optimal number of eigenvectors without extensive experimentation remains an open problem. The current framework assumes a correspondence between the domains, and is not directly adapted in order to deal with partial mappings, where just a subset of one domain can be mapped to other. Extending the results in our neural representation setting, is a promising way overcome the full correspondence assumption. Future research can further explore the potential of LFM in other domains and tasks, and modalities. The versatility of the functional representation can be further explored to define new ad-hoc constraints and regularizers for different settings. In particular, ts performance in fully unsupervised settings, while promising still an area requiring further research and improvement, In conclusion, the Latent Functional Map framework offers a novel and effective approach to under- standing and utilizing neural network representations, promising significant advancements both theoretical and practical aspects of machine learning.
49 47
10

---

Acknowledgments and Disclosure of Funding
MF is supported by the MSCA IST-Bridge fellowship which has received funding from the European Union's 's Horizon 2020 research and innovation program under the Marie Sklodowska-Cure grant agreement No 101034413. ER and VM are supported by the PNRR MUR project PE00000L3-FAIR, MP is supported by the Sapienza grant "Predicting and Explaining Clinical Trial Outcomes", prot. RG12218166FA3FIS
References
[1] D. Alvarez-Melis and T. S. Jaakkola. Gromov-wasserstein alignment of word embedding spaces. arXiv preprint 100.:08800000.... 2018.
[2] Y. Bansal, P. Nakkiran, and B. Barak. Revisiting model stitching to compare neural representa- tions. Advances in neural information processing systems, 34:225-236 2021.
[3] Barannikov, Trofimov, N. Balabin, and Burnaev. Representation topology divergence: method for comparing neural network representations. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1607-1626. PMLR, 2022.
[4] Y. Bengio, A. Courville, and P. Vincent. Representation Learning: Review and New Perspec- tives. ArXiv preprint, abs/1 200.5558, 2012.
[5] Bojanowski, E. Grave, A. Joulin, and T. Mikolov. Enriching word vectors with subword the association for computational linguistics, 5:135-146, 2017. information. Transactions the
[6] L. Bonheme and M. Grzes. How do variational autoencoders learn? insights from representa- tional similarity. ArXiv preprint, abs/2205, .08399. 2022.
[7] Calder and N. G. Trillos. Improved spectral convergence rates for graph laplacians on e-graphs and k-nn graphs. Applied and Computational Harmonic Analysis, 60: 10:12231 2022.
[8] I. Cannistraci, L. Moschella, M. Fumero, V. Maiorca, and E. Rodolà. From bricks to bridges: Product invariances to enhance latent space communication. In International Conference on Learning Representations, 2024.
[9] T. A. Chang, 7. Tu, and B. K. Bergen. The geometry of multilingual language model represen- tations. ACI, 2022.
[10] M. Davari, S. Horoi, A. Natik, G. Lajoie, G. Wolf, and E. Belilovsky. Reliability of cka as similarity measure in deep learning. arXiv preprint arXlv:2210.16156. 2022.
[11] Deng, W. Dong, Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[12] L. Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 1946):141-14,, 2012.
[13] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
[14] M. Fumero, F. Wenzel, L. Zancato, A. Achille, E. Rodolå, Soatto, B. Schölkopf, and F. Locatello. Leveraging sparse and shared feature activations for disentangled representation learning. A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors. Advances in Neural Information Processing Systems, volume 36, pages 77682-27698. Curran Associates, Inc., 2023.
[15] Guth, B. Ménard, G. Rochette, and S. Mallat. rainbow deep network black boxes. arXiv preprint
2023,

---

[16] J. V. Haxby, M. I. Gobbini, M. L. Furey, A. Ishai, J. L. Schouten, and P. Pietrini. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293(5539):2425-22 24330, 2001.
[17] J. Hermanns, A. Tsitsulin, M. Munkhoeva, A. Bronstein, D. Mottin, and P. Karras. Grasp: Graph alignment through spectral signatures. In Asia-Pacific Web (APWeb) and Web-Age Information Management (WAMM) Joint International Conference on Web and Big Data, pages 44 44-52. Springer, 2021.
[18] H. Hotelling. Relations between two sets of variates. Breakthroughs in statistics: methodology and distribution, pages (62-190, 1992.
[19] M. Huh, B. Cheung, T. Wang, and P. Isola. Position: The platonic representation hypothesis. In Forty-first International Conference on Machine Learning. 2024.
[20] P. Indyk and R. Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirlieth annual ACM symposium on Theory of computing, pages 604-613, 1998.
[21] J. Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, ((3):335-547) 201,
[22] M. Klabunde, T. Schumacher, M. Strohmaier, and F. Lemmerich. Similarity of neural network models: A survey of functional and representational measures. arXiv preprint arXiv:2305.06329, 2023.
[23] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519-3529. PMLR, 2019.
[24] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical Report, 2009.
[25] A. Laakso and G. Cottrell. Content and cluster analysis: Assessing representational similarity in neural systems. Philosophical Psychology, 13:47 - 76, 2000.
[26] Z. Lähner and M. Moeller. On the direct alignment of latent spaces. In Proceedings of UniReps: the First Workshop on Unifying Representations in Neural Models, volume 243 of Proceedings of Machine Learning Research, pages 158-169 169. PMLR, 2024.
[27] G. Lample, A. Conneau, M. Ranzato, L. Denoyer, and H. Jégou. Word translation without par- allel data. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[28] G. Lample, A. Conneau, M. Ranzato, L. Denoyer, and H. Jégou. Word translation without parallel data, International conference on learning representations, 2018.
[29] K. Lene and A. Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 991-999. IEEE Computer Society, 2015.
[30] Y. J. Yosinski, J. Clune, H. Lipson, and J. E. Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Y. Bengio and Y. LeCun, editors, 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[31] M. Maier, M. Hein, and U. von Luxburg. Optimal construction of -niiero graphs for identifying noisy clusters. Theoretical Computer Science, 114919):1749-176: 2009. Algorith- mie Learning Theory.
[32] V. Maiorea, L. Moschella, A. Norelli, M. Fumero, F. Locatello, and E, Rodolà. Latent space translation via semantic alignment. Advances in Neural Information Processing Systems, 36. 2024.
[33] R. Mehta, V. Albiero, I.. Chen, Evtimov, Glaser, Z. Li, and Hassner. You only need a good embeddings extractor to fix spurious correlations. ArXiv, 2022.
12

---

[34] S. Melzi, J. Ren, E. Rodolà, A. Sharma, P. Wonka, M. Ovsjanikov, et al. Zoomout: spectral upsampling for efficient shape correspondence. ACM TRANSACTIONS ON GRAPHICS, 38(6):1- 14, 2019.
[35] J. Merullo, L. Castricato, C. Eickhoff, and E. Pavlick. Linearly mapping from image to text space. In The Eleventh International Conference on Learning Representations, 2023.
[36] T. Mikolov, Q. V. Le, and Sutskever. Exploiting similarities among languages for machine translation. arXiv preprint 1rXiv:1309.400 2013.
[37] M. Moayeri, K. Rezaei, M. Sanjabi, and S. Feizi. Text-to-concept (and back) via cross-model alignment. In International Conference on Machine Learning, pages 25037-25060. PMIR, 2023.
[38] A. Morcos, M. Raghu, and S. Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 31, 2018.
[39] L. Moschella, V. Maiorca, M. Fumerr, A. Norelli, L. Francesco, E. Rodola, al. Relative representations enable zero-shot latent space communication. In International Conference on Learning Representations, 2023.
[40] Y. Movshovitz-Atiass A. Toshev, T. K. Leung, S. loffe, and S. Singh. No fuss distance metric learning using proxies. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 360-368. IEEE Computer Society, 2017.
[41] D. Nogneng and M. Ovsjanikov. Informative descriptor preservation via commutativity for shape matching. In Computer Graphics Forum, volume 36, pages 259-267. Wiley Online Library, 2017.
[42] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.07199, 2023.
[43] M. Ovsjanikov, M. Ben-Chen, F. Chazal, and L. Guibas. Analysis and visualization of maps between shapes. In Computer Graphics Forum, volume 32, pages 135-145. Wiley Online Library, 2013.
[44] M. Ovsjanikov, M. Ben-Chen, J. Solomon, A. Butscher, and L. Guibas. Functional maps: a flexible representation of maps between shapes. ACM Transactions on Graphics (ToG), 31 3144):1-11, 2012.
[45] M. Ovsjanikov, E. Corman, M. Bronstein, E. Rodolå, M. Ben-Chen, L. Guibas, F. Chazal, and A. Bronstein. Computing and processing correspondences with functional maps. In SIGGRAPH ASIA 2016 Courses, pages 1-60. ACM, 2016.
[46] M. Pegoraro, R. Marin, A. Rampini, S. Melzi, L. Cosmo, and E. Rodolà. Spectral maps for learning on subgraphs. In NeurlPS 2023 Workshop on Symmetry and Geometry in Neural Representations, 2023.
[47] E. Postolache, M. Fumero, L. Cosmo, and E. Rodolà. A parametric analysis of discrete hamiltonian functional maps. Computer Graphics Forum, 3905):03-1118, 2020.
[48] M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
[49] E. Rodolà, L. Cosmo, M. M. Bronstein, A. Torsello, and D. Cremers. Partial functional correspondence, 2015.
[50] G. Roeder, L. Metz, and D. Kingma. On linear identifiability of learned representations. In International Conference on Machine Learning, pages 9030-9039. PMLR, 2021.
[51] G. Somepalli, L. Fowl, A. Bansal, P. Yeh-Chiang, Y. Dar, R. Baraniuk, M. Goldblum, and Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. IEEE CVF, 2022.
13

---

[52] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6800 2014.
[53] Sun, M. Ovsjanikov, and L. Guibas. concise and provably informative multi-scale signature based heat diffusion. In Computer graphics forum, volume 28, pages 1383-1392. Wiley Online Library, 2009.
[54] D. Ting, L. Huang, and M. Jordan. An analysis of the convergence of graph laplacians. arXiv preprint arXiv::101.554 2011.
[55] A. Tsitsulin, M. Munkhoeva, D. Mottin, Karras, A. M. Bronstein, V. Oseledets, and E. Müller. The shape of data: Intrinsic distance for data distributions. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[56) I.. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 11 2008
[57] U. Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17:395 416, 2007.
[58] I. Vulić, S. Ruder, and A. Søgaard. Are all good word vector spaces isomorphic? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3178-3192, Online, 2020. Association for Computational Linguistics.
[59] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
[60] F-D. Wang, N. Xue, Y. Zhang, G.-S. Xia, and M. Pelillo. functional representation for graph matching. IEEE transactions pattern analysis and machine intelligence, 2019.
[61] L. Wang, L. Hu, J. Gu, Y. Wu, Z. Hu, K. He, and J. Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation, 2018.
[62] K. Weiss, T. M. Khoshgoftaar, and D. Wang. A survey of transfer learning. Journal of Big data, 3:1 3:1 2016.
[63] A. H. Williams, E. Kunz, S. Korblithh and S. Linderman. Generalized shape metrics on neural representations. In M. Ranzato, A. Beygelzimer, Y. Dauphin, Liang, and W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 4738 4750. Curran Associates, Inc., 2021.
[64] H. Xiao, K. Rasul, and R. Vollgraf. mashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.077473 2017.
[65] X. Zhang, J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction gives an overview of the paper's 's scope clearly stating the contributions in a dotted list.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We put our limitations in the conclusion, after analyzing the results of our method.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
The paper does not include new theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In the Experiment and Appendix, report all the information needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We will share the code after the paper acceptance. All the data we use are open-source and publically available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guides/ Staalees//ooeeSSmmimiiionoffee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppllligggddpeeeeeeeeeee Cooee//ddeeeebmimcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We report all the training and test details in the Experiment and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The experiments that are run over multiple settings are reported as error bars that depict the whole distribution of the results obtained.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
In the Appendix we report information about the resources used for our experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics fittpa://neurpp..
Answer: [Yes]
Justification:
We have read and respect the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our method is not tied to particular applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There are no risks posed by our work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the models and data used in our experiments are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20

---

A Latent Functional Maps
A.1 Building the graph
In the following section we give more details about how to construct the graph in the latent space in order to approximate the manifold domain from sample estimate.
Throughout the paper, we assume the eigenvalues (and corresponding eigenvectors) are sorted in non-descending order Ai S A2 - .... An. One may consider subset of eigenvectors, namely those associated with the k smallest eigenvalues, to compactly approximate graph signal, employing techniques akin to Fourier analysis. As demonstrated in multiple recent works (54] 77. the eigenvalues and eigenvectors of the graph Laplacian associated with a k-NN graph approximate the weighted
Aoplace-Beltrami operator, placing us setting similar the original one of [44]. particular in [7] it has been derived optimal bounds for the choices of k. bounds based on the dimensionality of the underlying manifold and the number of sampled points. In our experiments we choose our parameter k according to these estimates. For an estimate of manifold dimensionality we use the latent space dimensionality, which correspond an upper bound to it. In practice, we typically set k 300 with graph in the order of few thousands nodes.
A.2 Additional Regularizers
In Equation
we improve the computation of the functional map by incorporating two additional Laplacian commutativity and descriptor operator commutativity. Both regularizers exploit the preservation of linear functional operators SC F[G, R) - F(G, R). enforcing that the
regularizers:
functional map C commutes with these operators: ||S@C CS ||F
The Laplacian commutativity regularizer, first introduced by [4] is formulated as:
Pc = Ac. C CAGx l1
where Ag represents the diagonal matrices eigenvalues. This regularizer ensures that the functional map C preserves the spectral properties of the Laplacian.
The descriptor operator commutativity regularizer, introduced by [41, extracts more detailed informa- tion from a given descriptor, resulting in a more accurate functional map even with fewer descriptors. The formulation of this regularizer is as follows:
Py(C)= Σ | Y CS x
where SÇ (T are the descriptor operators.
A.3 Spectral refinement
Once we have solved the optimization problem defined in Equation we refine the resulting
functional map C using the spectral upsampling algorithm proposed by [34] We start by considering the matrix C as the submatrix of size k x k of the current LFM estimate. Then we apply iteratively the following two steps:
1. Compute the pointwise map T encoded as matrix II from the current LFM estimate C via solving: arg min C( Cy Gx
)l Vz E X, where क.. correspond to the æ-th column
of the matrix
2. Set C++1
..k T Gx
& k Gy. where i correspond to the k-th column of the matrix +.
A.4 LFM similarity properties
In this section we will demonstrate that the LFM similarity property sim(XX = | CyI?? (C' C)
defined in Section 3.3 is metric, it satisfies, positiveness, simmetry and triangle inequality.
21

---

Namely, for pairs of spaces X, Y:
sim X, Y)
1999
sim X,Y) sim(Y, X)
sim tim(X,2) (X, S sim(X, sYm(Y, 2)
Since CI Coff(C"))ll = II! we will consider sim(X,
C"
Iding C C for simplicity. The former property is trivial as the ration between two norms is positive. For proving simmetry we have that:
sim(X, Y)
C
simmXX Y) |diag( C CFC)
(8)
|diag o &y)l
sim(X, (X, Y)
|1 4y )
for the numerator nator:
(y we trivially have diag Φ 2y) ) = diag( ( For the denomi-
Since 1|
Σ
Bigl 1-PP--Trrie (r(("4)
(10)
V
We have that:
1 2 (r) Φy) "x 1,000xx7001 -
(11)
(12)
(13)
where the last equality is verified for the Trace operator being invariant under cyclic permutations.
For proving triangle inequality will assume that the matrices of eigenvectors full rank (i.e. all eigenvectors are considered). We have to prove the following:
|diag x +z)
diagg < Py )l?
|iaag (a0gg((fff ( +z)
S 2
(14)
< (2)))
[1 x 2y 1
1[ (2)
|diag
|iiag(
T Ty 1
|daa ( 2
(15)
|| { 2 1|
[ +
Using the fact that the Frobenius norm invariant under multiplication by orthogonal matrices the क
denominator of each term in Ed 1414 [|(
> 1
11 Tr xx
Tr[Id]
N. Then:
diagg T 2 ))
|diag( o o 1|
Idiag(
( + ))
(16)
N
N
N
Since |diag Y) than the LHS.
< |IX |
for any X, Y then the RHS of Eq,
will always be smaller
B
Experimental details
B.1
Architecture Details
All non-ResNet architectures are based on AII-CNN-C 52
B.2
Pre-trained models
In Section 4.2
we used four pretrained models: 3 variations of [13] "google-vit-base-pptee 16-224,,
waw.-i-largg-patch------ siwKaaals-vitt-smll-paaaaa (6-22")))224) and the model proposed by
42
facebook-dinoy2-bass
22

---

Tiny-10
3 x 3 conv. 16-BN-ReLu x2
conv. 32 stride 2-BN-ReLuu
3 X conv. 12-BBNReLLu x
3 X 3 conv. 64 stride 2-BN-ReLu
3 X 3 conv. 64 valid padding-BN-Rel..
1 conv. 64-BN-ReLu Global average pooling Logits
Table
B.3 Parameters and resources
In all our experiments we used gpu rtx 3080ti and 3090. In order to compute the eigenvector and functional map graph of 3k nodes employ not more than 2 minutes.
B.4 Mean Reciprocal Rank (MRR)
Mean Reciprocal Rank (MRR) a commonly used metric evaluate the performance of retrieval systems [39]. It measures the effectiveness of system by calculating the rank of the first relevant item in the search results for each query.
To compute MRR, we consider the following steps:
1. For each query, rank the list of retrieved items based on their relevance to the query.
2. Determine the rank position of the first relevant item in the list. If the first relevant item for query is found rank position rs, then the reciprocal rank for that query is A.
3. Calculate the mean of the reciprocal ranks over queries. If there are Q queries, the MRR is given by:
Q
MRR
(17)
Here, r; the rank position of the first relevant item for the i-th query. If a query has no relevant items in the retrieved list, its reciprocal rank is considered to be zero.
MRR provides single metric that reflects the average performance of the retrieval system, with higher MRR values indicating better performance.
C Additional Results
In this section, we provide additional results and ablation studies that further support and expand upon the experiments presented in Section
C.1 Functional maps structure
In Section 3.2.1 we have shown that latent graphs can be constructed using different distance metrics. While angular distance was primarily used in our experiments, we now highlight its effectiveness and the resulting functional maps.
In Figure we present visualization of the structure of the functional map in synthetic setting. The experiment was conducted as follows: given an input set X consisting of test embeddings extracted from MNIST, we aimed to observe the degradation of the functional map structure as the space is perturbed. The perturbation involved an orthogonal transformation combined with additive Gaussian noise at increasing levels. the first row, the functional maps were computed from k-nearest neighbor (knn) graphs using the angular distance metric, while in the second row, knn graphs were constructed using the 1.2 distance metric. Below each functional map, the LFM similarity score and MRR retrieval scores are displayed.
23

---

Angular
Noise level 0.5
Angular
distance
MRR
1.00
0.99
0.99
0.99
Similarity
1.00
0.50
0.55
0.49
12 distance
MRR
1.00
0.81
0.04
0.03
Similarity
1.00
0.6
0.15
0.13
(a) Functional maps structure
(b) Scores
Figure 7: Functional maps structure at increasing level noise. Given a set of MNIST embeddings, plot the degradation of the functional map structure the space is perturbed. We compare the graph built with two different metrics (Angular, L2) and report MRR and LFM similarity score.
0.7
23
10
10
10
23
20
20
32-
20-
42-
42-
40
20
0.949
0.949
0.921
0.044
0.011
Figure 8: Descriptors ablation. We compare the latent functional maps computed on MNIST using different descriptors. For each map report the MRR score.
We observed that (i) when noise is absent (first column), the two spaces are isometric, and the functional map diagonal, (ii) constructing the graph with the cosine distance metric more robust to increasing noise, and (iii) the LMF similarity score correlates with the MRR retrieval metric, indicating that more structured functional maps reflect better alignment between spaces.
C.2 Ablation on the choice of descriptors
In Figure we performed an ablation study on the choice of descriptors, comparing supervised descriptors (geodesic distance and cosine distance using 10 correspondences), weakly supervised descriptors (label descriptor), and fully unsupervised descriptors (heat kernel signature (53 and wave
kernel signature (53)).
To conduct this study, we performed a retrieval task on the test embeddings two convolutional autoencoders trained MNIST, which differed by their parameter initialization. We visualized the structure of the functional map and reported the performance in terms of mean reciprocal rank (MRR), observing the following: (i) Geodesic and cosine descriptors performed best (ii) The geodesic distance (shortest path) is a good choice as is agnostic of the metric chosen to build the initial graph, yet provides the result using the metric itself; (iii) The structure of the functional map reflects the performance of retrieval.
C.3 Retrieval
In Table we report the numerical results for the experiment in Figure more transformations from the method [32] orthogonal (Ortho), linear (Linear) and affine (Affine). From the value in the table, we can see that all the methods that involve the latent functional map (LFM) saturate at anchors, reaching top performance. In Figure we show the performance of the latent functional map methods depends the number of eigenvectors used to compute the map. In particular,
24

---

10
LFM+Ortho
LFMGT Orhho
LFM Space LFMGT
Relatives
302
Namber eigenvectors
Figure 9: Retrieval word embeddings for increasing number eigenvectors. We report the MMR score for the methods in Figure 6 with fixed number of anchors (5) and increasing number of eigenvectors.
Table 2: MRR Score for the retrieval of word embeddings. We report the value of the results depicted in Figure audding more kind transformation between spaces (Orthogonal, Lin- ear and Affine.
Table 3: MRR Score for the retrieval of CUB. We report the results on the CUB including the additional baseline
Procustes 29
Number of anchors
Number anchors 50
Method
10
25
50
75
100
150
200
300
Method
Relatives
0.01
0.01
0.05
0.55
0.72
0.79
0.84
0.87
Relatives
0.01
0.07
0.14
Ortho
0.01
0.01
0.01
0.15
0.34
0.60
0.82
0.93
0.97
Linear
0.01
0.01
0.01
0.05
0.26
0.49
0.66
0.77
0.74
0.01
Procrustes
0.00
0.06
0.32
Affine
0.01
0.01
0.01
0.19
0.45
0.64
0.81
0.89
0.95
Ortho
0.00
0.06
0.33
Lincar
0.04
LFM Space
0.01
0.85
0.85
0.85
0.85
0.85
0.85
0.85
0.85
0.85
Procrustes
0.24
LFM+Ortho
0.05
0.24
0.34
LFMeLLinear
0.01
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
LI'M+Liner
0.04
0.16
0.22
LFM4
0.01
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
notice that the performance drastically increases 25 eigenvectors, reaching the same score when using the functional map computed from the ground truth correspondences (LFMGT).
C.3.1 Extended comparison
In Table 3 we added a comparison with multiple baselines: Relative [39 Ortho 32 Linear 28).
Procrustes [29]. With the same foundation models and settings used in Section we perform retrieval task the Caltech-UCSD Birds-200-2011 (CUB) dataset. We demonstrate that LFM consistently superior in performance and can be used on top of any method which computes an explicit mapping between spaces.
C.4 Additional qualitative results
In Figure I0 we show additional qualitative results on theMNIST, FMNIST, Cifar10 datasets, akin to the experiment in Figure 5 in the main manuscript.
25

---

BIDI DI
E2 D2
Absol
Orhe
LFM
EI DI
E2
Absol.
Ortho
LFM
EI
E2 D2
Absul.
Ortho
LFM
__
S
S
S
4
7
7
7
0
O
4
4
4
4
4
4
9
9
7
9
3
3
3
2
2
7
2
(a) MNIST
(b) FashionMNIST
CIFAR-10 (c)
Figure 10: Additional qualitative results on stitching.
26


Reviews:
[{"review_id": "NJJ4t7boTF", "reviewer": "Reviewer_5Rrs", "summary": "The paper tackles the problem of modeling relationships between latent spaces learnt by different models. It proposes using functional maps that have been used in 3D vision and graph matching for the purpose. It does so by approximating the latent space structure using a knn graph constructed using anchor points, on which the functional map (LFM) is defined.\n\nLFM is used to stitch together encoders and decoders from separate CNNs trained independently, where it is shown to perform significantly better and with lesser variability than previous work, while simultaneously using less anchors than them.", "questions": "I would like to know if the gains in performance due to the method still hold for more practical netowrks that have larger, more complex latent spaces like ImageNet", "limitations": "The authors have sufficiently addressed limitations", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "Tackles an important, practically useful problem using a novel approach while showing strong, although limited experimental results.", "weaknesses": "**Writing:** The writing quality in the paper could be significantly improved in terms of:\n\n1. Including more details about the experiments in the main manuscript (defining the goal of the experiment formally, detailing backbones, optimizers used in training, specifying which models/layers are stitched). An example of such a lack of detail is Sec 5.2, L235.\n2. Describing closely related past works [22], [29] in more detail.\n3. Clarity and organization of Sec 3, 4.2.\n\n**Limited Experiments:** The current experimental evaluation seems very limited to models trained on very small datasets, e.g. only CIFAR-100 for the zero-shot stitching despite the relative efficiency of the method. I would like to see if the method performs well on more realistic datasets with more complex latent spaces like ImageNet, or even smaller ones like CUB200.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and suggestions. We are gonna address their questions and concerns in the following and remain available for every additional question.\n\n**Writing quality** \n\nWe have significantly improved the manuscript's clarity, making it self-contained and clear. In particular: \nWe structured the experimental section by adding details about the settings in the experimental setting and adding a smaller subsection depicting the experiment's goal and takeaway. \nIn the experimental section, we added an in-detail description for [29] and [22], making it clear that we are in the most similar setting to have a fair comparison with them.\nWe added more details in section 3 and section 4.2, also adding more information in the appendix, complementary to the one already present in Appendix A, to make everything self contained and avoid pointing the reader to past work. Although we thank and agree with the Reviewer that our manuscript is clearer right now, we would like to respectfully point out that Reviewer yAi6 highlighted writing clarity, with confidence 5.\n\n**Limited Experiments**\n\nWe included multiple new experiments in the paper with the objective of: \n(i) Validating our stitching performance on multiple datasets (CUB, ImageNet,MNIST, AgNews), including suggested datasets and datasets that lead to more complex latent spaces such as Imagenet;\n(ii) validating our retrieval performance on multiple methods, showcasing the applicability of LFM to state-of-the-art methods;\n(iii) perform experiments on multiple modalities to show how the LFM can be applied successfully to different representations.\nThe experiments and their analysis are reported in the attached PDF in the general response. \n\n**ImageNet performance**\n\nAs detailed in the general answer, we show stitching experiments on Imagenet, validating the usefulness of the LFM despite the dataset being complex. We would like to respectfully point out that the performance is also dependent on the architecture, for which we employed pre-trained state-of-the-art vision models for all the datasets. The models have been pre-trained with different objectives (classification, self-supervision)  and on multiple modalities (image, text) and can handle complex and large-scale vision and text datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yZhpgPcWEy", "reviewer": "Reviewer_neES", "summary": "The paper presents a new way to compare neural representations: Latent Functional Maps (LFM). The later one is achieved by 1) building symmetric knn graphs 2) calculating Laplace eigenfunctions 3) calculating optimal mapping between them.\nApplications include: (i) compare different spaces in an interpretable way and measure their intrinsic similarity; (ii) find correspondences between them, both in unsupervised and weakly supervised settings, and (iii) to effectively transfer representations between distinct spaces. \nExperiments with CNNs and word embeddings are provided.\nThe language is fine and the manuscript is well organized, but details of the proposed method and experiments are hard to understand.", "questions": "1. Is the LFM score symmetric?\n2. Is it a metric?\n3. What is a descriptor operator? (line 152)\n4. What is stitching of latent spaces?\n5. Why the method from Moschella et al. is not used for comparison?\n6. Why do you use this particular set of functions:\n\"As a set of corresponding functions, we use the geodesic distance functions\"\n7. line 145, 146: do you have one-to-one correspondence between X and Y ?\n\nMoschella, L., Maiorca, V., Fumero, M., Norelli, A., Locatello, F., & Rodolà, E. (2022). Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430.\n\n**Other:**\n\nI think that the proposed idea is intersting, especially becase it gives not only a similarity score, but tools for scrutiny of embeddings.\nBut the paper has issues with clarity (see above).", "limitations": "none", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "In terms of novelty, to the best of my knowledge, this is a first time when a concept of Latent Functional Mapping is introduced to deep learning. In this paper, it is applied for comparing neural representations.\nDeveloping new scores for comparison of neural embeddings is still an important research topic and could enjoy plantly of applications.\nLFM is robust to translations in a directions orthogonal to a separating hyperplane, in opposite to well known CKA.\nExperiments include modern ViT architectures.", "weaknesses": "1. The notion of Functional Maps is applied \"as is\", without significant modifications. So, from the theoretical perspective, there is no contribution.\n2. Many details of the proposed method and experiment are missing, instead it authors refer to original papers, see below. \nIt makes the manuscript hard to read. In my opinion, the paper must be self-contained.\nSettings of these experiments are not a common knowledge. You can include several small paragraphs introducing them to a reader. Examples:\n\n* line 152: and incorporate regularizers for Laplacian and descriptor operator commutativity, as defined in [31]:\n* line 158: in the other latent space. Once we have solved the optimization problem defined in Equation 2, we refine the resulting functional map C using the algorithm proposed by [24].\n* line 235: We test the use of the latent functional map in the task of zero-shot stitching, as defined in [29],\n* line 185: This functional alignment can be used similarly to the method proposed by [29] to establish a \"relative\" space where the representational spaces X and Y are aligned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and suggestions. We are gonna address their questions and concerns in the following and remain available for any additional questions.\n\n**Theoretical contribution**\nWe would like to respectfully point out that the goal of the paper is to make aware of the representation learning community of the functional map framework, which can have many impactful applications (retrieval, stitching) that were not primarily explored in the geometry processing setting. From the theoretical perspective the dimensionality of the problem is very different. \n\nAccording to the NeurIPS submission guidelines, combining established methods to address new problems is a valuable contribution. Applying well-studied tools with theoretical guarantees can often lead to significant advancements and insights. In our study, we applied the Functional Map framework with minor adaptations, specifically in graph building within the latent space. Surprisingly, this approach already yielded considerable improvements over previous methods, demonstrating its practical efficacy and potential. We believe these findings are noteworthy and beneficial to the community. Furthermore, our results pave the way for future research, where we plan to explore more tailored modifications of the method.\n\n**Self-containment**\n\nWe improved clarity in the paper, in particular: \nline 152: we explained and reported the definition of the operators in Appendix A of the paper in the original submission. We made the link clearer adding references to the main text and details in Appendix. We also incorporated part of it in the main paper to improve the flow of the writing. \nline 158: We reported an in-detail explanation of the zoom-out iterative refinement strategy of [24] in Appendix A. In the main text, we briefly described how the algorithm works and referenced the detailed explanation in the Appendix.\nline 235: We added a self-contained explanation for all experiments, including the definition of zero-shot stitching, which, for clarity, we report below.\nline 185: Adding an in more detailed explanation of the [29], we hope the concept of functional “relative” space is now more clear.\t\t\n\n**LFM similarity score**\n\nWe regard the LFM similarity score as a pseudometric: concerning symmetry,  as the functional map commutes with the laplacian eigenbases of the two spaces (when optimizing for it, this is enforced by the regularizer in equation 3 in appendix A) . This enforces that the inverse of the functional map will correspond to its transpose. The formula for the similarity is invariant to the transpose operator, making the score symmetric.\nConcerning triangle inequality we believe that the score does not satisfy this property as distinct perturbations applied to space may have similar scores.\nThese properties are discussed in the manuscript, and we have included the formal proofs in the Appendix for clarity and completeness.\n\n\n**Descriptor Operator**\n\nAs defined in Appendix A, the descriptor operator expresses the set of descriptor functions  \\mathbf{F}_{G_X} (likewise \\mathbf{F}_{G_Y} ) in eq 2 as an operator in the spectral basis of the graph.  The intuition behind the commutativity constraint in the regularizer eq3 of Appendix A, is for indicator functions of regions of constant values of each pair of descriptors f^{G_X}_i, f^{G_Y}_i to be preserved.\nWe expanded the explanation of the definition, and made the reference to the appendix and the text in the main clearer. \n\n**Stitching**\n\nWe define a generic stitched model as the composition of an encoder model that embeds data with a decoder specialized in a downstream task (classification, reconstruction). The stitching operation is always performed without training or fine-tuning of the encoder/decoder, in a zero-shot fashion. \nIn [29], the notion of zero-shot stitching different neural components was introduced, treating them as frozen black-box modules. Nevertheless we remark that [29] still needed to train a decoder module once in order to process relative representations, before being able to perform stitching, while our method is fully zero-shot.\n\n**Comparison with [29]**\n\nWe compare with [29] in the retrieval experiment in Figure 5 of the paper. We added further comparison in the retrieval experiment on the CUB dataset reported in the general answer for this rebuttal. In the stitching experiments, we do not report comparison with [29]  for two reasons: (i) [29] requires training decoder modules once before performing stitching, while we directly estimate the map between the two latent spaces, without requiring any additional training; (ii) We compare with [22] (under the name of ortho), however, whose performance is superior to the one of [29], as reported in the experimental section in [22]. By demonstrating better performance w.r.t. [22] our method is therefore superior to [29]. \n\n**Choice of descriptor operators**\n\nIn general in our framework descriptor functions can either be semi supervised (e.g. express on each node of the ) weakly supervised (use the indicator functions of labels ) or fully unsupervised ( unsupervised descriptors of the graph, such as the heat kernel signature). \nWe explored different strategies for the choice of descriptor functions on top of the KNN graph, and chose the geodesic distance on the graph as it is agnostic of the choice of the metric used to construct the KNN graph  and provides a good choice in terms of performance.\nWe included an ablation experiment on the choice of descriptor in the PDF of the general answer. We clarified this in the main manuscript and reported the ablation experiment in the Appendix.\n\n**Correspondence between domains**\n\nIn general in our framework we don’t assume to have access to a direct correspondence between the 2 domains, and we can generalize to setting when this information is accessible for some samples (semi supervised) or not available at all (unsupervised).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l6vTcsIXIQ", "reviewer": "Reviewer_9utW", "summary": "This paper proposes using the functional maps paradigm for comparing and aligning the latent spaces of different neural architectures, possibly trained with different setups: initialization, different datasets, noise, etc. \n\nThe main contribution of this paper is to view the latent space as a Riemannian manifold, and comparing between such spaces is modeled as a correspondence problem. The authors leverage the functional maps paradigm prominent in geometry processing that effectively uses the spectral geometry of manifolds to obtain a compact and accurate representation of the map.\n\nThe authors demonstrate their idea on 3 (mostly synthetic) setups: (1.) Using F-Maps as a measure to compare between spaces (Sec 4.3), (2.) Stitching latent spaces (Sec 5.2) and (3.) Retrieval using word embeddings. The experiments demonstrate that the functional paradigm is quite good when the number of anchors (ground truth correspondences) is less making a case for the effective use of spectral geometry in such applications", "questions": "- What is the significance of comparing LFmaps + Ortho with only Ortho? \n- Why is the variance in the Fine Labelling case (4b) more than (4a)?", "limitations": "I do not see any direct negative social impact. Overall: I think this is a submission with some interesting findings and the use of functional maps in this setting is quite refreshing. However, I lean negatively on (1.) The lack of good baselines (2.) Stronger evidence on the applicability of functional maps (3.) Lack of clear communication on: progress over [29] and the experimental section not being fully self-contained.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The core idea is interesting i.e. to view a latent space as a manifold and then use spectral geometry and F-Maps to compare multiple such spaces \n- I found the experiments making a good proof of concept", "weaknesses": "- The submission builds considerably on the ideas introduced in [29] and it is important to highlight and compare the key conceptual differences (aside from just \"using\" F-Maps)\n- The success of this approach depends on the availability of good anchors and label-based desriptors. Designing these descriptors for 3D shapes (HKS/WKS/SHOT) was straightforward. It is unclear from the submission how these descriptors can be generalized in the setting of representation spaces?  \n- Figure 2 is misleading if the colorbars are not the same. \n- I found the baselines used for comparison to be rather limited. Why not compare with [18, 22, 25, 27, 20, 13]? To this aid, it is important to show some reconstruction examples similar to Figure 4 in [29] and compare with it \n- As a general comment, the experimental section seems very hurried. Figures 3 and 5 need a more descriptive caption explaining what is to be learned there? \n- It would help to make a figure/experiment showing the functional map matrices. While it is acceptable to digest that \"similar objects have similar representations\" and that \"global distances are preserved\", I am not fully convinced that this translates into an exact near-isometry model for latent representations. A good evidence would be to show F-Map matrices in different settings. To this aid, there could be some adjustments in the laplacian and descriptor commutativity constraints in Eq 2 and I find it a bit disappointing to not anything new here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and suggestions. We are gonna address their questions and concerns in the following, and remain available for any additional questions.\n\n**Comparison with [29]**\n\nThe method in [29] is one of many recent works [13,18, 20,22, 25, 27] that focus on the emergence of similar representation independently to either measure similarity or solving for a map (implicitly or explicitly) between the two domains. \nThe conceptual differences with [29] are in: \n- Solving the alignment problem by optimizing for a map between the eigenbasis of the two domains (functional map). Therefore our framework optimizes for a map between the spaces, while [29] maps both spaces independently to a shared relative space and the mapping is implicit.\n- By employing cosine similarity, [29] assumes that the transformation is a global rotation of the space plus a local rescaling, so that cosine similarity is invariant to the mapping. This means that when there is a more complex transformation [29] will not be able to handle it well, while a functional map can in principle handle complex non linear transformations by expressing the map as a linear sparse matrix in the spectral domain.\n- Our framework comprises at the same time (i) an interpretable similarity measure, (ii) a way to find correspondences, and (iii) solving for an explicit mapping between different spaces in a single framework, differently from [29] which attempts to solve just the latter implicitly.\nWe included a discussion in the manuscript.\nIn terms of qualitative and quantitative performance we added experiments in Table 2 comparing with [29] in the general Answer, demonstrating better qualitative and quantitative performance.\n\n**Descriptors**\n\nIn general in our framework descriptor functions can either be semi supervised (e.g. express on each node of the ) weakly supervised (use the indicator functions of labels ) or fully unsupervised ( unsupervised descriptors of the graph, such as the heat kernel signature[c]).\n\nNeural models, compared to 3D shapes, involve many quantities that can be assumed to be preserved across latent spaces, as indicated in [13, 29, 22, b]. Labels are a clear example, but unsupervised properties related to distributions in latent spaces can also be preserved. For instance, in models that assume a prior distribution in latent spaces, such as Variational Autoencoders and derived models, specific quantities can be conserved and exploited to develop useful descriptors.\n\nWe agree that designing better unsupervised descriptors tailored to the data and task at hand is an intriguing direction. This is a key focus area for our future work. We report an ablation study on the choice of descriptors in the General answer.\n\n_[a] Sun, J. et al.. A concise and provably informative multi‐scale signature based on heat diffusion. In Computer graphics forum_ \n_[b]Damodaran, B.et al.. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation.ECCV_\n\n**Figure 2**\n\nWe adjusted the colorbars in Figure 2 for improved visualization. \n\n**Baselines**\n\nWe added a comparison with different baselines [18,25,22] on the CUB dataset on a retrieval task in the general answer. With this experiment, we remark that LFM can be used on top of any off-the shelf method as [18, 22, 25, 27, 20, 13].\n\nWe did compare with the method in [22], referred to as “ortho” in Figures 3 and 5 of the manuscript, and in the additional experiments of the general answer. In the main paper we chose to compare mostly with it as it holds the state of art in the stitching and retrieval benchmarks considered.\nWith respect to [18,25,22,17] the assumptions are pretty similar, in that they assume a linear mapping and optimize for it.\nWe stress that with respect to methods like [13], that compute a similarity kernel   between latent spaces without solving for any mapping, can be easily incorporated in our framework as descriptors of the graph nodes.\n\nRegarding the visualization of stitched autoencoder reconstructions, we have added an experiment using the MNIST, FashionMNIST, and CIFAR-10 datasets. The results, presented in the attached PDF in the general answer section, demonstrate superior reconstruction quality w.r.t. [22].\n\n**Clarity on Figure 3 and 5**\n\nWe improved the experimental section stating better the goal and takeaways from each experiment. We will add the update in the camera ready.\n\n**Functional Map visualization**\n\nWe added functional maps visualization for experiment in the pdf attached in the general answer in two experiments. \nIn Figure, we conduct a synthetic experiment by perturbing the space with simple transformations, such as rotation. We observe that the map remains perfectly diagonal when recovering an isometry of the space. Additionally, we note that it preserves a diagonal-like structure even when noise is added. \nIn Figure we show  functional maps visualizations in real case scenarios.\n\n**Comparison with Ortho [22] clarification**\n\nIn both Figures 3 and 5 the purpose of comparing between Ortho and LFM + Ortho is to show how our framework can expand the set of correspondences from very few (e.g 3.) to many one and outperform the direct estimation of the Orthogonal mapping by just using the same initial information. We choose to fit the same orthogonal transformation on top of the correspondence found by LFM to make the comparison with [22] the fairest possible, although in principle any off-the-shelf methods could be used to estimate the transformation once the new correspondence is found.\n\n**Variance fine labeling**\n\nThe fine labels case is more complex than the coarse label one, as the latter correspond to assign the superclass information to each sample, therefore resulting in having less classes. Therefore estimating the map can be more difficult in some settings. We observed this especially when trying to match self supervised vision models to classification based ones (i.e. DINO vs ViT).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W5AEqxzQfC", "reviewer": "Reviewer_yAi6", "summary": "The paper demonstrates the possibility to use the functional map tool on the embedding space of neural networks. The idea is that the embedding space of a neural network is usually lay on a low dimensional manifold, and networks that are trained for the same tasks even with different architecture result in similar manifolds in the embedding spaces. It was suggested in the paper to use the functional map as a tool for representation transfer for various tasks. The authors showed that using the functional maps improves over previous methods on several benchmarks (similarity measures, stitching, and word retrieval).", "questions": "see above sections.", "limitations": "The authors should add the known limitations of the functional maps, such as the setting of full and partial manifolds [2, 3, 4]. Additionally, experiments that would significantly strengthen the claims are those that would show that the method presents the same limitations as those used in functional maps for 3D analysis. E.g an experiment showing the part-to-full manifold limits (e.g. training a network with part of the data that another network was trained on, and showing a slanted-diagonal structure of the functional map).\n[2] Litany, O., et al.. \"Fully spectral partial shape matching.\" In Computer Graphics Forum, vol. 36, no. 2, pp. 247-258. 2017.\n[3] Rodolà, E., et al.. \"Partial functional correspondence.\" In Computer graphics forum, vol. 36, no. 1, pp. 222-236. 2017.\n[4] Bracha, A., et al.. \"On Partial Shape Correspondence and Functional Maps.\" arXiv preprint arXiv:2310.14692 (2023).", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents a novel and elegant idea to utilize the functional map framework for neural representation transfer. In general, the paper is written clearly. In addition, the existing experiment shows superiority in comparison to previous methods.", "weaknesses": "There are several concerns regarding this paper that should be addressed by the authors before upgrading the rank. First, it is unclear why there is a need for least squares for computing the functional maps as the correspondence is known by construction. Why not calculate it directly? It is more accurate and does not require the zoomout post-processing. If the authors believe otherwise, they should add an ablation study showing it and explain why this is the case. Additionally, even if it was decided to calculate it via least squares, why the chosen descriptors were used the way they are - as there is no guarantee for their consistency in the two latent spaces. The authors should explain this important point, and may do an ablation study regarding the appropriate descriptors.\nSecond, In general, I would like to see more experiments (or at least similar experiments on more datasets) as in other papers in the field [1]. Additionally, in the stitching experiment, to build the graph the authors used 3000 points with 300 neighbors per point, in that case calculating geodesic distances is almost similar to calculating Euclidean distances in the latent space, which might impair the descriptors. Moreover, the method was not compared to the other baselines. The retrieval experimental setting is unclear.\nSmall issues: I think there is a mix up when using the notation X, Y, M and N (e.g. see caption of the overview figures). \n[1]  Maiorca, Valentino, et al. \"Latent Space Translation via Semantic Alignment.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and suggestions. We are gonna address their questions and concerns in the following, and remain available for any additional questions.\n\n**Know the correspondence**\n\nWe fully agree with the reviewer that when the correspondence is known there is no need to optimize for the functional map C. We would like to clarify however, that the correspondence between the two domains is not (fully) available in most of our experiments. In general we would like to address both the cases in which the correspondence is known (for example measuring similarity between projections of the same samples in different networks, when inputs are available) and settings in which are not as stitching and retrieval tasks. \n\nIn the experiment proposed (retrieval and stitching) our goal is to recover a good correspondence starting from very  few or coarse (labels) correspondences between the domains..\n\nWe were glad to see however that the optimization recovered very good estimates of the ground truth functional maps.\n\n\n**Geodesic distance descriptor**\n\nWe remark that the geodesic distance descriptor on the graph is computed just w.r.t to the available correspondences (anchors) on each node of the graph. Therefore the descriptor just encodes the anchors correspondence as distance functions, which is instead provided explicitly to fit the transformation in [22]. \nWe chose to employ the geodesic distance descriptor as it is agnostic to the metric used to build the graph (in this case cosine distance). Therefore the geodesics (shortest paths) on the graph will be a discretization of the distance metric used to build the knn graph, and are expressed just w.r.t. to the anchors.We clarified this in the paper.\nIn general we explored different strategies for the choice of descriptor functions on top of the KNN graph, and choose the geodesic distance on the graph as it is agnostic of the choice of the metric used to construct the KNN graph  and provides a good choice in terms of performance. We included an ablation experiment on the choice of descriptor in the PDF of the general answer. \n\n\n**Experiments and baselines**\n\nWe included many additional experiments in the general answers on diverse and large scale datasets. In Figure 1 we test stitching on diverse datasets (ImageNet,  MNIST, AGNews, DBPedia14, CUB) to validate the performance of LFM on multiple settings, including  large scale, complex, datasets (Imagenet).\n\n**Partiality limitation**\n \nWe agree that the partiality case must be still explored and will include it in the limitation section.\n\n**Notation**\n\nWe fixed the notation in the introduction. We thank the reviewer for spotting this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their feedback and suggestions. In the following we addressed some general comments raised by the reviewers and attached a pdf with the experiments performed during the rebuttal period. We remain available for any further question or clarification during the discussion period.

## Additional Experiments

We significantly extended our experimental analysis, reporting the results in the attached pdf and explaining the experiments performed in the following:


**Different datasets** 

We benchmarked our method on multiple datasets (ImageNet [a], Mnist [b], AGNews [c],  Caltech-UCSD Birds-200 [e]) on stitching task in Figure 1. This includes large scale datasets with complex structure such as ImageNet, and multiple modalities (text and images). The stitching experiments confirmed how LFMAP is consistently superior in all these settings to the state of the art baseline of [22], especially when the number of known correspondences is low or absent (labels descriptor case).

**Multiple baselines** 

In Table 2 we added a comparison with multiple baselines  [18,25,22,29,f] on the Caltech-UCSD Birds-200-2011 (CUB) dataset on retrieval tasks with the same foundation models and settings used in the main paper. We demonstrate that LFM is consistently superior in performance and can be used on top of any method which computes an explicit mapping between spaces.

**Qualitative evaluation** 

In Figure 2 we added qualitative experiments on the MNIST, FashionMNIST and CIFAR 10 datasets,  concerning visualizing reconstructions of stitched autoencoders. For these experiments we trained convolutional autoencoders with two different seeds, and stitched their encoder decoder module. For our method and [22] we used 10,10, and 50 correspondences for the  MNIST, FashionMNIST [g] and CIFAR 10 datasets, respectively. We observe consistently superior reconstruction quality w.r.t. [22] and the absolute baseline.

**Choice of Descriptors** 

In Figure 4, we performed an ablation study on the choice of descriptors, comparing supervised descriptors (geodesic distance and cosine distance using 10 correspondences), weakly supervised descriptors (label descriptor), and fully unsupervised descriptors (heat kernel signature [i] and wave kernel signature [h]). 
To conduct this study, we performed a retrieval task on the test embeddings of two convolutional autoencoders trained on MNIST, which differed by their parameter initialization. We visualized the structure of the functional map and reported the performance in terms of mean reciprocal rank (MRR), observing the following: (i) Geodesic and cosine descriptors performed best (ii) The geodesic distance (shortest path) is a good choice as it is agnostic of the metric chosen to build the initial graph, yet provides the same result as using the metric itself; (iii) The structure of the functional map reflects the performance of retrieval.

**Functional map structure** 

In Figure 3 and Table 1, we present a visualization of the structure of the functional map in a synthetic setting. The experiment was conducted as follows: given an input set \(X\) consisting of test embeddings extracted from MNIST, we aimed to observe the degradation of the functional map structure as the space is perturbed. The perturbation involved an orthogonal transformation combined with additive Gaussian noise at increasing levels. In the first row, the functional maps were computed from k-nearest neighbor (knn) graphs using the cosine distance metric, while in the second row, knn graphs were constructed using the Euclidean distance metric. Below each functional map, the LFM similarity score and MRR retrieval scores are displayed. We observed that (i) when noise is absent (first column), the two spaces are isometric, and the functional map is diagonal, (ii) constructing the graph with the cosine distance metric is more robust to increasing noise, and (iii) the LMF similarity score correlates with the MRR retrieval metric, indicating that more structured functional maps reflect better alignment between spaces.

## Clarity

We have significantly improved the manuscript's clarity, making it self-contained and clear. In particular: 
We structured the experimental section by adding details about the settings in the experimental setting and adding a smaller subsection depicting the experiment's goal and takeaway.  In the experimental section, we added an in-detail description for [29] and [22], making it clear that we are in the most similar setting to have a fair comparison with them. We added more details in section 3 and section 4.2, also adding more information in the appendix, complementary to the one already present in Appendix A, to make everything self contained and avoid pointing the reader to past work. Although we thank and agree with the Reviewer that our manuscript is clearer right now, we would like to respectfully point out that Reviewer yAi6 highlighted writing clarity, with confidence 5.

_[a] Deng, J. et al., 2009. Imagenet: A large-scale hierarchical image database.IEEE_

_[b] Deng, L.,. The mnist database of handwritten digit images for machine learning research. IEEE_

_[c] Zhang,et al "Character-level convolutional networks for text classification." Advances in neural information processing systems 28 (2015)_

_[e] Wah,et al The Caltech-UCSD Birds-200-2011 Dataset_

_[f] Conneau, A., et al,. Word translation without parallel data, ICLR_

_[g] Xiao, H.; Rasul, K. & Vollgraf, R. (2017), Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning_

_[h] Aubry, M., Schlickewei, U., & Cremers, D. (2011, November). The wave kernel signature: A quantum mechanical approach to shape analysis. In 2011 IEEE international conference on computer vision workshops (ICCV workshops) (pp. 1626-1633). IEEE_

_[i] Sun, J. et al.. A concise and provably informative multi‐scale signature based on heat diffusion. In Computer graphics forum_

Meta-review: ### Summary
This paper proposes a new approach for comparing neural representations by approximating the associated manifold with a kNN graph and utilizing functional maps, a technique that leverages the spectral properties of the manifold to compute relationships between distinct representations, such as similarity.



$$$$

### Strengths
- **Significance**: The paper tackles an interesting and practically relevant problem. The proposed approach presents a novel combination of existing techniques in a simple and elegant manner, which could inspire future research. 
- **Experiments**: The experimental results effectively demonstrate the efficiency of the proposed approach.

$$$$

### Concerns
- **Clarity**: The writing could be improved to make the paper more self-contained and accessible to non-experts. Some aspects of the method and the experimental setup should be explained more explicitly. Additionally, a more thorough discussion of related work would be beneficial.

- **Limitations**: The paper does not address certain technical limitations, particularly those related to partial manifolds and the design of descriptors in general settings.

$$$$

### Overall
Considering the contributions of the paper -- particularly its simple yet effective modeling approach and potential impact -- along with the efforts made to address the reviewers' questions and the additional experiments provided in the rebuttal, I believe this paper is a reasonable contribution to the conference and recommend acceptance. While the clarity was criticized, the requested improvements can be implemented, and a discussion of the limitations can be included.
Decision: Accept (poster)