Latent Functional Maps: a spectral framework for representation alignment

Authors: marco  fumero
marco  pegoraro
valentino  maiorca
  sapienza
francesco  locatello
emanuele  rodolà
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:13:36

Abstract: Neural models learn data representations that lie on low-dimensional manifolds, yet modeling the relation between these representational spaces is an ongoing challenge.
By integrating spectral geometry principles into neural modeling, we show that this problem can be better addressed in the functional domain, mitigating complexity, while enhancing interpretability and performances on downstream tasks. 
To this end, we introduce a multi-purpose framework to the representation learning community, which allows to: (i) compare different spaces in an interpretable way and measure their intrinsic similarity; (ii) find correspondences between them, both in unsupervised and weakly supervised settings, and (iii) to effectively transfer representations between distinct spaces.
We validate our framework on various applications, ranging from stitching to retrieval tasks, and on multiple modalities, demonstrating that Latent Functional Maps can serve as a swiss-army knife for representation alignment.

Main paper: 
 introduction: 
 Neural Networks (NNs) learn to represent high-dimensional data as elements of lower-dimensional latent spaces. While much attention is given to the model's output for tasks such as classification, generation, reconstruction, or denoising, understanding the internal representations and their geometry is equally important. This understanding facilitates reusing representations for downstream tasks [62, 14] and the comparison between different models [23, 39] , thereby broadening the applicability of NNs, and understanding their structure and properties. Moreover, recent studies have shown that distinct neural models often develop similar representations when exposed to similar stimuli, a phenomenon observed in both biological [16, 25, 63] and artificial settings [38, 23, 39] . Notably, even when neural networks have different architectures, internal representations of distinct models can often be aligned through a linear transformation [61, 50] . This indicates a level of consistency in how NNs process information, highlighting the importance of exploring and understanding these internal representations. ϕ M 1 ϕ M 2 ϕ M k ϕ N 1 ϕ N 2 ϕ N k C : F(M) → F (N ) (i) (ii) (iii)Figure 1 : Framework overview: given two spaces X , Y their samples lie on two manifolds M, N , which can be approximated with the KNN graphs G X ,G Y . We can optimize for a latent functional map C between the eigenbases of operators defined on the graphs. This map serves as a map between functions defined on the two manifolds and can be leveraged for (i) comparing representational spaces, (ii) solving correspondence problems, and (iii) transferring information between the spaces.In this paper, we shift our focus from characterizing relationships between samples in distinct latent spaces to modeling a map among function spaces defined on these representational spaces. To this end, we propose leveraging spectral geometry principles by applying the framework of functional maps [44] to the field of representation learning. Functional maps represent correspondences between function spaces on different manifolds, providing a new perspective on the problem of representational alignment. In this setting, many complex constraints can be easily expressed compactly [45] . For instance, as shown in Figure 1 , the mapping (C) between two spaces M and N can be represented in the functional space as a linear map with a sparse structure.Originally used in 3D geometry processing and more recently on graphs applications [60, 46] , we extend this framework to handle arbitrary large dimensional latent spaces. Our proposed method, Latent Functional Map (LFM), is a flexible tool that allows (i) to compare distinct representational spaces, (ii) to find correspondences between them both in an unsupervised and weakly supervised way, and (iii) to transfer information between them. Our contributions can be listed as follows:• We introduce the framework of Latent Functional Maps as a way to model the relation between distinct representational spaces of neural models. • We show that LFM allows us to find correspondences between representational spaces, both in weakly supervised and unsupervised settings, and to transfer representations across distinct models. • We showcase LFM capabilities as a meaningful and interpretable similarity measure between representational spaces. • We validate our findings in retrieval and stitching tasks across different models, modalities and datasets, demonstrating that LFMs can lead to better performance and sample efficiency than other methods.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The abstract and introduction gives an overview of the paper's scope clearly stating the contributions in a dotted list.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Similarity between latent spaces. Comparing representations learned by neural models is of fundamental importance for a diversity of tasks [63] , ranging from representation analysis to latent space alignment and neural dynamics. In order to do so, a similarity measure between different spaces must be defined [22] . This can range from functional similarity (matching the performance of two models) to similarity defined in representational space [23] , which is where our framework falls in. A classical statistical method is Canonical Correlation Analysis (CCA) [18] , known for its invariance to linear transformations. Various adaptations of CCA aim to enhance robustness, such as Singular Vector Canonical Correlation Analysis (SVCCA) [48] , or to decrease sensitivity to perturbations using methods like Projection-Weighted Canonical Correlation Analysis (PWCCA) [38] . Closely related to these approaches, Centered Kernel Alignment (CKA) [23] measures the similarity between latent spaces while ignoring orthogonal transformations. However, recent research [10] reveals that CKA is sensitive to local shifts in the latent space.We propose to leverage LFMs as a tool to measure the similarity, or how much two spaces differ from an isometry w.r.t. to the metric that has been used to construct the graph.Representation alignment. Complementary to measuring the similarity of distinct representational spaces, several methods have been proposed to optimize for a transformation to align them [1] .The concept of latent space communication, introduced by [39] , builds on the hypothesis that latent spaces across neural networks, varying random seed initialization to architecture or even data modality, are intrinsically compatible. This notion is supported by numerous empirical studies [38, 30, 23, 6, 55, 3, 58, 27, 29, 4, 40, 9, 15, 8] , with the phenomenon being particularly evident in large and wide models [51, 33] . The core idea is that relations between data points (i.e., distances according to some metric) are preserved across different spaces because the high-level semantics of the data are the same and neural networks learn to encode them similarly [19] . With this "relative representation", the authors show that it is possible to stitch [29] together model components coming from different models, with little to no additional training, as long as a partial correspondence of the spaces involved is known. Indeed, [26, 32, 35, 37] show that a simple linear transformation is usually enough to effectively map one latent space into another, measuring the mapping performance via desired downstream tasks.With LFMs, we change the perspective from merely relating samples of distinct latent spaces to relating function spaces defined on the manifold that the samples approximate, showing that processing information in this dual space is convenient as it boosts performance while also being interpretable.Functional Maps. The representation we propose is directly derived from the functional maps framework for smooth manifolds introduced in the seminal work by [44] . This pioneering study proposed a compact and easily manipulable mapping between 3D shapes. Subsequent research has aimed at enhancing this framework. For instance, [41] introduced regularization techniques to improve the informativeness of the maps, while [34] developed refinement methods to achieve more accurate mappings. The functional map framework has been extended as well outside the 3d domain, for example, in [60] and [17] , who applied the functional framework to model correspondences between graphs, and in [46] , who demonstrated its utility in graph learning tasks. In particular, they have shown that the functional map representation retains its advantageous properties even when the Laplace basis is computed on a graph.Inspired by these advancements, our work leverages the functional representation of latent spaces of neural models. We demonstrate how this representation can be easily manipulated to highlight similarities and facilitate the transfer of information between different spaces, thereby extending the applicability of the functional maps framework to the domain of neural latent spaces.For this experiment, we construct k-NN graphs with a neighborhood size of 300 and compute the functional map using the first 50 eigenvectors. We evaluate the methods' performance using the Mean Reciprocal Rank (MRR), as detailed in Appendix B.4. Our functional map methods are compared with the method proposed by [39] (Relatives) and the orthogonal transformation method proposed by [32] (Ortho). We choose to fit the same orthogonal transformation on top of the correspondence found by LFM to make the comparison the fairest possible, although in principle any off-the-shelf methods could be used to estimate the transformation once the new correspondence is found. We illustrate this versatility in Table 3 of the Appendix, where LFM is combined with other methods in the same task.Result Analysis Figure 6 shows the performance of these methods as the number of anchors increases. Numerical results are detailed in Table 2 in the Appendix. The functional map significantly improves performance with just 5 anchors, achieving an MRR of over 0.8. As the number of anchors increases, the performance of competing methods improves but still falls short of FMAP+Transform at 300 anchors, which reaches an MRR of 0.99. Interestingly, the performance of the functional map methods does not improve beyond 5 anchors, suggesting that this number of anchors is sufficient to achieve an optimal functional map between the spaces. These findings are further supported by Table 3 in the Appendix, where LFM consistently achieves superior MRR scores, particularly with fewer anchors, outperforming other baselines such as [29] . In Appendix C.3, we analyze how the results improve as the number of eigenvectors used to compute the functional map increases. Notably, the MRR score drastically increases to over 0.6 with more than 25 eigenvectors.These results confirm that the latent functional map is a valuable tool in settings with little knowledge about correspondences. It significantly enhances retrieval performance with a minimal number of anchors, making it an efficient approach for aligning latent spaces. Moreover, its performance can be improved using a higher number of eigenvectors. In particular, our framework comprises at the same time (i) an interpretable similarity measure, (ii) a way to find correspondences, and (iii) solving for an explicit mapping between different spaces in a single framework, differently from [39] and [32] which attempt to solve just the latter implicitly.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We put our limitations in the conclusion, after analyzing the results of our method.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
background: 
 This section provides the basic notions to understand the framework of functional maps applied to manifolds. We refer to [45] for a comprehensive overview.Consider two manifolds M and N equipped with a basis Φ M (respectively Φ N ). Any squared integrable function f : M → R can be represented as a linear combination of basis functions Φ M :f = i a i Φ M i = aΦ M .Given a bijective correspondence T : M → N between points on these manifolds, for any real-valued function f : M → R, one can construct a corresponding function g : N → R such that g = f • T -1 . In other words, the correspondence T defines a mapping between two function spaces T F : F(M, R) → F(N , R). Such a mapping is linear [44] and can be represented as a matrix C such that for any function f represented as a vector of coefficients a, we have T F (a) = Ca.The functional representation is particularly well-suited for map inference (i.e., constrained optimization problems). When the underlying map T (and by extension the matrix C) is unknown, many natural constraints on the map become linear constraints in its functional representation. In practice, the simplest method for recovering an unknown functional map is to solve the following optimization problem:arg minEQUATIONwhere A and B are sets of corresponding functions, denoted as descriptors, expressed in the bases on M and N , respectively, and ρ(C) represents additional constraints deriving from the properties of the matrix C [45] . When the manifolds M and N are approximately isometric and the descriptors are well-preserved by the (unknown) map, this procedure provides a good approximation of the underlying map. In cases where the correspondence T is encoded as a permutation matrix P, the functional map can be retrieved as C = Φ † N PΦ M where Φ M and Φ N are the bases of the functional spaces F(M, R) and F(N , R), respectively, and † denotes the Moore-Penrose pseudo-inverse.We consider deep neural networks f := f 1 • f 2 • ...f n where each layer f i is associated to a representational space X i corresponding to the image of f i . In the following we're gonna drop the subscript i, when the layer considered is clear form the context. We assume that elements x ∈ X are sampled from a latent manifold M. Considering pairs of spaces (X , Y), and corresponding manifolds M, N our objective is to characterize the relation between them by mapping the space of functions F(M) to F(N ). Our framework can be summarized in three steps, which we are going to describe in the following sections: (i) how to represent M from a sample estimate X, by building a graph in the latent space X (ii) how to encode any known preserved quantity between the two spaces by defining a set of descriptor function for each domain (iii) optimizing for the latent functional map between M and N . An illustrative description of our framework is depicted in Figure 1 .Building the graph. To leverage the geometry of the underlying manifold, we model the latent space of a neural network building a symmetric k-nearest neighbor (k-NN) graph [31] . Given a set of samples X = {x 1 , . . . , x n }, we construct an undirected weighted graph G = (X, E, W) with nodes X, edges E, and weight matrix W. The weight matrix is totally characterized by the choice of a distance function d : X × X → R.Suitable choices for d include the L2 metric or the angular distance. Edges E are defined as E = {(x i , x j ) ∈ X × X | x i ∼ k x j or x j ∼ k x i }, where x i ∼ k x j indicates that x j is among the k nearest neighbors of x i . The weight matrix W ∈ R n×n ≥0 assigns a weight ω(x i , x j ) = α(d(x i , x j )) to each edge (x i , x j ) ∈ E, and W i,j = 0 otherwise, with α being some monotonic function. Next, we define the associated weighted graph Laplacian operator L G = I -D -1/2 WD -1/2 , where D is the diagonal degree matrix with entries D i,i = n j=1 W i,j . L G is a positive semi-definite, self-adjoint operator [57] ), therefore, it admits an eigendecompositionL G = Φ G Λ G Φ T G, where Λ G is a diagonal matrix containing the eigenvalues, and Φ G is a matrix whose columns are the corresponding eigenvectors. The eigenvectors form an orthonormal basis for the space of functions defined on the graph nodes (i.e., Φ T G Φ G = I). We give comprehensive details on the choice of k in Appendix A.1.Choice of descriptors. To define the alignment problem between pair of spaces X , Y we will introduce the notion of descriptor function. We define as descriptor function any real valued function f : G → R k defined on the nodes of the graph. Informally descriptors should encode the information which is shared between X and Y, either explicitly or implicitly . We categorize descriptor functions into supervised, weakly supervised and unsupervised descriptors. For the former we assumed to have access to a partial correspondence between the domains X , Y, defined as a bijective mappingΓ S : A X → A Y where A X ⊂ X , A Y ⊂ Y. Then descriptors takes the form of distance functions d X : X × A X → R + , d Y : Y × Γ S (A X ) → R + .As an example considering the geodesic distance (shortest path distance on the k-NN graph) from each node of the graph to the nodes in the anchor set, is an instance of supervised descriptor. In the case of weakly supervised descriptor, we assume to have access to an equivalence relation defined Γ W : A X × A Y → {0, 1}. Example of these are multi-to-multi mappings, like mappings between labels. Unsupervised descriptors are quantities that depends on the topology and the geometry of the graph itself and they don't rely on any pre-given correspondence or equivalence relation. An example of this is the Heat Kernel Signature [53] node descriptor, based on heat diffusion over the manifold. We give examples of descriptors and ablation in Appendix C.2.Computing LFMs. We now describe the optimization problem to compute a latent functional map between X and Y. We model each space using a subset of training samples X = {x 1 , . . . , x n } and Y = {y 1 , . . . , y n } and build the k-NN graphs G X and G Y from these samples, respectively. For each graph, we compute the graph Laplacian L G and derive the first k eigenvalues Λ G and eigenvectors Φ G = [ϕ 1 , . . . , ϕ k ], which serve as the basis for the function space defined on the latent spaces.Given the set of descriptorsF G X = [f G X 1 , . . . , f G X n f ] and F G Y = [f G Y 1 , . . . , f G Y n f ],we consider the optimization problem defined in Equation 1 and incorporate two regularizers which enforce commutativity for the Laplacian and the descriptors, respectively, with the map C:EQUATIONwhere FG = Φ T G F G are the spectral coefficients of the functions F G , ρ L and ρ f are the Laplacian and descriptor operator commutativity regularizers respectively, akin to [41] . We give full details on the regularizers in Appendix A. Once we have solved the optimization problem defined in Equation 2, we refine the resulting functional map C using the spectral upsampling algorithm proposed by [34] , as detailed in Appendix A.3.Once computed, the functional map C can serve as a measure of similarity between spaces. The reason is that for isometric transformations between manifolds, the functional map is volume preserving (see Thm 5.1 in [44] ), and this is manifested in orthogonal C. By defining the inner product between functionsh 1 , h 2 ∈ F(M ) as ⟨h 1 , h 2 ⟩ = M h 1 (x)h 2 (x)µ(x), it holds that ⟨h 1 , h 2 ⟩ = ⟨ ĥ1 , ĥ2 ⟩when the map preserves the local area, where ĥ denotes the functional representation of h. In other words, when the transformation between the two manifolds is an isometry, the matrix C T C will be diagonal. By measuring the ratio between the norm of the off-diagonal elements of C T C and the norm of the full matrix, we can define a measure of similarity sim(X, Y ) = 1 -||off((C T C)|| 2 F ||C T C|| 2 F. In Appendix A.4 we prove that this similiarity measure is a proper distance on the space of functional maps, allowing to compare measurements from collections of spaces. Furthermore, this quantity is interpretable; the first eigenvector of C T C can act as a signal to localize the area of the target manifold where the map has higher distortion [43] , as we show in the experiment in Figure 3a .The map C computed between two latent spaces can be utilized in various ways to transfer information from one space to the other. In this paper, we focus on two methods: (i) Expressing arbitrary points in the latent space as distance function on the graph and transferring them through the functional domain; (ii) Obtaining a point-to-point correspondence between the representational spaces from the LFM (see the first step in Appendix A.3) , starting from none to few known pairs, and leverage off-the-shelf methods to learn a transformation between the spaces. Additional strategies could be explored in future work.Space of Functional Coefficients. The space of functional (or spectral) coefficients offers an alternative representation for points in the latent space X . Using the equation fG = Φ T G f G , any function f G ∈ F(G, R) can be uniquely represented by its functional coefficients fG . We leverage this property to represent any point x ∈ X as a distance function f d ∈ F(G, R) from the set of points X G , which correspond to the nodes of the graph G. The functional map C between two latent spaces X and Y aligns their functional representations, enabling the transfer of any function from the first space to the second. This functional alignment can be used similarly to the method proposed by [39] to establish a shared space where the representational spaces X and Y are aligned.Finding correspondences. As explained in Section 3, the functional map C represents the bijection T in a functional form. [44] demonstrated that this bijection can be retrieved as a point-to-point map by finding the nearest neighbor for each row of Φ G Y C in Φ G X . This process can be efficiently implemented and scaled up using kd-tree structures or approximation strategies [21, 20] . Starting from an empty set or few known correspondences (anchors) between the two spaces X and Y, we can extend the correspondence to the entire set of nodes X and Y . This extended set of anchors can then be used to fit an arbitrary transformation between the latent spaces, for example an orthogonal mapping [32] . In our experiments, we demonstrate that by using a very small number of anchors (typically ≤ 50), we can retrieve optimal transformations that facilitate near-perfect stitching and retrieval tasks.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: The paper does not include new theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 In this section, we present a series of experiments designed to evaluate the effectiveness of the Latent Functional Map (LFM) framework. We explore its application across various tasks, including similarity evaluation, stitching, retrieval performance, and robustness to perturbations in latent space. By comparing LFM to existing methods under different conditions, we aim to demonstrate its versatility and superior performance in aligning and analyzing neural network representations.Additional ablations and qualitative visualizations on the choice of distance metric to construct the graph and descriptors selection are reported in the Appendix C.We demonstrate the benefits of using latent functional maps for comparing distinct representational spaces, using the similarity metric sim(X, Y ) defined in Section 3.3Experimental setting In order to validate experimentally if LFMs can serve as a good measure of similarity between distinct representational spaces, we run the same sanity check as [23] . We train 10 CNN models (the architecture is depicted in Appendix B.1) on the CIFAR-10 dataset [24] , changing the initialization seed. We compare their inner representations at each layer, excluding the logits and plot them as a similarity matrix, comparing with Central Kernel Alignment (CKA) measure [23] and Canonical Correlation Analysis (CCA) [18, 48] . We then measure the accuracy of identifying corresponding layers across models and report the results comparing with CKA and CCA as baselines. For CCA, we apply average pooling on the spatial dimensions to the embeddings of the internal layers, making it more stable numerically and boosting the results for this baseline compared to what was observed in [23] .Result analysis Figure 2 shows that our LFM-based similarity measure behaves correctly as CKA does. Furthermore, the similarities are less spread around the diagonal, favoring a slightly higher accuracy score in identifying the corresponding layers across models.While CKA (Centered Kernel Alignment) is a widely used similarity metric in deep learning, recent research by [10] has shown that it can produce unexpected or counter-intuitive results in certain situations. Specifically, CKA is sensitive to transformations that preserve the linear separability of two spaces, such as local translations. Our proposed similarity measure is robust to these changes and demonstrates greater stability compared to CKA.Experimental setting We compute the latent representations from the pooling layer just before the classification head for the CIFAR10 train and test sets. Following the setup in [10] , we train a Support Vector Machine (SVM) classifier on the latent representations of the training samples to find the optimal separating hyperplane between samples of one class and others. We then perturb the samples by translating them in a direction orthogonal to the hyperplane, ensuring the space remains linearly separable. We measure the CKA and LFM similarities as functions of the perturbation vector norm, as shown in Figure 3a . In the accompanying plot on the right, we visualize the area distortion of the map by projecting the first singular component of the LFM C into the perturbed space and plotting it on a 2d TSNE [56] projection of the space.Result Analysis We start by observing that when the latent space is perturbed in a way that still preserves its linear separability, it should be considered identical from a classification perspective, as this does not semantically affect the classification task. Figure 3a shows that while CKA degrades as a function of perturbation intensity, the LFM similarity remains stable to high scores. To understand this difference, we can visualize the area distortion as a function of the samples by projecting the first singular component of C onto the perturbed space. In Figure 3b , we use t-SNE [56] to project the perturbed samples and the distortion function into 2D. The visualization reveals that distortion is localized to the samples corresponding to the perturbed class.In latent communication tasks, a common challenge is combining an encoder that embeds data with a decoder specialized in a downstream task, such as classification or reconstruction-this process is often referred to as stitching. Previous works, like [29] and [2] , have used trainable linear projections, known as stitching layers, to enable the swapping of different network components. However, [39] introduced the concept of zero-shot stitching, where neural components can be marged without any additional training procedure. It's important to note that while [39] still required to trained a decoder module once for processing relative representations before stitching could be performed, our method is fully zero-shot, with no need for additional training. In the following experiments, stitching is conducted without any training or fine-tuning of the encoder or decoder, adhering strictly to a zero-shot fashion.Experimental Setting We consider four pre-trained image encoders (see Appendix B.2 for details) and stitch their latent spaces to perform classification using a Support Vector Machine (SVM) on five Figure 5 : Qualitative results on stitching. We combine the encoder decoder modules of two convolutional autoencoder (E1 D1, E2 D2) using three different approaches: no transformation (Absol.), the orthogonal transformation from [32] (Ortho), the orthogonal transformation augmented with LFM (LFM).different datasets: CIFAR100 [24] with coarse and fine-grained labelling, ImageNet [11] , Caltech-UCSD Birds-200-2011 (CUB) [59] , MNIST [12] , AgNews [65] . These datasets were chosen to encompass both large-scale, complex datasets like ImageNet and a variety of modalities, including text and images. To evaluate the effectiveness of integrating the functional map, we extend the correspondences to determine an orthogonal transformation [32] between the latent spaces. For each encoder, we compute a graph of 3,000 points with 300 neighbors per node. We optimize the problem in Equation 2 using the first 50 eigenvectors of the graph Laplacian and consider two different descriptors: the distance functions defined from the anchors (LFM+Ortho) and the labels (LFM+Ortho (Labels)). For each dataset class, the latter provides an indicator function with 1 if the point belongs to the class and 0 otherwise. This descriptor does not require any anchor as input, representing a pioneering example of stitching requiring no additional information beyond the dataset.Result Analysis Figure 4 shows the accuracy results for all possible combinations of encoder stitching. The latent functional map (LFM+Ortho) consistently outperforms other methods across all datasets, particularly with a low number of anchors. Even without anchors, using label descriptors (LFM+Ortho (Labels)) achieves strong performance across different labeling schemes. The orthogonal transformation computed directly from the anchors (Ortho) only matches LFM's performance when more than 50 anchors are used, at which point LFM's effectiveness is constrained by the number of eigenvectors. Comparing the CIFAR100 coarse and fine-grained labeling, the fine-grained task is more complex due to the larger number of classes, making the estimation of the map more challenging. This complexity is especially evident when aligning self-supervised vision models with classification-based ones (e.g., DINO vs. ViT), where the training strategies differs.This experiment shows that the latent functional map is highly effective when few anchors are available (≤ 50). It significantly enhances performance in zero-shot stitching tasks, outperforming direct orthogonal transformations at low or no anchor counts. This suggests that the latent functional map method provides a robust means of aligning latent spaces with minimal correspondence data, making it a valuable tool for tasks requiring the integration of independently trained models.In Figure 5 , we present qualitative experiments on the MNIST, FashionMNIST [64] , and CIFAR-10 datasets, focusing on visualizing the reconstructions of stitched autoencoders. For these experiments, we trained convolutional autoencoders using two different seeds and stitched their encoder and decoder modules together. As a baseline, we used no transformation (Absol.), where the latent representation from the first encoder is directly input into the second decoder. We then compared this baseline with the orthogonal transformation from [32] (Ortho) and its LFM-augmented version (LFM). For both our method and [32] , we used 10, 10 and 50 correspondences for the MNIST, FashionMNIST, and CIFAR-10 datasets, respectively. Our method consistently produced superior reconstruction quality compared to both the baseline and the method from [32] .We extend our analysis to the retrieval task, where we look for the most similar embedding in the aligned latent space.Experimental Setting We consider two different English word embeddings, FastText [5] and Word2Vec [36] . Following the approach of [39] , we extract embeddings of 20K words from their shared vocabulary using pre-trained models. We use 2K random corresponding samples to construct the k-NN graphs and evaluate the retrieval performance on the remaining 18K word embeddings. We test two settings in our experiments:1. Aligning functional coefficients (LFM Space).Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: In the Experiment and Appendix, we report all the information needed to reproduce the experiments. 
conclusions: 
 In this paper, we explored the application of latent functional maps (LFM) to model and analyze the relationships between different latent spaces. Our approach leverages the principles of spectral geometry, providing a robust framework for comparing and aligning representations across various models and settings. By extending the functional map framework to high-dimensional latent spaces, we offer a versatile method for comparing distinct representational spaces, finding correspondences between them in both unsupervised and weakly supervised settings (few or no anchors) and transferring information across different models.Limitations and future work The performance of LFM can be sensitive to the number of eigenvectors used, as observed in some of our experiments (see Appendix C.3). Finding the optimal number of eigenvectors without extensive experimentation remains an open problem. The current framework assumes a correspondence between the domains, and is not directly adapted in order to deal with partial mappings, where just a subset of one domain can be mapped to other. Extending the results in [49, 47] to our neural representation setting, is a promising way to overcome the full correspondence assumption. Future research can further explore the potential of LFM in other domains and tasks, and modalities. The versatility of the functional representation can be further explored to define new ad-hoc constraints and regularizers for different settings. In particular, ts performance in fully unsupervised settings, while promising is still an area requiring further research and improvement. In conclusion, the Latent Functional Map framework offers a novel and effective approach to understanding and utilizing neural network representations, promising significant advancements in both theoretical and practical aspects of machine learning.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We will share the code after the paper acceptance. All the data we use are open-source and publically available. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We report all the training and test details in the Experiment and Appendix. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: The experiments that are run over multiple settings are reported as error bars that depict the whole distribution of the results obtained. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: In the Appendix we report information about the resources used for our experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We have read and respect the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our method is not tied to particular applications.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: There are no risks posed by our work. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All the models and data used in our experiments are properly cited. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Namely, for a pairs of spaces X, Y :sim(X, Y ) ≥ 0 (5) sim(X, Y ) = sim(Y, X) (6) sim(X, Z) ≤ sim(X, Y ) + sim(Y, Z) (7) Since ∥diag(C T C)∥ 2 F + ∥off(C T C)∥ 2 F = ∥(C T C)∥ 2 F we will consider sim(X, Y ) = sim(X, Y ) = ∥diag(C T C)∥ 2 F ∥(C T C)∥ 2 Ffor simplicity. The former property is trivial as the ration between two norms is positive. For proving simmetry we have that:EQUATION)for the numerator∥(Φ T X Φ Y )∥ F we trivially have diag(Φ T X Φ Y ) = diag(Φ T Y Φ X ).For the denominator:EQUATIONWe have that:EQUATIONEQUATIONwhere the last equality is verified for the Trace operator being invariant under cyclic permutations.For proving triangle inequality we will assume that the matrices of eigenvectors are full rank (i.e. all eigenvectors are considered). We have to prove the following:EQUATIONUsing the fact that the Frobenius norm is invariant under multiplication by orthogonal matrices the denominator of each term in Eq14 becomes:∥(Φ T X Φ Y )∥ 2 F = ∥Φ X ∥ 2 F = T r[Φ T X Φ X ] = T r[Id] = N . Then: ∥diag(Φ T X Φ Z )∥ 2 F N ≥ ( ∥diag(Φ T X Φ Y )∥ 2 F N + ∥diag((Φ T Y Φ Z )∥ 2 F N ) -1 (16) Since ∥diag(X T X Y)∥ 2 F ≤ ∥(X T X Y)∥ 2F for any X, Y then the RHS of Eq. 16 will always be smaller than the LHS. We observed that (i) when noise is absent (first column), the two spaces are isometric, and the functional map is diagonal, (ii) constructing the graph with the cosine distance metric is more robust to increasing noise, and (iii) the LMF similarity score correlates with the MRR retrieval metric, indicating that more structured functional maps reflect better alignment between spaces. 
 
 Appendix: • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.In Figure 8 , we performed an ablation study on the choice of descriptors, comparing supervised descriptors (geodesic distance and cosine distance using 10 correspondences), weakly supervised descriptors (label descriptor), and fully unsupervised descriptors (heat kernel signature [53] and wave kernel signature [53] ).To conduct this study, we performed a retrieval task on the test embeddings of two convolutional autoencoders trained on MNIST, which differed by their parameter initialization. We visualized the structure of the functional map and reported the performance in terms of mean reciprocal rank (MRR), observing the following: (i) Geodesic and cosine descriptors performed best (ii) The geodesic distance (shortest path) is a good choice as it is agnostic of the metric chosen to build the initial graph, yet provides the same result as using the metric itself; (iii) The structure of the functional map reflects the performance of retrieval.In Table 2 , we report the numerical results for the experiment in Figure 6 adding more transformations from the method of [32] : orthogonal (Ortho), linear (Linear) and affine (Affine). From the value in the table, we can see that all the methods that involve the latent functional map (LFM) saturate at 5 anchors, reaching top performance. In Figure 9 , we show how the performance of the latent functional map methods depends on the number of eigenvectors used to compute the map. In particular, we We report the MMR score for the methods in Figure 6 with fixed number of anchors ( 5) and increasing number of eigenvectors.Table 2 : MRR Score for the retrieval of word embeddings.We report the value of the results depicted in Figure 6 adding more kind transformation between spaces (Orthogonal, Linear and Affine). notice that the performance drastically increases at 25 eigenvectors, reaching the same score when using the functional map computed from the ground truth correspondences (LFMGT).In Table 3 we added a comparison with multiple baselines: Relative [39] , Ortho [32] , Linear [28] , Procrustes [29] . With the same foundation models and settings used in Section 4.3, we perform a retrieval task on the Caltech-UCSD Birds-200-2011 (CUB) dataset. We demonstrate that LFM is consistently superior in performance and can be used on top of any method which computes an explicit mapping between spaces.In Figure 10 we show additional qualitative results on theMNIST, FMNIST, Cifar10 datasets, akin to the experiment in Figure 5 in the main manuscript.  


Decision: ### Summary
This paper proposes a new approach for comparing neural representations by approximating the associated manifold with a kNN graph and utilizing functional maps, a technique that leverages the spectral properties of the manifold to compute relationships between distinct representations, such as similarity.



$$$$

### Strengths
- **Significance**: The paper tackles an interesting and practically relevant problem. The proposed approach presents a novel combination of existing techniques in a simple and elegant manner, which could inspire future research. 
- **Experiments**: The experimental results effectively demonstrate the efficiency of the proposed approach.

$$$$

### Concerns
- **Clarity**: The writing could be improved to make the paper more self-contained and accessible to non-experts. Some aspects of the method and the experimental setup should be explained more explicitly. Additionally, a more thorough discussion of related work would be beneficial.

- **Limitations**: The paper does not address certain technical limitations, particularly those related to partial manifolds and the design of descriptors in general settings.

$$$$

### Overall
Considering the contributions of the paper -- particularly its simple yet effective modeling approach and potential impact -- along with the efforts made to address the reviewers' questions and the additional experiments provided in the rebuttal, I believe this paper is a reasonable contribution to the conference and recommend acceptance. While the clarity was criticized, the requested improvements can be implemented, and a discussion of the limitations can be included.

Reviews: review: paper_summary: paper_summary: The paper tackles the problem of modeling relationships between latent spaces learnt by different models. It proposes using functional maps that have been used in 3D vision and graph matching for the purpose. It does so by approximating the latent space structure using a knn graph constructed using anchor points, on which the functional map (LFM) is defined.

LFM is used to stitch together encoders and decoders from separate CNNs trained independently, where it is shown to perform significantly better and with lesser variability than previous work, while simultaneously using less anchors than them.
strength_weakness: strength_weakness: **Writing:** The writing quality in the paper could be significantly improved in terms of:

1. Including more details about the experiments in the main manuscript (defining the goal of the experiment formally, detailing backbones, optimizers used in training, specifying which models/layers are stitched). An example of such a lack of detail is Sec 5.2, L235.
2. Describing closely related past works [22], [29] in more detail.
3. Clarity and organization of Sec 3, 4.2.

**Limited Experiments:** The current experimental evaluation seems very limited to models trained on very small datasets, e.g. only CIFAR-100 for the zero-shot stitching despite the relative efficiency of the method. I would like to see if the method performs well on more realistic datasets with more complex latent spaces like ImageNet, or even smaller ones like CUB200.
questions: questions: I would like to know if the gains in performance due to the method still hold for more practical netowrks that have larger, more complex latent spaces like ImageNet
limitations: limitations: The authors have sufficiently addressed limitations
paper_summary: paper_summary: The paper presents a new way to compare neural representations: Latent Functional Maps (LFM). The later one is achieved by 1) building symmetric knn graphs 2) calculating Laplace eigenfunctions 3) calculating optimal mapping between them.
Applications include: (i) compare different spaces in an interpretable way and measure their intrinsic similarity; (ii) find correspondences between them, both in unsupervised and weakly supervised settings, and (iii) to effectively transfer representations between distinct spaces. 
Experiments with CNNs and word embeddings are provided.
The language is fine and the manuscript is well organized, but details of the proposed method and experiments are hard to understand.
strength_weakness: strength_weakness: 1. The notion of Functional Maps is applied "as is", without significant modifications. So, from the theoretical perspective, there is no contribution.
2. Many details of the proposed method and experiment are missing, instead it authors refer to original papers, see below. 
It makes the manuscript hard to read. In my opinion, the paper must be self-contained.
Settings of these experiments are not a common knowledge. You can include several small paragraphs introducing them to a reader. Examples:

* line 152: and incorporate regularizers for Laplacian and descriptor operator commutativity, as defined in [31]:
* line 158: in the other latent space. Once we have solved the optimization problem defined in Equation 2, we refine the resulting functional map C using the algorithm proposed by [24].
* line 235: We test the use of the latent functional map in the task of zero-shot stitching, as defined in [29],
* line 185: This functional alignment can be used similarly to the method proposed by [29] to establish a "relative" space where the representational spaces X and Y are aligned.
questions: questions: 1. Is the LFM score symmetric?
2. Is it a metric?
3. What is a descriptor operator? (line 152)
4. What is stitching of latent spaces?
5. Why the method from Moschella et al. is not used for comparison?
6. Why do you use this particular set of functions:
"As a set of corresponding functions, we use the geodesic distance functions"
7. line 145, 146: do you have one-to-one correspondence between X and Y ?

Moschella, L., Maiorca, V., Fumero, M., Norelli, A., Locatello, F., & Rodolà, E. (2022). Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430.

**Other:**

I think that the proposed idea is intersting, especially becase it gives not only a similarity score, but tools for scrutiny of embeddings.
But the paper has issues with clarity (see above).
limitations: limitations: none
paper_summary: paper_summary: This paper proposes using the functional maps paradigm for comparing and aligning the latent spaces of different neural architectures, possibly trained with different setups: initialization, different datasets, noise, etc. 

The main contribution of this paper is to view the latent space as a Riemannian manifold, and comparing between such spaces is modeled as a correspondence problem. The authors leverage the functional maps paradigm prominent in geometry processing that effectively uses the spectral geometry of manifolds to obtain a compact and accurate representation of the map.

The authors demonstrate their idea on 3 (mostly synthetic) setups: (1.) Using F-Maps as a measure to compare between spaces (Sec 4.3), (2.) Stitching latent spaces (Sec 5.2) and (3.) Retrieval using word embeddings. The experiments demonstrate that the functional paradigm is quite good when the number of anchors (ground truth correspondences) is less making a case for the effective use of spectral geometry in such applications
strength_weakness: strength_weakness: - The submission builds considerably on the ideas introduced in [29] and it is important to highlight and compare the key conceptual differences (aside from just "using" F-Maps)
- The success of this approach depends on the availability of good anchors and label-based desriptors. Designing these descriptors for 3D shapes (HKS/WKS/SHOT) was straightforward. It is unclear from the submission how these descriptors can be generalized in the setting of representation spaces?  
- Figure 2 is misleading if the colorbars are not the same. 
- I found the baselines used for comparison to be rather limited. Why not compare with [18, 22, 25, 27, 20, 13]? To this aid, it is important to show some reconstruction examples similar to Figure 4 in [29] and compare with it 
- As a general comment, the experimental section seems very hurried. Figures 3 and 5 need a more descriptive caption explaining what is to be learned there? 
- It would help to make a figure/experiment showing the functional map matrices. While it is acceptable to digest that "similar objects have similar representations" and that "global distances are preserved", I am not fully convinced that this translates into an exact near-isometry model for latent representations. A good evidence would be to show F-Map matrices in different settings. To this aid, there could be some adjustments in the laplacian and descriptor commutativity constraints in Eq 2 and I find it a bit disappointing to not anything new here.
questions: questions: - What is the significance of comparing LFmaps + Ortho with only Ortho? 
- Why is the variance in the Fine Labelling case (4b) more than (4a)?
limitations: limitations: I do not see any direct negative social impact. Overall: I think this is a submission with some interesting findings and the use of functional maps in this setting is quite refreshing. However, I lean negatively on (1.) The lack of good baselines (2.) Stronger evidence on the applicability of functional maps (3.) Lack of clear communication on: progress over [29] and the experimental section not being fully self-contained.
paper_summary: paper_summary: The paper demonstrates the possibility to use the functional map tool on the embedding space of neural networks. The idea is that the embedding space of a neural network is usually lay on a low dimensional manifold, and networks that are trained for the same tasks even with different architecture result in similar manifolds in the embedding spaces. It was suggested in the paper to use the functional map as a tool for representation transfer for various tasks. The authors showed that using the functional maps improves over previous methods on several benchmarks (similarity measures, stitching, and word retrieval).
strength_weakness: strength_weakness: There are several concerns regarding this paper that should be addressed by the authors before upgrading the rank. First, it is unclear why there is a need for least squares for computing the functional maps as the correspondence is known by construction. Why not calculate it directly? It is more accurate and does not require the zoomout post-processing. If the authors believe otherwise, they should add an ablation study showing it and explain why this is the case. Additionally, even if it was decided to calculate it via least squares, why the chosen descriptors were used the way they are - as there is no guarantee for their consistency in the two latent spaces. The authors should explain this important point, and may do an ablation study regarding the appropriate descriptors.
Second, In general, I would like to see more experiments (or at least similar experiments on more datasets) as in other papers in the field [1]. Additionally, in the stitching experiment, to build the graph the authors used 3000 points with 300 neighbors per point, in that case calculating geodesic distances is almost similar to calculating Euclidean distances in the latent space, which might impair the descriptors. Moreover, the method was not compared to the other baselines. The retrieval experimental setting is unclear.
Small issues: I think there is a mix up when using the notation X, Y, M and N (e.g. see caption of the overview figures). 
[1]  Maiorca, Valentino, et al. "Latent Space Translation via Semantic Alignment." Advances in Neural Information Processing Systems 36 (2024).
questions: questions: see above sections.
limitations: limitations: The authors should add the known limitations of the functional maps, such as the setting of full and partial manifolds [2, 3, 4]. Additionally, experiments that would significantly strengthen the claims are those that would show that the method presents the same limitations as those used in functional maps for 3D analysis. E.g an experiment showing the part-to-full manifold limits (e.g. training a network with part of the data that another network was trained on, and showing a slanted-diagonal structure of the functional map).
[2] Litany, O., et al.. "Fully spectral partial shape matching." In Computer Graphics Forum, vol. 36, no. 2, pp. 247-258. 2017.
[3] Rodolà, E., et al.. "Partial functional correspondence." In Computer graphics forum, vol. 36, no. 1, pp. 222-236. 2017.
[4] Bracha, A., et al.. "On Partial Shape Correspondence and Functional Maps." arXiv preprint arXiv:2310.14692 (2023).

Comments: title: Response to reviewer's comment
Rebuttal Response
Response to Reviewer's comment
Answer
Following up comment on the rebuttal and review
Following up comment on the rebuttal and review
Response to Reviewer's comment
Response to Reviewer comment
Authors made a nice effort.
Reply to Rebuttal
comment: We thank the Reviewer for their feedback and for adjusting their score.

Regarding the unsupervised descriptors: we agree that the development of fully unsupervised descriptors is a very relevant direction, we believe it's achievable by encoding information that remains consistent across domains, within a single graph representation.

To illustrate this, we tested two unsupervised descriptors on the same descriptor ablation experiment of Figure 4 in the Rebuttal pdf. The new descriptors are computed in the following way:

- For the first descriptor, we computed the K-nearest neighbor distances for each node, sorted in increasing order based on the distance function used to build the graph (e.g., cosine distance).
- For the second descriptor, we concatenated to the first descriptor to a fraction  of K (specifically 1/10 of K) of the farthest nodes in terms of distance.

The goal is to capture both intra-cluster and inter-cluster statistics. We refer to these descriptors as "K-nn (unsupervised)"  and "K-nn + farest (unsupervised)". Their performance, alongside other descriptors, is presented in the table below.


|      MRR      | dist_geod (supervised) | dist_cosine (supervised) | labels (weakly supervised)|   Hks (unsupervised)   |  Wks (unsupervised)  | **K-nn (unsupervised)** | **K-nn + farest (unsupervised)** |
|:-------------:|:---------:|:-----------:|:------:|:-------:|:------:|:---------------------:|:------------------------------:|
| **Value**     |  0.9491   |   0.9491    | 0.9291 |  0.0436 | 0.0111 |        0.6793         |             0.7341             |


While these unsupervised methods do not fully match the performance of their supervised counterparts, they achieve strong MRR scores. This demonstrates the feasibility and potential of unsupervised descriptors, which we plan to explore further in future work.

We hope this addresses the Reviewer's concerns on this and we remain available for any further clarifications or questions throughout the remainder of the discussion period.
I thank the authors for the rebuttal. Broadly, I am ok with the response and mostly like what I see (Figure 2,4 and Table 2 in the response pdf). I am still concerned with the descriptor problem and, it appears one needs good concrete landmarks to get a good FMap. It is unclear how to generally make such descriptors for representation spaces - fully unsupervised as rightfully pointed out. 

I would still critique the authors for a less favorable presentation (even in the rebuttal pdf, there are no captions for each figure and the fonts are too small - making it rather cumbersome to see what issue was being resolved with these additional experiments). Nevertheless, this is an interesting paper that could be compiled much better for long-term impact. Although I am not 100% convinced, I think I can live with a borderline accept score.
We thank the Reviewer for their response and for adjusting their score.

We agree that further exploring the theoretical aspects of latent functional maps would be valuable for future research. A potential starting point could be to better analyze convergence rates of graph Laplacians' spectra to the Laplace-Beltrami Operator's spectra for KNN graphs [5] in our setting, considering that dimensionality and the approximation strategy of the manifold are two key differences in our setting compared to the 2D dimensional manifolds in the original functional map framework.

We will include a brief discussion of this in the paper’s conclusions and, as discussed, provide comprehensive, self-contained descriptions of the experiments in the main paper.

We remain available throughout the remainder of the discussion period for any further questions.

_[5] Calder, J., & Trillos, N. G. (2022). Improved spectral convergence rates for graph Laplacians on ε-graphs and k-NN graphs. Applied and Computational Harmonic Analysis, 60, 123-175._
Thank you for a detailed response. I agree that "combining established methods to address new problems is a valuable contribution", my point was that the paper doesn't contain new theoretical results. I acknowledge the novelty of the paper in a context of deep learning. I think that descriptions of experiments must be more detailed to make reader's task easier.
Overall, I'm raising my score.
We thank again the Reviewer for their feedback.

We hope that our rebuttal has adequately addressed their concerns, and we kindly request for feedback on this, particularly in light of the positive responses to the rebuttal from Reviewers 5Rrs, yAi6.

We remain available for any further questions or concerns during the remainder of the discussion period.
We thank again the Reviewer for their feedback.

We hope that our rebuttal has adequately addressed their concerns, and we kindly request for feedback on this, particularly in light of the positive responses to the rebuttal from the other Reviewers: 5Rrs, yAi6 and neES.

We remain available to clarify any further questions or concerns during the remainder of the discussion period.
We thank the reviewer for their response and for adjusting their score. We remain available for any further questions during the remainder of the discussion period.
We thank the reviewer for their response and suggestions.

We will include the discussion on the computations of the FM, the geodesic descriptor, and the experiments performed in the rebuttal period in the main paper.

Since the reviewer has expressed satisfaction with our rebuttal, we kindly ask for consideration to adjust their score accordingly, as also mentioned in their initial review.

We remain available for any further inquiries during the remainder of the discussion period.
The authors' made an effort addressing our concerns regarding the computation of the FM, the geodesic descriptor, and the limited experiments. The explanations are clear and satisfactory. We hope that the motivation for the FM computation and the process of construction geodesic descriptors would be included in paper.
I have gone through the rebuttal and other reviews, and it takes care of my concerns on experiments, so I increase my score.