Relative Representations: Topological and Geometric Perspectives

Authors: alejandro  garcía-castellanos
giovanni luca marchetti
danica  kragic
martina  scolamiero
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-17 09:48:11

Abstract: Relative representations are an established approach to zero-shot model stitching, consisting of a non-trainable transformation of the latent space of a deep neural network. Based on insights of topological and geometric nature, we propose two improvements to relative representations. First, we introduce a normalization procedure in the relative transformation, resulting in invariance to non-isotropic rescalings and permutations. The latter coincides with the symmetries in parameter space induced by common activation functions. Second, we propose to deploy topological densification when fine-tuning relative representations, a topological regularization loss encouraging clustering within classes. We provide an empirical investigation on a natural language task, where both the proposed variations yield improved performance on zero-shot model stitching.

Main paper: 
 introduction and related work: 
 The ability to infer semantically-rich representations is, perhaps, the cornerstone of the success of contemporary deep learning models [4] . Latent spaces of deep neural networks extract features from data that are general and transferable, meaning that, after training a model on a given task, its representations can be leveraged upon for addressing a variety of related tasks. This principle is nowadays exploited in the form of foundation models [5] -neural networks trained via selfsupervision on large-scale multi-task datasets to extract ready-to-use representations.Surprisingly, it has been argued that neural networks trained on diverse tasks, architectures, and domains, infer structurally similar representations [34, 25, 38] -a hypothesis sometimes referred to as 'representational universality' or 'convergent learning' [29] . For example, there is extensive empirical evidence that the representations extracted by neural networks are isometric (up to scale) -i.e., are related by an affine distance-preserving transformation -as the hyperparameters and initialization vary [35] . Even though partial theoretical explanations have been proposed -e.g., based on harmonic analysis [32, 11] or on philosophical principles [22] -these phenomena remain mostly mysterious. Nonetheless, they have motivated the introduction of techniques for zero-shot transfer and latent space communication, such as model stitching [28, 2, 12] -a method consisting in a trainable layer that connects latent representations of two different networks. On a similar note, relative representations [35, 7] involve a non-trainable isometry-invariant layer on top of the representation. Assuming the representational universality hypothesis, factoring out rigid transformations results in features that are independent of nuances in initialization, hyperparameters, and, to some extent, task and architecture.In this work, we propose two improvements of relative representations, enhancing the resulting zero-shot model stitching. These improvements are inspired by insights from geometry and topology, respectively.Geometric Perspective. From the geometric side, we consider symmetries of neural networks [18, 15] , i.e., transformations of the weights that do not alter the function defined by the network. These transformations are sometimes referred to as intertwiners [17] , and form a group that depends on the activation function. We argue that these symmetries are partially responsible for the universality of representations with respect to initialization and training noise, and consequently design a relative transformation that is invariant to the intertwiner group of common activations. Our idea is simple: we propose to deploy batch normalization before the relative representation layer. This factors out non-isotropic rescalings, which are the non-isometric transformations induced by intertwiner groups for a wide class of activation functions.Topological Perspective. From the topological side, we draw inspiration from topological data analysis and, in particular, from methods for topological regularization [33, 21, 20, 10, 37, 9] . The latter aims at enforcing specific topological features in the latent representation of a neural network. To this end, a recently-introduced method deemed topological densification [21] forces data classes to be represented in compressed clusters, resulting in a representation that is coherent with the decision boundaries of the neural network. We propose to deploy topological densification in conjunction with relative representations, and explore various alternatives for combining the two. Our intuition is that consistent densified representations share a similar topology, and are therefore more universal, while still preserving generality and transferability. Moreover, topological regularization prevents potential overfitting of large pre-trained models -such as foundation models -when fine-tuning them via relative transformations in low data regimes.We implement and validate empirically both of the above proposals in an experimental scenario similar to the original work [35] . The scenario consists of a natural language task, where domains correspond to different languages, and model stitching enables zero-shot translation. Results show that both our variant of the relative transformation and the additional topological densification result in significantly improved performance with respect to the original version. In summary, our contributions are:• A novel normalized variant of the relative representation that is invariant to the intertwiner group of common activation functions.• The introduction of topological densification for fine-tuning relative representations.• An empirical investigation on a natural language task, showcasing improved performance. 
relative representations: 
 In this section, we overview relative representations [35] -a technique enabling zero-shot model stitching. The core idea is introducing a transformation in latent spaces that increases compatibility, without the need for training additional components. As explained in Section 1, the approach is based on empirical evidence that latent representations inferred by neural networks are, usually, close to being isometric (up to scale).Let φ : X → Z be a representation, i.e., a map encoding data in X to a latent space Z. Moreover, let A = {a 1 , . . . , a k } ⊂ Z be a set whose elements are referred to as anchors, and let sim : Z × Z → R be a function representing a measure of similarity on Z.). The relative representation of z ∈ Z w.r.t. A isT A rel (z) = (sim(z, a 1 ), . . . , sim(z, a k )) ∈ R k .If Z = R m and sim = z • z ′ /∥z∥∥z ′ ∥ is cosine similarity, then T rel is invariant to linear isometries and rescalings. More precisely, if z and all the anchors in A are transformed via z → αU z -where α ∈ R, and U is an orthogonal m × m matrix -obtaining z ′ and A ′ , then T A rel (z) = T A ′ rel (z ′ ). Relative representations enable zero-shot model stitching as follows. If f 1 = γ 1 • φ 1 and f 2 = γ 2 • φ 2 are two neural networks with the same architecture decomposed in a representation map φ • and a head γ • , then one can replace γ 1 with γ 2 • T A rel to exploit the representation inferred by f 1 in conjunction with the head of f 2 , and vice versa. Assuming the representational universality hypothesis, this model stitching procedure extends to cross-domain setups, i.e., for two networks f 1 and f 2 that are trained over two distinct datasets from the same semantic domain (see Figure 1 ). In this case, to obtain coherent anchors, we fix A as an encoding via φ 1 of some data, and exploit a known correspondence between the datasetsbased on domain knowledge -to obtain anchors A * as encodings via φ 2 of the corresponding data. Lastly, for our purposes it will be convenient to resort to an end-to-end training setup, similarly to [27] . This consists in deploying pretrained representation maps φ 1 , φ 2 , and training the networksγ 1 • T A rel • φ 1 and γ 2 • T A * rel • φ 2with the weights of both representation maps unfrozen, using their corresponding domain-specific dataset.In this section, we overview a recent work that investigates the symmetries in neural networks [17] . The central idea is that certain activation functions induce similar symmetries in both weight space and latent representations. These symmetries generate the so-called intertwiner group.Let GL n be the groups of n × n invertible matrices and σ : R → R a function. The latter represents the activation function of a neural network, and we will consequently extend it coordinate-wise as a map σ : R n → R n . Definition 2.2. The intertwiner group of σ is:G n σ = {A ∈ GL n | ∃B ∈ GL n : σ • A = B • σ}.Suppose that σ(I n ) is invertible, and for each A ∈ GL n define λ σ (A) = σ(A)σ(I n ) -1 . It can be shown that, under the mild condition on σ, G n σ is a subgroup of GL n and λ σ : G n σ → GL n is a homomorphism such that σ • A = λ σ (A) • σ. Moreover, for common activation functions such as ReLU, GELU, and sigmoid, all the elements of G n σ can be decomposed as a product of a permutation matrix and a diagonal one [17] . Since permutation matrices are isometries and isotropic diagonal matrices are (non-isotropic) rescalings, this draws a connection with the transformations factored out by the relative representations from Section 2.1.The following elementary result shows that symmetries from the intertwiner group induce symmetries in the latent representations of a deep neural network. Let f (x, W ) be a multi-layer perceptron with input x, activation function σ, and layer-wise weightsW = (W 1 , b 1 , . . . , W l , b l ) with W i ∈ R ni×ni-1 and b i ∈ R ni .For each 1 ≤ m ≤ l, consider the decomposition f = γ m • φ m into the latent representation at the m-th layer and the corresponding head. Proposition 2.1 ( [17] ). For each1 ≤ i < l pick A i ∈ G ni σ , and consider W = (A 1 W 1 , A 1 b 1 , A 2 W 2 λ σ (A -1 1 ), A 2 b 2 , . . . , W l λ σ (A -1 l-1 ), b l ). Then for each m: φ m (x, W ) = λ σ (A m ) • φ m (x, W ), γ m (x, W ) = γ m (x, W ) • λ σ (A m ) -1 . In particular, f (x, W ) = f (x, W ) for all x ∈ R n0 .The above symmetries provide a theoretical explanation for the emergence of structurally-similar representations in networks with different initializations.In this section, we recall the basics of hierarchical clustering, and review the topological densification method from [21] . To this end, let X be a metric space with distance function d : X × X → R ≥0 , and D ⊂ X be a finite subset. In practice, D will represent a dataset in an ambient space X , or in a representation. Definition 2.3. Given ε ∈ R >0 , the truncation graph Γ ε (D) is the finite undirected graph with elements of D as vertices and an edge between x, y ∈ D if, and only if, d(x, y) < ε. Connected components of the truncation graph can be interpreted as clusters of data. This is the idea behind density-based spatial clustering methods [14] . Instead, hierarchical density-based clustering [6] -as well as the related notion of 0-th persistent homology -considers the evolution of clusters as the parameter ε varies. For ε = 0 all the points in D form a distinct connected component. As ε grows, edges are added to the truncation graph. Thisimplies that if ε ≤ δ, each component in Γ ε (D) is con- tained in a component of Γ δ (D). Moreover, two distinct components in Γ ε (D)can merge together and become connected in Γ δ (D). The value of ε at which two connected components merge is called a death time. Since more than two connected components can merge at the same parameter value, death times form a multi-set in R ≥0 with multiplicity given by the number of merged components minus one. This multi-set is denoted by †(D), and it is equivalent to the persistence diagram of 0-dimensional persistent homology. It can be shown that death times coincide exactly with the lengths of the edges of the minimum spanning tree of D, i.e., the connected tree with vertices D and that is of minimal total edge length. This enables to compute the persistence diagram via, for example, Kruskal's algorithm [26] , whose time complexity is O(|D| 2 log |D|).The concept of connectivity presented above can be exploited to design an optimization objective for deep learning, encouraging the model to distribute data densely inside its corresponding decision boundaries. To this end, in the notation of Section 2.2, let φ : X → Z be an encoder mapping data to a latent space Z equipped with a metric d. In practice, we usually have Z = R m , equipped with Euclidean metric. Moreover, suppose that data is subdivided into classes, meaning that there is a partitionD = D 1 ⊔ • • • ⊔ D K ,where K is the number of classes and D i is the data in the i-th class. Definition 2.4 ( [21] ). Given a hyperparameter β ∈ R >0 , the topological densification loss is:R = K i=1 w∈ †(φ(Di)) |w -β|.For a Euclidean latent space Z = R m , it can be shown that R is differentiable w.r.t. φ(D i ) almost everywhere, with an explicit expression for the derivatives. This allows to deploy gradient-based methods to minimize the topological densification loss over (the parameters of) φ.We remark that the content of this section can be extended to higher-dimensional persistent homology [13] . The above definitions corresponds to an explicit construction of the 0-dimensional persistent homology of the Vietoris-Rips complex, given that in this case all the birth times are 0. By considering i-th dimensional homology, it is possible to define persistence diagrams for every i ≥ 0 -see Appendix (Section A). The topological loss can then be extended based on features of the points in the corresponding persistence diagrams. Moreover, other filtration constructions, such as the more efficient Witness complexes [36] , can replace the Vietoris Rips complex. However, following [21] , we focus on the 0-dimensional Vietoris-Rips persistent homology for simplicity, leaving further investigation of higher-dimensional extensions and other filtrations for future work. 
robust relative transformation: 
 In this section, we introduce a variation of the relative transformation that makes it invariant to the intertwiner group induced by common activation functions. This factors out non-isotropic rescalings in the relative representation, resulting in more robust model stitching. The idea behind our new relative representation boils down to introducing a Gaussian normalization with respect to a batch of data, i.e., a simple form of batch normalization [23] (without learnable parameters).In the notation of Section 2.1, let φ : X → Z = R m be a representation and A = {a 1 , . . . , a k } ⊂ Z be a set of anchors. Moreover, the following definition will depend on an additional set B ⊂ Z, which in practice will correspond to the representation of a batch of data. For z ∈ Z we denote by z B its Gaussian normalization w.r.t. B, obtained by subtracting to z the mean of B and by dividing each component of z by the standard deviation of B in the corresponding direction. Technically, we assume that the standard deviations of B are non-vanishing, which is a generic condition. Definition 3.1. The robust relative representation of z ∈ Z w.r.t. A and B isT B,A rob (z) = T " A Brel ( z B ). Intuitively, the introduction of the normalization transforms all the components of the latent space to the same canonical scale induced by B. Therefore, the robust version is invariant to scaled permutations and shifts, as shown by the following result. Proposition 3.1. Suppose that sim is the cosine similarity, and consider an m × m permutation matrix P , a diagonal one D, and a vector h ∈ R m . Denote by A ′ , B ′ , z ′ the image of A, B, z via z → DP z + h, respectively. Then:T B,A rob (z) = T B ′ ,A ′ rob (z ′ )Proof. Let µ be the mean of B and Σ be the diagonal matrix of its standard deviations in the corresponding components. Then, by definition, z B = Σ -1 (z -µ). The mean and standard deviations of B ′ are DP µ + h and P ΣP ⊤ D, respectively. Denote ‹ D = P ⊤ DP . Then:" z ′ B ′ = Ÿ DP z + h B ′ = (P ΣP ⊤ D) -1 (DP z + h -(DP µ + h)) = D -1 P Σ -1 P ⊤ DP (z -µ) = D -1 P Σ -1 ‹ D(z -µ) = D -1 P ‹ DΣ -1 (z -µ) ( ‹ D and Σ -1 are diagonal) = P Σ -1 (z -µ),and similarly for "a ′ i B ′for every a i ∈ A. Since the cosine similarity is invariant to linear isometries (i.e., orthogonal matrices), we obtain: simÅ " a ′ i B ′ , " z ′ B ′ ã = sim Σ -1 (a i -µ), Σ -1 (z -µ) = sim Ä " a i B , z B ä ,which implies the desired claim.As a consequence, the robust relative transformation is invariant to the intertwiner groups induced by common activation functions (e.g. GELU, ReLU, sigmoid) -see discussion after Definition 2.2. Note that this invariance property differs from the one satisfied by the original version of the relative transformation (Definition 2.1). The latter is invariant to isotropic rescalings -or, equivalently, to diagonal matrices with equal diagonal entries -and linear isometries. Therefore, our version trades off invariance to isometries other than permutations with more general non-isotropic rescalings. We claim that this tradeoff is advantageous in high dimensions. An arbitrary orthogonal matrix can be approximated by a permutation one, with the error decreasing as the dimension grows [3] . Since latent spaces of contemporary deep learning models are typically high-dimensional, the robust relative transformation approximately exhibits, to an extent, invariance to arbitrary isometries, together with the added invariance to arbitrary non-isotropic rescalings. We empirically compare the robust version with the classical one as part of our experimental investigation -see Section 4.1.In this section, we explore various options and improvements for combining the topological densification loss (Definition 2.4) with (robust) relative representations. Since topological features of high-dimensional spaces encode semantic information [39] , this additional regularization is expected to improve both the performance of the individual models, as well their zero-shot stitching capabilities, especially in low-data regimes. Moreover, to further improve the latent space similarity between models, we will apply a consistent regularization by using the same β hyperparameter.A fundamental choice is whether to apply the topological densification loss before or after the relative transformation. We refer to these setups as pre-relative and post-relative, respectively. Figure 3 summarizes these options, including their combination consisting of regularizing both before and after the transformation. We will evaluate and compare all of these options empirically in Section 4. Additionally, we consider the problem of constructing data batches for the topological densification loss. Batching is necessary for stochastic gradient descent and, in general, for efficiency. However, topological losses are notoriously subtle to implement in a batched fashion since, intuitively, persistent homology captures global features in data spaces, which cannot be extracted from those of the batches. The original work [21] suggests constructing batches where each data class is equally represented. This is implemented by sampling (with replacement) data from the same class. However, this approach can cause conflicts with other components of the model. In particular, it is incompatible with batch normalization, which is a fundamental ingredient in our robust relative transformation.To address these challenges, we introduce a novel batch construction method. The latter consists in aggregating sub-batches sampled from class-specific datasets, with an additional sub-batch sampled from the original dataset. This results in an efficient batch construction procedure that is compatible with both the topological densification loss and batch normalization. A more detailed description is provided in the Appendix (Section B). 
experiments: 
 In this section, we provide an empirical investigation of our proposed improvements to relative representations. To this end, we train Transformer models on a natural language task. The task is split across languages, corresponding to different domains. The cross-domain stitching procedure enables to transfer between languages, i.e., it implements a form of zero-shot translation.Data. We perform a random subsampling of 1% of the Amazon Reviews dataset [24] , where the task consists in predicting user ratings in a range from 1 to 5 stars. We consider such task in two languages: English ('en') and French ('fr'). The anchor sets A and A * (notation from Section 2.1) consist of translated texts. Specifically, we first select a number of anchors from the English dataset equal to the dimensionality of the latent space (768 dimensions), and then translate them to other languages using Google Translate. A Python code implementing our models and experiments is available at a public anonymized repository: https://tinyurl.com/TopoRelTrans.Training. We use pre-trained RoBERTa models [30] as the base for our representation maps φ, and a single layer on top of these representations for the head γ. The models are fine-tuned end-to-end on the task described above via two approaches: the 'Absolute' case, where no relative transformation is applied during training, and the 'Relative' case, where the relative transformation is employed, as described at the end of Section 2.1. We train the models via the AdamW optimizer [31] with a layer-wise learning rate decay on the parameters of φ. The initial learning rate is 3.5 • 10 -5 and decay rate of 0.65. The learning rate of γ is 2•10 -4 . Training is performed over 40 epochs, with batch size of 16 and gradient accumulation every 6 steps. Additionally, we use a linear cyclic scheduler for the weights of the topological densification loss, which is a common strategy for optimizing combined objectives [16] . For the relative case, we update the 768 anchor embeddings every 500 optimization steps. This reduces the computational cost, and has minor effects on training stability due to the learning rate schedule for φ.Evaluation Metrics. The performance of the is evaluated using three metrics: the Mean Absolute Error (MAE) between user ratings seen as integers, the F 1 classification score, and the classification accuracy (Acc). All the scores are multiplied by 100 for better readability. As depicted in Figure 1 , to test the stitching performance, we match the representation network's language with the test dataset's language, while the classification head is drawn from a network trained in the other language.Our experimental setup is similar to the original work [35] . However, in order to test the models in a more challenging fine-tuning scenario, we train our models on fewer data (only 1% of the Amazon Reviews dataset). Moreover, we deploy a simpler linear head γ, instead of a deeper network. This naturally results in worse reported performance as compared to the original work.In our first experiment, we compare the robust relative transformation (Section 3.1) with the original one, showcasing improved performance on the considered cross-domain stitching scenario. To this end, we experiment with both the original relative transformation ('Relative Vanilla') and our robust version ('Relative Robust'), together with the baseline ('Absolute') where no transformation is deployed. Table 1 reports the results in terms of mean and standard deviation for 5 experimental runs. On the cross-domain setup -i.e., γ = en and φ = fr, and vice versa -our robust version significantly outperforms the original one on all the metrics considered. Specifically, when stitching from English to French, accuracy and F 1 increase by around 9 and 19 respectively, while MAE decreases by 15.The performance gain is even more drastic when stitching from French to English, with an increase of 25 and 40 in F 1 and accuracy, and a decrease of 34 in MAE. The larger improvements in the latter setup can be explained by the fact that the RoBERTa models are pre-trained better in the English language. This results in fine-tuned representations φ whose extracted features are, generally speaking, more transferable, mitigating the effect of the stitching technique deployed. The explanation is confirmed by the fact that the cross-domain performance without relative representations (Absolute column in the table) is better in the English-to-French setup than vice versa. Lastly, we remark that, surprisingly, the robust relative transformation exhibits slightly improved performance with respect to the Absolute version when the domain is not changed, i.e. when both γ and φ are set to either fr or en. This does not happen for the non-robust transformation. We conclude that not only our proposed transformation is advantageous to a large extent for transferring across domains, but the model can benefit from it even for the original task on low data regimes.In this section, we assess the benefits of the topological densification loss in the context of relative representations, as described in Section 3.2. In all the following experiments, we deploy our robust version of the relative transformation, which has been assessed in the previous section. We start by comparing the three possible approaches for implementing the topological densification -see Figure 3 . Initially, we have investigated the prerelative and post-relative options which, however, had negative impact on the performance. For the prerelative case, a possible explanation is that the relative transformation can fail to preserve the topology of latent data. An example of this is presented in Figure 4 , where the representation consists of three clusters, two of which possess (approximately) collinear centroids. These two clusters are merged by the relative transformation, becoming indistinguishable. For the post-relative case, we hypothesize that the model discovers an anchor configuration that yields tighter clusters in the post-relative space, while preserving their spread in the pre-relative space. This is confirmed by examining the distribution of death times (see Figure 5 ), which exhibit a significantly smaller mean in the post-relative space than in the pre-relative one. We believe this configuration can lead to information bottlenecks, because compressing the clusters with this nonlinear transformation might cause a loss of expressiveness in the latent space.We conclude that the best option is deploying the two topological regularization losses simultaneously, which we implement via a linear combination of them with two weights λ 1 (pre-relative) and λ 2 (post-relative). Through an extensive hyperparameter search, we discovered the optimal values of λ 1 = 2 × 10 -3 and λ 2 = 1.8 × 10 -2 , along with a topological densification parameter β = 3 for both the French and English datasets. With this setup, the pre-relative and post-relative distributions of death times overlap -see Figure 6 . This addresses the above-mentioned challenge, and indicates that, with this setup, the relative transformation preserves the topological information.Table 2 reports the performance scores when fine-tuning with topological regularization for 5 experimental runs. As compared to Table 1 , a slight improvement is evident in the cross-domain context. Specifically, for the English-to-French setup, all the scores improve by around 0.5, while they improve by around 1 in the French-to-English setup. A similar increase can be observed when the domain is not changed. This shows that topological densification is beneficial for transferring between domains. However, it comes with a higher computational cost, as discussed in Section 2.3. This additional cost concerns only the training phase, while the zero-shot transfer procedure is unaffected. In this work, we have introduced two improvements to relative representations. These improvements consist in a novel normalized version of the relative transformation, and in the deployment of a topological regularization loss in the fine-tuning procedure. We have investigated empirically our proposals on a natural language task, showcasing improved performance as compared to the original relative representations.Our robust relative transformation is invariant to non-isotropic rescalings and permutations, which are the only symmetries of common activation functions. However, the original relative transformation is invariant to all the isometries (and isotropic rescalings) of the latent space. Even though trading off non-permutational isometries with non-isotropic rescalings is advantageous in high dimensions (see discussion after Definition 3.1), the challenge of designing a robust relative transformation that is invariant to all isometries and all rescalings remains open. This would lead to an even more robust zero-shot model stitching procedure, and therefore represents a fundamental challenge for future research.Another future direction from the topological perspective is exploring topological regularization procedures beyond densification. This includes extensions of the latter to higher-dimensional persistent homology, as discussed at the end of Section 2.3 and in Section A, or alternative regularization losses -see [19] for an overview. The core challenges behind scaling up such regularizers are their computational complexity and the difficulty of batching. Yet, they might result in advantages for either general or specific domains when applying model stitching techniques.is funded by the Hybrid Intelligence Center, a 10-year programme funded through the research programme Gravitation which is (partly) financed by the Dutch Research Council (NWO).In Section 2.3 we describe 0-th persistent homology in relation to topological densification [21] . We now present a concise definition of n-th persistent homology for n ≥ 0, and elaborate on possible generalizations of the topological densification loss (Definition 2.4). For an introduction to persistent homology and topological data analysis we refer the reader to [8] .The persistent homology pipeline can be subdivided in steps:From Data to Geometry. The first step consists in associating a nested sequence (i.e., a filtration) of simplicial complexes to the data. Simplicial complexes are simple geometric objects that generalize the notion of a graph and can be described combinatorially. If data is in the form of a finite metric space D the filtration of simplicial complexes can be for example constructed by computing the Vietoris-Rips complex, the Cech complex or the Witness complex at scale ε, for increasing values of the parameter ε ∈ R.From Geometry to Algebra. The topology of a simplicial complex and in particular its homology can be computed in an algorithmic way through simplicial homology. Fixed a natural number n and a coefficient field, the n-th homology of a simplicial complex is a vector space encoding connectivity of the simplicial complex. The 0-th homology describes connected components, 1-st homology describes cycles that are not bounded by faces, 2-nd homology describes voids and for n ≥ 2 one can think of higher dimensional analogues. In the cases of a sequence of simplicial complexes, n-th homology can be applied to the whole sequence to obtain a sequence of vector spaces and linear maps. This is referred to n-th persistent homology module.Representing Persistent Homology. Persistent homology modules are well known and simple algebraic objects, due for example to the fact that the sequence of vector spaces constituting a persistence module is indexed by R and that it is essentially discrete. Due to Gabriel's theorem, or equivalently the decomposition theorem of modules over a Principle Ideal Domain, persistent homology modules admit a barcode decomposition, they can be decomposed as sum of addends called bars. A bar is completely described by a pair of real numbers: the birth and the death of the bar. The collection of all birth and death pairs of the bars in a barcode decomposition of a persistence module is the so called persistence diagram. Persistence diagrams and their features are used both as data descriptors and for improving data representations in deep learning, for example through regularization. As mentioned in Section 3.2, the original work on topological densification [21] proposes the following batch construction. A batch consists of b sub-batches, where each sub-batch contains n samples from the same class.... More specifically, the dataloader in the original work is implemented as follows (see Figure 7 for an illustration):1. A batch of size b is sampled from the original dataset. 2. For each datapoint x in the batch, a sub-batch is constructed by sampling with replacement n -1 more datapoints with the same class as x.This approach comes with two major issues. Firstly, in the presence of significant class imbalance, classes are likely to be under-represented within batches. This can potentially lead to catastrophic forgetting and unstable training. Secondly, this method is computationally intensive, since training requires processing the original dataset n times in a single epoch. We propose a new dataloader that follows the same high-level principle, but addresses the above concerns. Our dataloader is implemented as follows (see Figure 8 for an illustration):1. For each class, we create a separate dataloader containing all the data from that class. Additionally, we create a standard dataloader containing all the data. 2. We aggregate n samples from each dataloader, resulting in a batch consisting of K + 1 sub-batches, where K is the number of classes.This new dataloader preserves the class structure necessary for applying the topological regularization loss. Its computational overhead is comparable to training directly with the standard dataloader. Moreover, it prevents class imbalance, since each class is guaranteed to be represented at least n times within a batch.The introduction of the sub-batch from the standard dataloader is relevant for the normalization procedure involved in our robust relative transformation. Namely, we use the (encoding via φ of the) sub-batch from the standard dataloader as B (notation from Section 3.1) when fine-tuning robust relative representations. This is crucial, for example, when the original dataset exhibits class imbalance. In that case, the normalization should be performed with a set B that is representative of the original dataset -which is addressed by the sub-batch from the latter -while the topological loss still benefits from class-balanced data -which is addresses by the other class-specific sub-batches. 
 
 Appendix:  


Decision: In light of the reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript, especially regarding the baseline consistency.

Reviews: review: title: title: Novel application of geometry and topology to improving relative representations
main_review: main_review: The paper presents two improvements to relative representations, a method used in zero-shot model stitching: a normalization procedure that enhances invariance to non-isotropic rescalings and permutations, and the introduction of topological densification as a regularization technique to improve the robustness of model representations. The work aims to enhance model performance in cross-domain tasks, demonstrated through natural language experiments where models are fine-tuned for zero-shot translation between English and French. Both improvements are evaluated empirically and shown to improve stitching performance in low-data regimes.

### Pros
- The application of topological data analysis to relative representations is innovative and pushes the boundary of model stitching and zero-shot learning. Additionally, the idea of creating representations invariant to the intertwined groups aligns well with recent trends in invariant representation learning. These contributions are novel within the scope of zero-shot learning.
- The empirical results clearly demonstrate performance improvements over the baseline. The improvements in performance suggest that the techniques could be broadly applicable to many tasks requiring cross-model or cross-domain representation transfer. 

### Cons
- The clarity of the paper is mixed. The introduction and motivations are well articulated, but the technical depth and mathematical rigor may overwhelm readers without a background in topology or geometric deep learning. The figures are helpful though.
- The practical significance is limited by the computational cost of topological regularization, which may hinder adoption in large-scale models or real-time applications.
title: title: Relative Robust Representation with Batch Normalization and Topological Loss
main_review: main_review: The paper introduces a new variant of relative representation, termed “Relative Robust,” by incorporating batch normalization before the relative representation computation and adding a topological loss both before and after the representation step. The proposed method is tested on the task of zero-shot stitching, showing improved performance over standard relative representations.

The theoretical framework is sound, and the explanation of the method is clear and solid. However, there are notable concerns in the experimental section:

	1.	Inconsistent Baseline Performance: The reported performance for vanilla relative representation (particularly in en-en and fr-fr tasks) shows a drastic drop, which is inconsistent with the results from the original paper. This raises concerns about whether the method was correctly implemented. Stronger justification for these results is necessary, as they currently undermine the credibility of the proposed method’s improvement.
	2.	Limited Dataset Reproduction: The experiments seem narrow in scope, especially when compared to the original dataset used in prior work. To reinforce the validity of the results, it would be beneficial to reproduce more experiments from the original dataset, including additional datasets and languages as tested in the original study.
	3.	Topological Loss Efficacy: While the introduction of topological loss is interesting, its contribution to performance improvement is marginal when compared to batch normalization alone. This raises the question of whether the additional computational complexity of topological loss is justified by the limited performance gains. Further analysis of the added computation time and whether it meaningfully enhances the results would be valuable.
	4.	Presentation of Results:
	•	Figures 5 and 6 would benefit from clearer labeling, especially the x-axis.
	•	Tables 1 and 2 could be combined for better readability and ease of comparison.

Overall, while the core idea of improving relative representation through batch normalization and topological loss is intriguing, the current experimental results are not fully convincing. Strengthening the experimental section, particularly with respect to reproducibility and addressing the suspicious baseline results, will be critical to enforce the paper’s claims.
title: title: Very well written paper. All sections are clear and concise. The topics fits very well into the scope of this workshop. The paper fails to describe the vanilla relative representation, which has implications on the validity of the results.
main_review: main_review: The paper proposed two mechanisms to improve Relative Representations for model stitching: Normalization and Densification. The paper has a nice and clear theoretical motivation behind the proposed mechanisms.

1) Normalization suggests a batch normalization layer to become invariant to non-isometric scaling. In 4.1, the authors report the results, comparing the Robust Rel. Repr. to a Vanilla framework, using cosine distance as a similarity measure. It is unclear whether the Vanilla framework includes centering the latent space. If e.g. the latent space is only in the positive domain and far away from 0, all angles will be small. Thus, it is not clear whether the reported improvement comes from the centering or the scaling part of the batch norm.

2) Densification: this improvement is rather small compared to reported standard deviations. I wonder whether other similarity measures e.g. euclidean distance, will benefit more from densification, as projections onto the unit sphere in the cosine similarity will lead to a densification of the latent space.

Even though the baseline method is not entirely clear, the results and theoretical motivation of the proposed methods provide us with a better understanding of the Relative Representation framework. I therefore rate the paper as above acceptance.
title: title: Two modifications to the relative representations method
main_review: main_review: The paper introduces two modifications to the relative representation approach.

The first modification introduces a transformation from absolute to relative representations that is invariant to combinations of permutations, element-wise rescalings, and shifts (eq. at line 190). In contrast, the transformation proposed by Moschella et al. (2023) is invariant to linear isometries and global rescalings (eq. at line 77). The paper’s readability could be improved by comparing these equations side by side or numbering them, which would make it easier for readers to compare the two approaches.

The original and modified methods are invariant to different, though overlapping, classes of transformations, making it an empirical question which will perform better. The authors argue that their proposed transformation is nearly as expressive as the original in high dimensions. Experiments suggest that the new transformation performs better, at least for the task considered.

I found the explanation in lines 199-208 helpful. In my view, the significance of the proposed approach would have been clearer if this motivation had been introduced earlier in the paper. Without it, the reasoning behind this modification of the relative representations approach is unclear, especially since the paper begins with statements such as:
> For example, there is extensive empirical evidence that the representations extracted by neural networks are isometric

which seems to vindicate the original approach. Although the authors provide some motivation for their modification in lines 35-43, it's unclear how these statements align with one another.

The motivation for the second modification, fine-tuning based on topological densification, was not entirely clear to me. Unlike the first modification, it's unclear what class of transformations this fine-tuning is invariant to. In some passages, the authors refer to "decision boundaries", even though their method is meant to work beyond supervised settings. However, the empirical investigation of their hypothesis could still be of interest.

Overall, the paper introduces valuable additions to the relative representations method, which are relevant to the workshop's theme. However, the two contributions feel somewhat disconnected, and the readability of some sections could be improved.

Comments: comment: We are grateful to all the reviewers for their comments and feedback, which we have incorporated into our paper. 

A major concern was raised regarding our reported results for the baseline, which are significantly lower than the ones in the original work [1]. We wish to clarify that this is due to the fact that we drastically subsample the data from Amazon Reviews – a large-scale natural language dataset. We only use 1% of the training data for fine-tuning, as mentioned in Section 4 (paragraph ‘Data’). Moreover, we use a simple linear layer as a classification head $\gamma$, as mentioned in Section 4 (paragraph ‘Models and Training’), instead of a deep network as in the original work. This was done not only for compute reasons, but to also test the models on a more challenging fine-tuning scenario, leading to lower performance scores. We acknowledge that these differences are important to clarify explicitly –  we have included an additional paragraph in Section 4 to this end. 


[1] Moschella et al., Relative Representations Enable Zero-Shot Latent Space Communication, ICLR 2023.