Relative representations enable zero-shot latent space communication
Authors: Luca Moschella
Valentino Maiorca
Marco Fumero
Antonio Norelli
Francesco Locatello
Emanuele Rodolà
Venue: ICLR 2023 notable top 5% (2023)
Keywords: relative representation
zero-shot
stitching
invariance
latent communication
isometry
representation learning

Abstract: Neural networks embed the geometric structure of a data manifold lying in a high-dimensional space into latent representations. Ideally, the distribution of the data points in the latent space should depend only on the task, the data, the loss, and other architecture-specific constraints. However, factors such as the random weights initialization, training hyperparameters, or other sources of randomness in the training phase may induce incoherent latent spaces that hinder any form of reuse. Nevertheless, we empirically observe that, under the same data and modeling choices, the angles between the encodings within distinct latent spaces do not change. In this work, we propose the latent similarity between each sample and a fixed set of anchors as an alternative data representation, demonstrating that it can enforce the desired invariances without any additional training. We show how neural architectures can leverage these relative representations to guarantee, in practice, invariance to latent isometries and rescalings, effectively enabling latent space communication: from zero-shot model stitching to latent space comparison between diverse settings. We extensively validate the generalization capability of our approach on different datasets, spanning various modalities (images, text, graphs), tasks (e.g., classification, reconstruction) and architectures (e.g., CNNs, GCNs, transformers).

Published as a conference paper at ICLR 2023
RELATIVE REPRESENTATIONS ENABLE ZERO-SHOT LATENT SPACE COMMUNICATION
Luca Moschella"
Valentino Maiorca
Marco Fumero
Antonio Norelli-
Francesco Locatello
Emanuele Rodolà
'Sapienza University of Rome
Amazon Web Services
ABSTRACT
Neural networks embed the geometric structure of data manifold lying in high-dimensional space into latent representations. Ideally, the distribution of the data points in the latent space should depend only on the task, the data, the loss, and other architecture-ssecific constraints. However, factors such as the random weights initialization, training hyperparameters. or other sources of randomness in the training phase may induce incoherent latent spaces that hinder any form of reuse. Nevertheless, we empirically observe that, under the same data and model- ing choices, the angles between the encodings within distinct latent spaces do not change. In this work, we propose the latent similarity between each sample and fixed set of anchors can enforce the desired invariances without any additional training. We show how neural architectures can leverage these relative representations to guarantee, in practice, invariance to latent isometries and rescalings, effectively enabling latent space communication: from zero-shot model stitching to latent space comparison between diverse settings. We extensively validate the generalization capability of our approach on different datasets, spanning various modalities (images, text, graphs), tasks (c.g., classification, reconstruction) and architectures (c.g., CNNs, GCNs, transformers).
alternative data representation, demonstrating that it
INTRODUCTION
Neural Networks (NN) learn transform high dimensional data into meaningful representations that are helpful for solving downstream tasks. Typically, these representations are seen as elements of vector space, denoted latent space, which corresponds the constrained output (explicitly or implicitly) of a key component of the NN. e.g., the bottleneck in an Autoencoder (AE), or the word embedding space in an NLP task. The underlying assumption is that the learned latent spaces should be an optimal encoding given the data distribution, the downstream task, and the network constraints.
In practice, however, the learned latent spaces are subject changes even when the above assump- tions remain fixed. We illustrate this phenomenon in Figure [ where we show the latent spaces produced by an AE with two-dimensional bottleneck, trained on the MNIST dataset several times from scratch. These spaces differ from one another, breaking the fundamental assumptions made above. The distribution dom initialization of the network weights, the data shuffling, hyperparameters, and other stochastic processes in the training phase. Although different, the Iearned representations in Figure [ are in- trinsically similar: the distances between the embedded representations are approximately the same across all spaces, even their absolute coordinates differ. Indeed, the learned latent spaces are the same up to a nearly isometric transformation
the latent embeddings is affected by several factors, such as the ran-
This symmetry is a consequence of the implicit biases underlying the optimization process (Soudry) al. 2018) forcing the model generalize and, therefore, to give similar representations similar samples with respect to the task. There exist infinitely many spatial arrangements complying with these similarity constraints, each associated with a different isometry the example of Figure I
"Equal contribution.
Work done outside of Amazon. To the best of our knowledge, the first acknowledge this behavior was Olah 2015 in blogpost.

---

Published as a conference paper at ICLR 2023
Train
Train 2
Train
Train
Figure 1: Latent spaces learned by distinct trainings of the same AE on the MNIST dataset. The bottleneck has size 2, thus there is no dimensionality reduction involved the visualization of the latent space. The stochasticity in the training phase induces intrinsically similar representations. As we show Figure this property holds even for high-dimensional latent spaces.
But while the resulting models will be equally good in terms of the task, one still encounters several practical problems. For example, is notoriously challenging to compare latent spaces across differ- ent trainings or across different NNs; perhaps more importantly, re-using neural components trained different embeddings of the same data becomes impossible, since they are incompatible. To overcome this, we propose adopting a local coordinate system defined by the data itself. Each data point becomes set of coefficients that encode the point function of other data samples, instead of an independent point in R.. The proposed relative representation directly encodes the intrinsic information underlying the data, and only depends on the angles between embeddings by construc- tion. Remarkably, this enables a form of compositionality between learning models; it allows, for instance, to stitch together an encoder trained on ImageNet with a decoder trained on CIFAR, we showcase in our experiments.
Our main contributions can be summarized as follows:
We show that the representations learned by NNs are subject to change due to several training factors; nonetheless, the angles between latent embeddings are preserved.
We introduce novel relative representation for latent embeddings, that is invariant by construction to the transformations induced by stochastic factors in the training process.
For the first time, we successfully demonstrate zero-shot stitching of neural components produced by distinct training regimens, c.g., due to different seeds or different neural ar- chitectures; validate findings on different data modalities (e.g. images, text).
Our framework also provides quantitative measure of performance while training neural models, which is differentiable, does not need any labeled data, and is correlated with standard performance measures such accuracy.
2 RELATED WORK
Representation similarity. Recently, there has been growing agreement that good networks learn similar representations across a variety of architectures, tasks and domains (Morcos et al., 2018
Li al. 2016: Kornblith al. 2019 Bonheme & Grzes 2022 Tsitsulin et al. [020] Barannikov
et al. 2022; Vulić et al. 2020 Lample et al. 2018 Lenc & Vedaldi 2015 Mikolov et al. 2013b
Antonello al. 2021). although this still debated (Wang et al.) 2018) and missing strong theoret-
ical justifications. Similar observations have been made in the context of biological models Laakso
Cottrell (2000); Kriegeskorte et al. (2008); |Raizada Connolly) (2012). Supported by the em- pirical evidence widely reported in these works, our method assumes that well-performing neural networks trained on similar tasks and data produce similar latent spaces, which allows us to define representation that unifies all these spaces.
Model stitching, Lenc & Vedaldi (2015) introduced trainable stitching layers that allow swapping parts of different networks, while Bansal et al. (202I): Csiszárik et al. (202T) employed stitching to quantitatively verify statements such as "good networks learn similar representations" and "more data, width or time better". Other works, such as Gygli et al. (202I): Biondi et al. (202T): Ya-
man et al. 2202); Bianchi al. (202)) tried to directly produce compatible and reusable network
2

---

Published as a conference paper at ICLR 2023
components without stitching layers; more generally, stitching has been adopted in the literature to analyze neural networks. In our work, we sidestep the need for trainable stitching layers and propose zero-shot model stitching to effectively reuse models.
Relative information. The attention mechanism (Vaswani et al., 2017) and its variants (Kossen ct al. 2021) exploit the relationship between features to extract meaningful representations. be Prototypical Networks (Snell al. 2017) learn metric space where the classification performed by measuring the distances to prototype representations. [Shalam & Korman (2022) proposed the Self Optimal Transport feature transform to enrich the sample representations with higher order relations between the instance features, while Alvarez-Melis et al.] (2019) proposed a general formulation of the optimal transport that accounts for global invariances the underlying feature spaces. Mathematically, our method bears resemblance to kernel method (Hofmann et al., 2008) as it employs inner products embedded features as a core ingredient. However, differently from kernel methods, we do not introduce learnable parameters and, crucially, we compute the representations explicitly without resorting to kernel trick.
METHOD
Given training set X. standard NNs learn embedding function Eg X - Ed parametrized by 0, which maps each sample (+)) E X to its latent representation, or absolute representation, Ca(()) Ee(x(() This representation is then exploited to solve downstream tasks, such as classification, reconstruction or generation, optimizing over some objective function of the general form:
min Ex (x)) + Rey(O))
(1)
9
Here, Ex denotes the expectation over the training distribution, and Reg(0) encodes additional constraints on the weights 0. As previously discussed, we argue that the learned weights 0" are not only a function of X and of the specific loss appearing in Equation [] but in practice they are also affected by the optimization process used to train the network due to weight initialization, data shuffling, hyperparameterss and other stochastic factors. We denote these factors collectively by ø. In particular, as shown in Figure [ changing these factors induces a transformation T over the latent space, i.c., o - o' implies E' ()) + T Ee(((())) We make the core assumption that I preserves the angles between elements of the latent space, namely ((eg,0, Cg(n) L(Te Te ) for every (x) (()) X. While this assumption might seem too restrictive, in practice arises in several real scenarios as we show in the following sections.
3.1 RELATIVE REPRESENTATIONS
To build our representation, we start by selecting subset A of the training data X, which we denote as anchor samples. Every sample in the training distribution will be represented with respect to the embedded anchors ean = (ia with a (1) e A. As a measure capturing the relation between the anchors and the other samples, we consider a generic similarity function sim : IR Ed IR, yield- ing a scalar score between two absolute representations = sim (, C2()) ). Given the anchors
sim(e
A in an arbitrary ordering ( (lA)), we define the relative representation of (() E X as:
(4) (sim(eeg)) e. (e, ,ir sim ()(
Figure iillustrates the key differences between absolute and relative representations.
Choice of the anchors. Anchors directly affect the expressivity of the relative representation space, and are related to the task at hand. For example, in a classification task, we should sample anchors from each class in the training set, in order well represent each data sample in X.
One case of interest arises when the data comes from different domains or modalities X, Y, and we given a partial correspondence I' Px + Py mapping from subset of X to a subset of Y. In this case, we can sample anchors Ag C Px and obtain corresponding anchors on the other domain directly T(A) (Norelli et al. 2022). We refer these parallel anchors. We show an example of parallel anchors in Section 5.2 where and Y are Amazon reviews in two different languages.
The choice of the anchors restricted elements in the training distribution. Given an encoder pre-trained on fixed training distribution, we can pick elements from set À that is out-of-domain
3

---

Published as a conference paper at ICLR 2023
simi+ *k)
simi+ pk)
sim(+
Figure 2: Left: Three anchors (colored stars) are selected on the data manifold; given a point on the manifold (blue dot), compute its similarity w.r.t. the three anchors, yiclding vector of dimensionality 3 (middle). Right: Each dimension is treated as coefficients a coordinate system defined by the anchors. Anchors are orthogonal in this example only for visualization purposes.
w.r.t. X, and build the relative representations on top of We refer to these as OOD anchors and ex- ploit them, e-g., solve domain adaptation tasks where do have access correspondence, and have scarce data labels. We refer again to the Sections 5.2 and 5.3 for real-world examples and Appendix A.2 for a preliminary analysis of different selection strategies.
Achieving latent invariance. In this work, we choose the cosine similarity as the similarity function due to the properties it induces on the relative representation. The cosine similarity Sc is the dot product unit vectors, corresponding the cosine of the angle between the two:
ab
Sc(a, b) |/
cos
(3)
|bl
Importantly, cos / does change we apply the same angle-preserving transformation T two vectors a and b, i.c., the cosine similarity is invariant to rotations, reflections, and rescaling. While this not true for translations, NNs commonly employ normalization techniques (e.g., Instan- ceNorm (Ulyanov et al. 2016)) to center the latent spaces. Under this assumption, cosine similarity guarantees a relative representation (n) invariant to ingl-ppreeeving transformations.
This means we have the freedom to change the embedding function Eg with any other function E that produces different representations with same angles, i.e.:
[Sc(e (e
), ................................ Sc(e, Ca(s)] = (Sc() ē.w),... Sc Ca(ss))
(4)
where εn( = (+) = ()) and T is an arbitrary angle-preserving transformation. A practical application of this invariance is the possibility comparing latent spaces across multiple trainings, and re-using models as demonstrated in Sections and
We remark that other choices of similarity function can be made enforce different invariances into the representation. For example, one may impose invariance to non-isometric deformations with bounded distortion. We did not find this be necessary in our experiments, as typically NNs that generalize sufficiently well can handle small perturbations. Nevertheless, this invariance can be enforced by design with vector quantization algorithms. Figure shoow a preliminary investigation, leaving further exploration to future work.
4 LATENT SPACE COMMUNICATION
In this section, we demonstrate how our relative representations can effectively be used to produce latent spaces that are stable under variety of factors. Our main question the following: Given two different learning models that are trained separately different data, can compare their latent embeddings? In asking this, we assume that the two models are trained on similar phenomenon, c.g., two different samplings of the English language
two different modalities.
We answer the positive, showing the gained invariance enables effective communication between different, but semantically equivalent latent spaces. In particular, we analyze how different word em- bedding spaces, once projected onto relative representations, are intrinsically the same (Section 4.1); then show how the similarity between the relative counterparts of two more embedding spaces a surprisingly good predictor of model performance (Section 4.2);; finally, we confirm that relative representations in the training phase are not detrimental performance (Section 4.3)

---

Published as a conference paper at ICLR 2023
4.1 WORD EMBEDDINGS
Experimental setting. We select two different word embeddings the English language, namely FastText [Bojanowski et al. 2017) and Word2Vec (Mikolov et al. 2013a). Both models are pre-trained on different data, but partly share vocabulary from which we extract 20K words. Using 300 randomly drawn parallel anchor, we convert each embedding space to a relative one. In Table (left), we show the original and the relative embeddings. For each word w, we consider its corresponding encodings and in the source and target space. We apply three different metrics to measure their similarity (in a setting similar to Vulić et al. (2020): (i) Jaccard: the discrete Jaccard similarity between the set of word neighbors of in source and target; (ii) Mean Reciprocal Rank: measures the (reciprocal) ranking of w among the top-k neighbors x in the target space; (iii) Cosine: measures the cosine similarity between and y. Additional details in Appendix A.3.I
FastText Word2Vec
Source
Target
Jaccard f
MRR 1
Cosine
Absolute
Absolute
1.00 + 0.00
1.00 + 0.00
1.00 = 0.00
FT
W2V
0.00 0.00
0.00 = 0.00
0.01 = 0.00
FT
0.00 1 0.00
0.00 + 0.00
0.01 1 0.00
W2V
W2V
1.00 0.00
1.00 0.00
1.00 = 0.00
Relative
Relative
FT
FT
W2V
1.00
0 0.01
1.00
0.00
0.86 = 0.00
1.00 = 0.00
FT
0.39 1 0.00
0.00
0.86 1 0.00
W2V
W2V
1.00 1 0.00
1.00 1 0.00
1.00 1 0.00
Table I: Qualitative (left) and quantitative (right) comparisons of English word embeddings using absolute and relative representations. PCA is applied only for visualization. All metrics are calcu- lated with K = 10 averaged over 20k words and across 10 different random seeds. See Figure I0 for other dimensionality reductions and Table 8 and fig. for the same experiment on CIFAR-10.
Result analysis. Table (left) highlights clusters of semantically similar words and shows that the absolute representations are incoherent across the two latent spaces, while the relative embeddings are highly similar. The average Jaccard distance reported in (rihh) says that the word neigh- borhoods of the relative representations are matched exactly 34% of the time in one direction, and 39% of the time in the other one (the missing 61% % is due to semantic differences, that not taken into account by the discrete nature of the Jaccard metric). By contrast, the absolute embeddings never matched exactly (Jaccard score equal zero); for match to happen, it would mean that the F Tast.Text: and Word2Vec embeddings of a given English word are almost the same, which highly unlikely. MRR, close perfect score for the relative representations, shows that the most-similar word to a given one is usually itself, even if their cosine similarity doesn't 't reach 1.
Overall, these results show that relative representations preserved across different word embed- ding models, validating our assumptions.
4.2 LATENT DISTANCE A PERFORMANCE PROXY
Experimental setting. In this experiment, we consider node classification task on the Cora graph dataset (Sen et al. 2008). We first train a reference model that achieves good accuracy on validation set. Then, we train 2000 models with various combinations of seed, number of epochs, number of layers, dropout probability, activation functions, optimizer type, learning rate type of graph embedder (Table 10 All the models are classically trained using absolute representations, which converted to relative post-training by projecting the embeddings onto 300 randomly drawn but fixed anchors. For each model, we measure its classification accuracy and compute the similarity of its space with the reference one. This similarity is computed the average cosine similarity between the node embeddings produced by given model and the corresponding embeddings in the reference.
Result analysis. The scatter plot in Figure 3 (left) shows that better-performing models tend to be the ones with the latent spaces most similar to the reference. The performance-simillrig correlation
5

---

Published as a conference paper at ICLR 2023
Performance
Performance
0.7
Similario
0.6
0.6
0.4
0.3
0.2
6.0
022
0.4
2.0
16
23
Similarity
Epochs
Figure 3: Graph node classification task Cora. Left:
Correlation between the performance of 2000 models and the similarity of their latent spaces with respect to a well-performing reference model. Right: The same correlation plotted over time. The mean Pearson correlation over all models 0.955, after filtering out the models having best validation accuracy below 0.5.
also holds over time, as shown in Figure 3 (riht). Additional correlation examples are Figure Interestingly, this metric is differentiable, enabling explicit supervision signal the latent space, which does not require labeled data and could be readily exploited in a teacher-student framework.
Overall, these results suggest that the similarity between the relative representations of latent spaces a remarkably good proxy to evaluate model performance.
4.3 TRAINING WITH ABSOLUTE VS. RELATIVE REPRESENTATIONS
Experimental setting. Finally, compare architectures that do do not employ the relative rep- resentation while training. In these experiments, the models vary slightly according to the dataset; however, the relative and absolute versions always comparable in terms of architecture, num- ber of learnable parameters and hyperparameters. We refer to the supplementary material and the open-source code for further details their implementation. In this section consider classifi- cation tasks on several datasets, spanning the image domain (Lecun et al. 1998 Xiao et al. 2017 Krizhevsky 2009) and the graph domain (Yang al. 2016).
Table 2: Performance comparison between relative and absolute representations on several image and graph datasets. The metric is the classification weighted Fl score (#: std), over 6 seeds.
Image Classification
Graph Node Classification
MHIST
F-MIIST
CIPAR-10
CIFAR-100
Cora
CiteSeer
PubMed
Relative 97.91 + 0.07
90.19
0.27
87.70 + 0.09
66. + 0.36
0.89 = 0.02
D.77 + 0.03 D.78 4 0.03
0.91 + 0.01
Absolute
97.95 0.19
10.32 0.21
87.80 + 0.06
68.88 + 0.14
0.90 0.01
0.91 0.01
Result analysis. The results, reported in Table show that relative representations, when used at training time, are not detrimental to performance in general. This is further shown Tables (3 and 15 to [8] where subset of the results compares the absolute and relative representations variety of domains, datasets and tasks.
Overall, these results show that relative representations effective when involved in end-to-end training, without significant performance drops.
5 ZERO-SHOT MODEL STITCHING
In this section, we illustrate how the latent space communication demonstrated in Section 4 en- ables zero-shot interoperability of pre-trained neural components. In previous works, such as Lenc & Vedaldi (2015); Bansal al. (2021), stitching layers are trainable linear projections that allow swapping parts of different networks. Instead, on relative representations unlocks the possibility of zero-shot stitching different neural components, treating them as frozen black-box modules.

---

Published as a conference paper at ICLR 2023
We define generic stitched model as the composition of encoder, that embeds data, plus relative decoder specialized in a downstream task (classification, reconstruction). The stitching operation is always performed without training fine-tuning, in zero-shot fashion. Hereafter, we showcase stitching capabilities across combinations of different stochasticity sources (Figure 4 and table 3). neural architectures (Tables 4 and 5) datasets (Table (). Finally, present strong real-world applications in NLP (Section 5.2) and CV (Section 5.3). e.g. zero-shot predictions on novel languages. Additional implementation details are given the supplementary materials.
2
3
4
5
6
7
9
2
3
4
5
6
7
9
Abx.
19
%
2
3
4
5
6
7
9
Rel.
2
3
5
6
7
9
2
3
4
5
6
7
9
Abs.
/
7
6
0
9
3
2
3
4
5
6
7
9
Rel.
2
3
4
5
6
7
9
Figure 4: Reconstruction examples. Each column is different image, row pairs are different archi- tectures. In each pair, we first report the non-stitched reconstructions, then the stitched ones.
5.1 IMAGE RECONSTRUCTION
Experimental setting. We perform zero-shot stitching with AEs and VAEs trained with relative rep- resentations end-to-end on several datasets. For each combination of model and dataset, we perform 5 trainings with different seeds, and zero-shot stitch together the resulting encoders and decoders.
Result analysis. In Figure stitched models that employ absolute representations (Abs.) produce erroneous predictions, since the latent spaces obtained from distinct trainings are incompatible. In- terestingly, although the absolute VAE does not produce compatible latent spaces, is regularized, thus all embeddings produced by the encoders correspond to wrong but semantically meaningful reconstructions. Relative representations (Rel.) exhibit almost indistinguishable reconstructions be- tween the models trained end-to-end and the stitched ones. Quantitative results are in Table 3
These results support our claim that relative representations are empirically invariant to training stochasticity.
Table 3: Stitching performance. The MSE (#: std) between the ground truth and the reconstruc- tions is computed over different seeds. Stitching with relative representations yields error up to two orders magnitude less than the absolute counterpart.
MNIST
F-MIIST
CIFAR-10
CIFAR-100
MSE.
Abs.
Non Stich.
0.66 0.02
1.57 :: 0.03
1.94 1 0.08
2.13 1 0.08
1.58 1: 0.05
AB
Stitch.
97.79 07.7942.44
120.54
6.81
86.74 + 1.37
97.17 + 3.50
100.56
4.29
Rel.
Non-Stitch. Stitch.
1.18 0.02
3.59 0.04
2.83 + 0.13
3.50 + 0.08
2.78 0.07
2.83:.0.22 0.20
6.37 = 0.29
5.39 - 1.18
18.03
12.46
8.16 + 3.53
ReL AWS.
Non. Stitch.
1.00
4.38 0.03
2.68 1 0.06
3.00 1 0.03
2.84 1: 0.04
MAR
69.02 - 1.54
78.57 + 1.88
91.27
Non-Stitch. Stitch.
2.97
6.81 0.06
5.18 + 0.22
5.93 + 0.14
5.22 0.14
13.43 6.79
24.03 1: 13.15
11.20 1 3.15
11.23 1 2.38
14.97 6.37

---

Published as a conference paper at ICLR 2023
5.2 TEXT CLASSIHICATION
In this Section, we show practical examples of the use of parallel anchors (Sec 3.1
Table 4: Cross-lingual stitching performance comparison. The table reports the mean weighted FI
(#: std) and MAE on Amazon Reviews coarse-grained, across 5 seeds.
Absolute
Relative
Trunslated
Wikipedia
Decoder Encoder
FScore
FScore
MAE
FScore
MAR
em
91.54 1 0.58
0.08 1. 0.01
90.06 1 0,60
0.10 1: 0.01
90.45
1 0.52
0.10 1 0.01
43.67
+ 1.09
0.56
4 0.01
82.78 + 0.81
0.17 0.01
78.53
0.30
0.21 + 0.00
en
Ir
54.41
1: 1.61
0.45
1 0.02
78.49
1 0.66
0.21 1: 0.01
70.41
1: 0.57
0.29 1 0.01
ja
48.72
4 0.90
0.51
0.01
65.72 + 0.55
0.34 0.01
66.31
0.34 + 0.01
Table 5: Cross-architecture stitching performance comparison. The table reports the mean weighted
Fl (#: std) for each dataset, across 5 different seeds.
TABC
Jepedia
Amazen Aeviews
Coarse
Fine
Rel. Abc.
Non-Stitch 91.70 + 1.39
98.02 0.58
87.81 + 1.58
55.35 + 3.19 19:01
Stitch
21.49 3.64
6.96 1.46
49.58 = 2.95
2:01
Non-Stitch
88.08 + 1.37
97.42 2.05
85.08 + 1.93
48.92 + 3.57
Stitch
75.89 5.38
21.14
72.37 = 7.32
33.21 7.21
Experimental setting. We consider two different text classification settings.
Cross-lingul: given review predict the associated star rating, done on multi- multi-linguul data from the Amazon Reviews ews dataset (Keung et al. (020). Following the original paper, we work on a binarized version of the task, with FScore and MAE as metries. In the supplementary material, report results the fine-grained formulation. We adopt four different pre-trained language- specific RoBERTa transformers (Liu et al., 2019) and evaluate their zero-shot stitching performance on languages never seen by the classifier. We use parallel anchors in two modalities: i) Translated: consider English reviews translated into the other languages; ii) Wikipedia: adopt an external cor- pus, WikiMatrix (Schwenk et al. 202T), providing parallel sentences extracted from Wikipedia.
Cross-architecture: assessed on three different datasets: TREC (coarse) (Hovy et al. 2001), DBpedia (Zhang al. 2015), Amazon Reviews (English split). We adopt two different pre- trained BERT (Devlin et al. 2019) transformers (cased and uncased version), ELECTRA (Clark)
al. 2020 and RoBERTa.
Result analysis. Tables 4 and show for the first time that it is possible to learn to solve down- stream task on specific language or transformer and perform predictions on another.
Stitching with absolute representations yields performances comparable random guessing across the board, proving that relative representations are a key element for the success of this kind of zero-shot stitching. Moreover, Table 4 highlights the robustness that relative representations have on the choice of anchors, even when they are noisy (Translated case), or their distribution differs from the one of the downstream task (Wikipedia case), long as their encoding can be handled correctly by the encoder. In our case, the encoder is pre-trained to represent variety of texts in specific language, thus, even if WikiMatrix has completely different domain from Ama zon Reviews, the transformer still computes meaningful and comparable representation with those of the reviews. We report in Tables [5] and 16 in Table 17 the performance obtained by complete results on all languages combination, and transformer. To the best of our knowledge, it is the only altermative for obtainining compatible representations across languages.
According to these results, relative representations show invariance to different architectures and data distribution shifts (e.g., different train languages).
We used the =G0GGI SLATE function available in Google Sheets.
8

---

Published as a conference paper at ICLR 2023
5.3 IMAGE CLASSIHICATION
In this Section, we show practical examples of the use of OOD anchors (Sec 3.1
Table 6: Stitching performance comparison with different encoding techniques. The table reports the mean weighted F1 (# std) CIFAR-100 coarse-graing and ImageNe 1,, across 5 seeds.
C1FAR-105 100
mmagetietls 1k
Decoder
Encoder
Absolute
Relative
Absolute
Relative
rexnet- 100
82.06 + 0.15
80.22 + 0.28
73.78 0.29
72.61 = 0.16
vi--base-aatch16-2224
54.98 + 0.41
37.39 0.36
rexact-100
vit be-bbse-essnt5--386
53.33 + 0.37
42.36 + 0.36
small patch16.22
59.82 1 0.32
43.75 0.27
rexnet-108 100
76.81 + 0.49
30.78 + 0.81
vit- vit-huse- patch1 6. 6-224
16
0.10
80.91
vit- ressertrsnntt0-33.. 384
6.21 0.33
81.42 + 0.38
0.07 0.05
44.72 + 0.57
vit-small-patchh6--2
84.29 + 0.8G
48.31 0.72
resnet- 100
79.79 +: 0.43
53.46 +: 0.68
vil-buse-cesnctf--38
vif-base-petehl6-2244
4.69
84.46 0.19
0.08 0.04
62.21 0.54
vit- hil-base-resnttS--BB
91.41 + 0.09
90.77 + 0.16
82.55 + 0.30
81.88 = 0.16
putch16-225
rexnet-100 100
73.35 0.41
37.58 0.44
small patch
base p6tch16 224
81.23 1: 0.31
50.08 0.63
vit base rew.et50.00ee.
78.35 1: 0.89
45.45 1.41
vit-small-patchiee 1atchh 90.07 0.119
88.85 + 0.44
77.73 0.41
76.36
0.40
Experimental setting. We consider a classification task on ImageNet 1k and CIFAR-100 with coarse labels (20), and 4 different pre-trained image encoders: three variants of the ViT transformer (Dosovisskyy et al., 2021) and RexNet (Han al. 2020).
Result analysis. The results in Table 6 highlight how the relative representations allow stitching modules with different encoding dimensionality. since the decoder receives relative representation with guaranteed equal size. Further, the results demonstrate the ability to generalize and perform zero-shot stitching on CIFAR-100, although that data was never seen by the encoder since it is frozen transformer trained on ImageNet 1k. Interestingly, rexnet-100 1000 is the only transformer whose latent dimensionality is higher than the number of anchors, and the biggest drop in stitching performance happens when the decoder is trained on it. This suggests the number of anchors is an important hyperparameter; refer Figure for for
deeper analysis.
Overall, these results prove that relative representations can bridge general-purposs encoders and pre-trained task spec-spe decoders.
6 CONCLUSION
In this work, we introduced the concept of relative representations to enable zero-shot latent space communication, with several practical consequences as showcased in our discussion and experiments. Our work proves that latent semantic correspondence between data domains, when present, can be exploited through a simple representation shift, without resorting to sophisticated processing or heavy training.
Limitations and future work. Our work is open several follow-up directions. While in this paper we considered the cosine similarity. different functions can enforce additional invariances in the relative representation. The study of invariant latent spaces as a general direction has the potential to lead to further impact; in Figure 7 we showed preliminary results of this possibility. Another interesting line of research to improve the representation expressivity would be to estimate geodesic distances over the data manifold instead of adopting Euclidean approximations. Similarly, we believe that the connections between the composition of the anchors set and the expressivity of relative representations demands additional research. For example, the training cost is directly affected by the number and update frequency of the anchors. Finally, the stitching procedure may be extended to multiple layers, promoting reusable network components.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors gratefully acknowledge the anonymous reviewers for the thoughtful remarks, and Luigi Gresele for the insightful discussions. This work is supported by the ERC Starting Grant No. 802554 (SPECGEO).
REPRODUCIBILITY STATEMENT
We describe in detail the relative representation computation in Section We describe the perimental settings for the various scenarios, and refer to the supplementary material for further implementation details (Appendix A.5). Moreover, release well-documented and modular codebase, with the relative representation layer being implemented as stand-alone PyTorch mod- ule. All the checkpoints used in the experiments are versioned with DVC (Kupriciev et al., (023) to easily reproduce all the figures and tables, The stand-alone module allows the integration of the relative representations in any existing neural network effortlessly.
REFERENCES
David Alvarez-Melis, Stefanie Jegelka, and Tommi Jaakkola. Towards optimal transport with global invariances. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Oki- PMLR, mmr.. ppeess/v8 wtlp///laaeeeeeeeeee -ereez-meeiiee 9a..19..... html
nawa, Japan, volume 89 of Proceedings of Machine Learning Research, pp. 2019. httLLttt://poceeeeetee
Richard Antonello, Javier Turek, Vy Vo, and Alexander Huth. Low-dimensional struc- ture in the space of language representations is reflected in brain responses. In M. Ran- zato, A. Beygelzimer. Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 8332 8344, Curran Asso- ciates, Inc., 2021. URL httttss:/prroceediiee meer neuri ps. cccppaper//02//EE1 4660744 79b72cbb 7 waw.aaaac6994444 P pd
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34; Anmual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, Decem- ber 6-14, 2021, virtual, pp. 225-236. 2021. URL htLhst:s::/ http:::::/pppooeccingge.o neurips cc/ paper/200 1a002//hashe hww.h/aaah/aaaa 01chh/11dddde ded425 111011ffet (et.feeeb3b0033999eeee00 9b0 www0036633330 9e Abs Abst .atttractttmmlll. .ttml
Serguei Barannikov, Ilya Trofimov, Nikita Balabin, and Evgeny Burnaev. Representation topol- ogy divergence: A method for comparing neural network representations. In Kamalika Chaud- huri, Stefanic Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceed- ings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1607-1626. PMLR, 2022. URL httpt:ttp////mm htttsst://poooeeeeeeee http:///pooeeeen....... ngg.
mlr. www.press/vll2. 2p2ess//112/barra..ko2222 wwt.////aaaannnneeeeeeeeeee .html
Federico Bianchi, Jacopo Tagliabue, Bingqing Yu, Luca Bigon, and Ciro Greco. Fantastic em- beddings and how to align them: Zero-shot inference in a multi-shop scenario. ArXiv preprint, abs/2007. 2020. URL https:: //appis////xxv...//////s org/ abs 2007 14906
Lukas Biewald. Experiment tracking with weights and biases, 2020. wandb. com/ Software available from wamdb.com.
URL https:// //www.
Niccolo Biondi, Federico Pernici, Matteo Bruni, and Alberto Del Bimbo.
Cores: Compati-
ble representations via stationarity. ArXiv preprint, abs/2111.07632, 2021. URL https:
/aaxiiv.org/abe/111..1r3
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Emriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017. doi: 10. 00.1162/aac...00001 htw.:/S....LLLL.......
org/ 0070/011-110 1010
Lisa Bonheme and Marek Grzes. How do variational autoencoders leam? insights from representa- tional similarity, 2022,
abs 08399
10

---

Published as a conference paper at ICLR 2023
Kevin Clark, Minh- Luong, Quoe V. Le, and Christopher D. Manning. ELECTRA: pre- training text encoders as discriminators rather than generators. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenRe- view.net, 2020. URL
net. fert://orummmd-eomm XMH1 Btva
Adrián Csiszárik, Péter Kórösi-Szabó, Ákos K. Matszangosz, Gergely Papp, and Dániel Varga. Similarity and matching of neural network representations, 2021. URL https://arxiv:
brgg/bb//1ns 14633
Jacob Devlin, Ming-Wei Mei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. nesota, 2019. Association for Computational Linguistics. doi: 10. 10.18663//1/N119-1222 1423.. URL htttp:://aclanthee logy
Minneapolis, Min-
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OeenReveew.net, 2021. id=YicbF CNTTY
net/ forum?
William Falcon and The PyTorch Lightning team. PyTorch Lightning, 2019, URL https://
github. com wtt.e/ccmmiiiiiiii Lightning wwwggghtnnnnnnnnnnnnniieeeeeeeee AI/lightning
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with Py PyTorch Geometric. In ICLR Workshop Representation Learning Graphs and Manifolds, 2019.
GrokAL. nn-template bootstraps pytorch projects by advocating reproducibility & best practices in deep learning, 2021. URL https: /etttp:://gihutte com/ com/grok het/nn-template Software available from https://gitubb.com/rroaaal
Michael Gygli. Jasper Uijlings, and Vittorio Ferrari. Towards reusable network components by learning compatible representations. AAAI, 3509),760--7629, 2021.
Dongyoon Han, Sangdoo Yun, Byeongho Heo, and YoungJoon Yoo. Rethinking channel dimensions for efficient model design. ArXiv preprint, abs/2007.00992, 2020. URL https://arxiv:
org/ abs/ 2007. 00992
Thomas Hofmann, Bernhard Schölkopf. and Alexander J Smola. Kernel methods in machine learn- ing. The annals of statisticy, 36(3): 2008.
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin- Lin, and Deepak Ravichandran. To- ward simantics-based answer pinpointing. In Proceedings of the First International Conference on Human Language Technology Research, 2001. URL htt.s:://cllatthollce
org/
H01-1069
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. The multilingual Amazon reviews corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4563-4568, Online, 2020. Association for Computational Linguis- tics. doi: 10.18653/v1/20222 1/2000001102200..m1mmmmm mainmlppmminn6666 369. htLhttps:: / org/2020.
emm main. 369
Simon Korbllith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML. 2019, 9- 15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3519-3529, PMLR, 2019. URL http: http:ttppppppceeee htt.:::/ppooceeinggg.o m.r. wwt.ppessp.... kornblithl 9a..a...mm ntml
Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Tom Rainforth, and Yarin Gal. Self- attention between datapoints: Going beyond individual input-output pairs in deep learning, 2021. URL URI|https: /ttps::///aaiii org. www./baa//aaa.....o. abtp///s./22..... 02584

---

Published as a conference paper at ICLR 2023
Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. Representational similarity analysis connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2, 2008 ISSN 1662-5137. doi: 0..3389/meuro0066004..2000
Alex Krizhevsky, Learning multiple layers of features from tiny images. pp. 32-33, 2009. URL http:::/www.c.toroonto.
www.cc...o www.rto.ooooo.... edw/ moop//krizz /eaia//eeaningg-e features 2009- TR. pdE.
Ruslan Kuprieiev, skshetry, Dmitry Petrov, Peter Rowlands, Pawet Redzyúski, Casper da Costa- Luis, Alexander Schepanovski, Gao, David de la Iglesia Castro, Ivan Shcheklein, Batuhan Taskaya, Jorge Orpinel, Fábio Santos, Dave Berenbaum, daniele, Ronan Lamy, Aman Sharma, Zhanibek Kaimuldenov, Dani Hodovic, Nikita Kodenko, Andrew Grigorev, Earl, Nabanita Dash, George Vyshnya, maykulkarni, Max Hora, Vera, and Sanidhya Mangal. Dve: Data version control -git for data & models, 2023. URL battps:///ooot org/ 10. 528l:///.....1//eeeeeeee wtw.eeeeodoo......... 7646429
Aarre Laakso and G. Cottrell. Content and cluster analysis: Assessing representational similarity in neural systems, Philosophical Psychology, 13:47 76, 2000.
Guillaume Lample, Alexis Conneau, Mare' Aurelio Ranzato. Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceed- ings. OvenReview.net, 2018. URL http://////peeeeeee.... net. www.fooruiiiiriooe H1966ainbnnn
Y Lecun, L Bottou, Y Bengio, and P Haffner. Gradient-based learning applied document recog- nition. Proc. IEEE, 86(1 (1:211))272222) 2324, 1998.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivari- ance and equivalence. In IEEE Conference on Computer Vision and Patter Recognition, CVPR 2015, Boston, MA, USA. June 7-12, 2015, pp. 991-999. IEEE Computer Society, 2015. doi: 10. 1109/CVPR.2015..298877 URL https: /oii org 10. 10t...119//00 1109/CVPR. 2015. 7298701]
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemnite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Can- wen Xu, Nicolas Patry. Angelina McMillan-Major Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Gochringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community li- brary for natural language processing. In Proceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 18.18v53/v1/ 2021. eemllp-demoo11. URL lntpt:t///// http:///ccaaaatoooooooooo org/ 2021. emm.. demo 21
Yixuan Li, Jason Yosinski. Jeff Clune, Hod Lipson, and John E. Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL /ttt:://arxivv /atp:::///xxi..og//aab.. org.abs 1511 07543
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.1 1692, 2019. URL htttttttt///:aaii..ooo http://////aii..oo///////////// 11692
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. ArXiv preprint, abs/1301.377 13711, 2013a. URL http:tt///xxxii..ot
abs/ 0os//131.310
Tomás Mikolov, Quoe V. Le, and Ilya Sutskever. Exploiting similarities among languages for ma- chine translation. CoRR, abs//309.4411 1168, 2013b. URL http::/ arxix. wwwpivvoorg/aaa. wttp://.rrg/bb//30...
4168
12

---

Published as a conference paper at ICLR 2023
Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights representational similar- ity in neural networks with canonical correlation. In Samy Bengio. Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò vances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- fion Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp-
and Roman Garnett (eds.), Ad-
5732-5741, 2018. URL https::/ htwp://cceecinnn.n..r........pps 3/a3d70c6d1 1aa/11199199baace11.. Att.:/000..aaaliaiaaaaaaaaaeea.. pttpppppp/////////aaat////
Abstract................a.... html
Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolå, and Francesco Locatello. Asif: Coupled data turns unimodal models to multimodal without train- ing, 2022.
http://ww..or/aaaa/11...0.00..1. 011738
Christopher Olah. Visualizing representations: Deep learning and human beings. //colah.github. 2015. Accessed: 2022-8-2.
http:
Rajeev DS Raizada and Andrew C Connolly. What makes different people's representations alike: neural similarity space solves the problem of across-subject fmri decoding. Journal of cognitive neuroscience, 24(4):868 068-8887778888 2012.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. Wiki- Matrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In Proceed- ings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pp. 1351-1361, Online, 2021. Association for Computational Linguis- 115. URL httttpss///llanthcoooooooom org.2/0010y..//22211111
tics. doi: 10.18653/v1/2021.cca
eacl main. 115
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective Classification in Network Data. AlMag., 29(3):93, 2008. ISSN 2371-9621. doi: 10.1609/aimag.v2999 v29:3.2117.
Daniel Shalam and Simon Korman. The self-opiimll-ranspor feature transform. ArXiv preprint, abs/2200.03065, 2022. URL
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for fho- learning. In Is- abelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pp. 4077-4087, 2017. URL htttpp::/pproceeiiggss.... www.snng......ii...omom paper/ 200017/aash/cch httphhhhhccaa267775556 f28 225 abba .ttll
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, and Nathan Srebro. The implicit bias of gra- dient descent separable data. In 6th International Conference Leurning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. utLpttpss:/ / wtwpeaawwwweeeeetttooorraaaaaaaa
Anton Tsitsulin, Marina Munkhoeva, Davide Mottin, Panagiotis Karras, Alexander M. Bronstein, Ivan V. Oseledets, and Emmanuel Müller. The shape of data: Intrinsic distance for data distribu- tions. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL Forum?id=Hyeip HYWB.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. ArXiv preprint, abs/1607.08022, 2016. URL /nttps://arxiv.
erq/abs/ 607.0070 08022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 0008-6008,0000000
wttpp//pe/////////aaeeeeee URL /t neurss http://awwaaaaeo.oooooooee//////
3f
13

---

Published as a conference paper at ICLR 2023
Ivan Vulić, Sebastian Ruder, and Anders Søgaard. Are all good word vector spaces isomor- phic? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1178-3192,, Online, 2020. Association for Computational Linguis- URLhttps:: //acl /
tics. doi: @mm p-ma 257
org/2020.
Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John E. Hopcroft. Towards understanding learning representations: To what extent do different neu- ral networks learn the same representation.
Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. hcc/paperr2018/aaaee html
2018. URL htt.:///0.....1111......6611111. 5fc34ed307
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-arr natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, 2020. Associ- ation for Computational Linguistics. doi: 10. (0.186533v1/2220.emml http:///20200mmmmmmmmmm......... URL https:
//aclanthology.o org /wt////g. emnlp demos.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion a novel image dataset for benchmark- ing machine learning algorithms, 2017. URL httthttss///ariiv wwwp://ggga//bs//ooee..oess 07747
Muammer Y. Yaman, Sergei V. Kalinin, Kathryn N. Guye, David Ginger, and Maxim Ziatdinov. Learning and predicting photonic responses of plasmonic nanoparticle assemblies via dual varia- tional autoencoders, 2022. URL/attps://axxii org/ abs///bs/2222 63861
Zhilin Yang. William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016. New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 40 48. .Mr.. 2016. URL htt.:httpp//oceeeeeeeee
wtw.mm...rrr////emmmmmmmmmm https///////a/aalll htmml
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 28: Annual Confer- ence on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Que- bec, Canada, pp. 649 657, 2015. URL http:ttstppppocceeeeiiiieee neur lps. CC cttsspccppperrr// /aper/
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 HIGHI-DIMENSIONAL LATENT SPACES
In Figure| [1 multiple trainings of the same two-dimensional AE produce intrinsically similar latent spaces; in Figure(5] we show this property also holds on AEs with high-dimensional bottleneck. In the first row, PCA is fitted indipendently in each column, and since the PCA transformation produces the same output everywhere the latent spaces are intrinsically the same. In the second row, PCA fitted only the first latent space; since in this case the PCA transformation produces different outputs, the latent spaces, although intrinsically similar, are extrinsically different.
Train 0
Train
Train 2
Train
Train 4
3.4
02
13
08.35
3.0
0.33
-62
-0.4
-0.
-00 500
-05
09
Figure 5: Latent spaces learned by distinct trainings of the same digh--imensional AE on the MNIST dataset. Each column is the latent space obtained by the AE with a different seed. On the first row, the dimensionality reduction performed through PCAs fitted independently on each latent space, meanwhile, on the second row PCA is fitted on the leftmost latent space and then applied to all of them.
A.2 ANCHORS ANALYSIS
The cardinality of the anchors set A and the choice of specific anchors is crucial to the quality of the relative representations. At the extreme, selecting single anchor or the same repeated data points for all anchors, will produce collapsed relative representations. We believe that additional research required to obtain better understanding set composed only by stopwords worse than the ones composed by meaningful and diverse words?" require empirical evidence and could help revealing the semantics of the latent space. Indeed, each anchor is associated with a dimension in relative representation; one could inspect the anchor data point to get a sense of the meaning that latent dimension.
the optimal choice for A. Questions like "Are anchors
Anchor number. Below, we report a preliminary study on the performance sensitivity against the cardinality of the anchors set. In Figure 6 report the performance the node classification task on Cora, with a model trained end-to-end adopting the relative representations while train- ing, and on image classification tasks on CIFAR- 100, with frozen encoder. The performance improves monotonically as the number of anchors increase when the absolute representations are frozen (right). Differently, training models end-to-end proves to be more susceptible model col- lapse and instabilities, as increasing the number of anchors does not always improve the performance (left). Further research on the relation between the absolute latent space dimensionality and the rel- ative representation dimensionality (i.e., the number of anchors) is needed to clarify how the two quantities impact the performance, when training end-to-end or not.
Anchor selection. In Tables and we analyze different anchor selection strategies under an experimental setting analogous the described in Section 4.1
15

---

Published as a conference paper at ICLR 2023
0.8
Performance
0.6
0.4
0.2
0
200 400
600
200
400
600
800
1000
Number of anchors
Number of anchors
Figure 6: Accuracy vs Number of anchors. Each point a trained model. Left: Trained embedder on Cora, node classification. Right: Frozen transformer Cifar100 coarse-grained, image classi- fication. Left is less stable because the absolute embeddings are trained, and we are working on a domain that is less stable (graphs). Some collapsed examples not visualized.
uniform The first selection strategy is the same adopted in the main manuscript. We ran- domly select the anchors with uniform probability distribution over all the available sam- ples;
fps We select the anchors according to farthest point sampling strategy;
kmeans We select the anchors as the words more close to the centroids K-means clus- tering with K number of anchors;
top[k} We select the anchors the is most frequent words, after skipping the first 400 which are mostly stopwords.
We expect strategies that better cover the absolute space with anchors to be the most effective ones. Indeed, the results are comparable across selection strategies, but fps reaches everywhere the best Jaccard and MRR scores while k-means the best Cosine ones. We attribute this behavior to their different nature: they both rely on the geometry of the latent spaces they are applied to, k- means also favors high-density regions, and this can become negative bias for the task at hand. In general, the uniform sampling is the most straightforward to apply, since does not require additional computation for the selection process, and still achieves good performances.
A.3 INVARIANCE WITH GUARANTEED BOUNDS
In this section, we explore slightly modified version of the similarity function adopted in the main paper. The experimental setting is the same as in Section 4.1 We want to measure the similarity between pairs of absolute embeddings and their relative counterparts. To get some kind of quan- titative measure, two embedding types, averaged. Therefore, lower score indicates the spaces are more similar. On top of the standard relative representations, the ones computed with sim = Sc, here try to improve the similarity measure with guaranteed robustness to bounded distortion. In Figure / we report preliminary results that adopt this technique: vector-quantized similarity function produces relative representations which are more similar (they have lower score). The vector-quantizatioo the absolute embeddings various thresholds We
add similarity score calculated the pairwise cosine distance between the
done through agglomerative clustering leave to future works the study of the trade-off between guaranteed invariance to arbitrary bounded distorsion and the expressiveness of the resulting representations.
A.4 DATASET INFORMATION
In Table we summarize the datasets utilized in our work, and for each one, we specify the number of classes, give idea about the classification difficulty.
16

---

Published as a conference paper at ICLR 2023
Abs.
Rel.
Rel. Qnt. / =
Rel. Qut. t= 1.5
Rel. Qnt. / =
Word2vec FastText
Score 3.30
1.24
1.19
1.09
1.02
Figure 7: The Fast Text and Word2Vec embeddings of subset of the English dictionary. The score is the pairwise distance average between the two embedding types, thus a lower score indicates the spaces are more similar. The absolute representations appear very dissimilar meanwhile the relative representations yield almost identical spaces. Quantizing the absolute representations by performing agglomerative clustering with distance threshold produces even more similar spaces.
A.5 IMPLEMENTATION DETAILS
In this Section, following the corresponing sections in the main paper, we report implementation details for all the experimental settings considered.
Tools & Technologies In all the experiments presented in this work, the following tools were used:
NN-Template GrokAI
2021 to easily bootstrap the project and enforce best practices;
PyTorch Lightning (Falcon & The PyTorch Lightning team 2019), to ensure reproducible results while also getting clean and modular codebase;
Weights and Biases (Biewald. 2020). to log experiments and compare runs across huge sweeps;
Transformers by HuggingFace (Wolf ct al. 2020), to get ready-to-use transformers for both text and images;
Datasets by HuggingFace (Lhoest et al. 2021 access most of the NLP datasets and ImageNet for CV;
DVC (uurreeev et al. 2023). for data versioning;
PyTorch Geometric (Fey & Lenssen 2019), to handle graph datasets and get ready-to-use
GNN architectures.
A.5.1 WORD EMBEDDINGS
For both the Figure and the Table in Section 4.1 the number of anchors is set to 300 for a fair comparison with the dimensionality of the original spaces. For visualization purposes, needed the figure to both show an easy clusterable and restricted set of word embeddings. They are obtained by subsampling the shared vocabulary with the following procedure: we select 4 random pivot words, and for each of them we consider the top-200 words in their neighborhood. This results in total of 800 points divided in 4 clusters, the ones used only for the visualization part. For the quantitative part (table results), select 20K random words from the shared vocabulary with a fixed seed for reproducibility purposes.
For the computer vision counterpart (Figure II and 8)ble the procedure is similar but with the fol- lowing differences: i) the number of anchors set to 500 to balance between the different encoding dimensions of the two transformers (384 for ViT-small and 768 for ViT-base); ii) the subsampling for visualization purposes done by selecting 4 classes and randomly picking 200 samples for each of them;
17

---

Published as a conference paper at ICLR 2023
Evaluation metrics Consider the set of 20k samples S (words for the NLP test, images for the CV one) and the source space X and target space Y and any sample E S, we compute its representation in X and Y through the functions fx + X and fy & Y and define the metrics as follows:
KNNN (fx(()) A KNN (fx(s))
Jaccard
KKNN (fx())) U KNN ((x(s))
MRR(s)
Ranky (fx(s), (s(()) fx(8) fx(()
Cosine(s)
llfx(s)
where KNN (v) a function that returns the k-top similar samples (according to cosine similarity) to in the space A. and Ranka (v, u) is function that returns the index at which u is found in the ordered KNN, (v). The final score for each metric is the mean over each S.
A.5.2 RELATIVE REPRESENTATION SPACE CORRELATIONS
In this section, we analyze how similarities in absolute and relative spaces are correlated. Let us consider two spaces alignable in the relative space. We denote elements of the spaces with [m and B E " and corresponding relative embeddings with C R xd D E Rm Xd Examples of and B can be the FastText and Word2Vec word embedding spaces. We already observed in Table T how the spaces A and B are well aligned in the relative space. We go further and analyze how self similarities each space are preserved by the relative transform. In only facilitate latent space communication, but
Figure we show that relative representations also preserve the underlying (absolute) latent space metric up to a certain degree.
FastText
Word2Vec
0.648
CC
B
0.791
0,003
O 0.565
0.775
A&A
B
0.674
Figure 8: Self similiarities correlations between each space, measured with the Pearson correlation coefficient. blue, we denote the self similarities the absolute spaces A, of FastText and Word2Vec; in green we depict the relative spaces C, DD. The correlation in the vertical arrows indicate how much the underlying metric in the abolute space is preserved by the relative coordinate transformation.
A.5.3 LATENT DISTANCE AS PERFORMANCE PROXY
The hyperperameters used in Section 4.2
summarized in Table 10
A.5.4 TRAINING WITH ABSOLUTE vs. RELATIVE REPRESENTATIONS
The models trained on relative representations do not backpropagate through the anchors, which encourages smoother optimization of the anchors' representations.
Image Classification The architecture is standard deep CNN. We sweep for each dataset where we vary only the random seed (over 10 possible in total). We then aggregate by dataset and encoding type obtain the final results with their standard deviation.
18

---

Published as a conference paper at ICLR 2023
Graph Classification We run sweep identical to the in Table 10 for the reference model, except that we sweep on the "Number of layers" with two values: 32 and 64. Each configuration is repeated with 10 different seeds, then aggregate by dataset and encoding type to obtain the final results with their standard deviation.
A.5.5 IMAGE RECONSTRUCTION
The relative and absolute models appearing in Figure 4 are vanilla AEs and VAEs, the same for all the datasets, and have comparable number of trainable parameters. Their architecture is composed by simple convolutions, deconvolutions and mean squared error as reconstruction loss. The number of anchors is 500 and the latent dimensionality of the absolute representations is 500.
A.5.6 TEXT CLASSIFICATION
We report in Tables
11 to 13 details the transformers and anchors adopted in Section 5.2
Preprocessing Following the original work in which the Amazon Reviews dataset was pro- posed (Keung al. 2020), utilize both the title and body of each review. We differ in not using the category and in how we merge them; namely, we add the title as prefix for the body and add full stop as separator when needed (avoiding duplicates). To obtain single latent encoding for each sample, with fixed shape, we take the last hidden state and select the representation corresponding the [CLS] token.
Wikipedia anchors We use WikiMatrix, corpus of sentences extracted from Wikipedia. The sentences are parallel between pairs of languages (i.e., same sentences translated in two languages), decided
and since to use the English language as a pivot to compute the intersection. To get the final results, we considered only the sentences with margin score 1.06, getting high-quality sentence alignments. In Table 13 we show the total number of parallel sentences when computing the intersections. We randomly selected 768 samples to use anchors.
looking for collection of parallel anchors between all 4 languages,
A.5.7 IMAGE CLASSIFICATION
The details of the transformers used in Section 5.3 are summarized in Table 14
A.6
ADDITIONAL RESULTS
In this section we report additional results on the correlation between latent similarity and perfor- mance in Figure g results on the multilingual stitching both with Amazon coarse-grained in Ta- ble 15 and fine-grained in Table 16] results on the image classification stitching on CIFAR- 100 fine-gratd ameeggrineed in Table I8 Moreover, we evaluate the stitching performance of multilingual trans- former in Table 17
19

---

Published as a conference paper at ICLR 2023
Figure 9: Correlation plot between performance and latent similarity with the reference model for multiple different models, time.
Fast
Word2Vec
Text
Word2Vec
Absolute
Relative
Figure 10: Same encodings in Table I (left) but with tSNE (left) dimensionality reduction or visualizing only their first two dimensions (right).
20

---

Published as a conference paper at ICLR 2023
Table 7: Extended results from Section 4.1 with different anchor selection strategies. The table
reports the mean score for each metric and std across 10 different seeds.
Mode
Type
Source
Target
Jaceard
MRR
Cosine
Absolute
FastText
FastText
1.00 + 0.00
1.00 1 0.00
1.00 + 0.00
Word2Vec
0.00 = 0.00
0.00 0.00
0.01 = 0.00
uniform
Word2vec
Fastllext
0.00 0.00
0.00++0.00
0.01 + 0.00
Word2Vec
1.00 0.00
1.00
1.00 = 0.00
FastText
1.00 1000:000 0.00
1.00
1.00 T1 0.00
Relative
FastText
Word2Vec
0.34 1.3410001
0.00
0.86 0.00
Fastllext
0.39 1:9+00000
0.98
0.86 + 0.00
Word2Vec
Word2vec
1.00 0.00
1.00
1.00 0.00
Absolute
FastText
FastText
1.00 10000000
1.00
1,00 = 0.00
Word2Vec
0.00 0.00
0.00
0.01 0.00
FastText
0.00
0.00
0.01 + 0.00
Word2Vec
Ips
Word2Vec
1.00
0.00
1.00 0.00
FastText
0.00 1.000
1.00 + 0.00
Relative
FastText
Word2Vec
0.94 0.01
0.81 0.00
FastTeex
0.41
0.00
0.83 0.00
Word2Vec
Word2Vec
1.00
1.00
1.00 0.00
Absolute
FastText
FastText
1.00
1.00
1.00 + 0.00
Word2Vec
0.00
0.00
0.01 0.00
kmeans
Mord2Vec
Fastllext
0.00 0.00
0.01 0.00
Word2vec
1.00
1.00
1.00 + 0.00
FastTTex
1.00
0.00
1.00 0.00
Relative
FastText
Word2Vec
0.35
0.00
0.87 = 0.00
FastText
0.00 0.00
0.87 0.00
Word2Vec
Word2vec
1.00
1.00 + 0.00
Absolute
FastText
FastText
1.00
1.00
1.00 0.00
Word2vec
0.00
0.00
0.01 = 0.00
top1000
Word2vec
FastText
0.00
0.000
0.01 0.00
Word2Vec
1.00
1.00
1.00 1 0.00
FastText
1.00
1.00
1.00 0.00
Relative
FastText
Word2Vec
0.27 0.01
0.04
0.85 1 0.00
FastText
0.35 0.01
0.97
0.85 + 0.00
Word2Vec
Word2vec
1.00 1.0+000 0.00
1.00+0.00
1.00 + 0.00
Absolute
FastText
FastTfext
1.00 +.00
1.00
1.00 + 0.00
Word2Vec
0.00 0.00
0.00
0.01 + 0.00
top5000
Word2Vec
FastText
0.00
0.00
0.01 + 0.00
Word2Vec
1.00
1.00
1.00 + 0.00
FastTeex
1.00
1.00
1.00 + 0.00
Relative
FastText
Word2Vec
0.32
0.00
0.86 + 0.00
FastText
0.38 0.00
0.97 0.000
0.86 1 0.00
Word2vec
Word2Vec
1.00
1.00
1.00 + 0.00
Absolute
FastText
FastText
1.00 1.00
1.00 1.00
1.00 1 0.00
Word2Vec
0.00 0.00
0,00 1.000
0.01 + 0,00
top10000
FastText
0.00 +.000
0.00
0.01 + 0.00
Mord2Vec
Word2Vec
1.00
1.00
1.00 + 0.00
Relative
FastText
Mastilext
1.00
1.00
1.00 + 0.00
Word2Vec
0.000
0.86 1 0.00
FastText
0.39 +.99
0.97
0.86 + 0.00
Word2Vec
Word2Vec
1.00 1.00+0.000
1.00
1.00
21

---

Published as a conference paper at ICLR 2023
Table 8: Generalization of the results from Section 4.1 on word embeddings to a different data modality, with different anchor selection strategies (See Appendix A.2 for their description). The dataset considered is CIFAR-10, and the table reports the mean score for each metric and its std 10 different seeds.
Mode
Type
Source
Target
Jaccard
MRR
Cosine t
Absolute
ViT-base
ViT-bass
1.00 + 0.00
1.00 + 0.00
1.00 + 0.00
ViT-small
uniform
ViT-small
Vir-base
ViT--sall
1.00 0.00
1.00 0.00
1.00 0.00
Relative
ViT-base
ViT-base
1.00 = 0.00
1.00 0.00
1.00 0.00
ViT-smal1
0.11 + 0.00
0.27 0.01
0.97 1 0.00
ViT-base
0,11 = 0.00
0.30
0.97 0.00
Vit'smal1
ViT-small
1.00 0.00
0.00
1.00 +
Absolute
ViT-base
ViT-base
1.00 = 0.00
1.00 0.000
1.00 +
ViT-small
ViT-base
Vit-smal1
fps
ViT-small
1.00 0.00
1.00 0.00
1.00 0.00
Relative
ViT-base
ViT-base
1.00 0.00
1.00
1.00 0.00
ViT-small
0.12 0.00
0.37
0.96 0.00
ViT-base
0.12 0.00
0.39
0.96 0.00
Vit-small
ViT-sma11
1.00 0.00
0.00
1.00 + 0.00
Absolute
ViT-base
ViT-base
1.00 = 0.00
1.00
1.00 0.00
ViT-small
kmeans
ViT-small
ViT-base
ViT-small
1.00 0.00
1.00 0.00
1.00 0.00
ViT-base
1.00 =.00
1.00 0.00
Relative
ViT-base
ViT-small
0.11 = 0.00
0.97 0.05
ViT-base
0.10 0.00
0.97 0.07
Vit-smal1
ViT-small
1.00 = 0.00
1.00
1.00 0.00
Table 9: All the datasets utilized in work with their number of classes.
Dataset
Number of Classes
MNIST
10
Image
Fashion MNIST
10
CIFAR-10
10
CIFAR-100
20 (coarse) - 100 (fine)
ImageNet1k
1000
Graph
Cora
7
CiteSeer
6
PubMed
3
Text
TRBC
6 (coarse) 50 (fine)
DBpedia
14
Amazon Reviews
2 (coarse)
5 (fine)
22

---

Published as a conference paper at ICLR 2023
Table 10: The reference model and exhaustive hyperparameter combinations pertaining Section 4.2
Hyperparameter
Reference Model
Sweep
Seed
1, 2, 3,
Epochs
500
30, 50
Number of layers
32
32, 64
Dropout Probability
0.5
0.1, 0.5
Hidden Activations
ReLU
ReLU, Tanh ReLU, Tanh
Convolution Activation
ReLU
Optimizer
Adam
Adam, SGD
Learning Rate
0.02
0.01, 0. 0.02
Graph Embedder
GCNConv
GCNConv, GINCOnv
Table 11: The HuggingFace transformers employed in Section 5.2 to tackle the Cross-lingual setting.
Language
HuggingFace transformers name
Encoding Dim
English
roberta-base
768
Spanish
768
French
ClassCatrroberta-aass--ff frrenc
768
Japanese
nlp-wasseda/robertta bawp:etttaabas--paaaeeeeee japanese
768
Table 12: The HuggingFace transformers employed in Section 5.2 setting.
to tackle the Cross-architecture
HuggingFace transformers name
Encoding Dim
bert-base-cased
768
hert-base-uncased google/electra-base discriiicrm roberta-base
768
768
768
Table 13: WikiMatrix analysis. Each row shows the number of parallel sentences having a transla- tion available in all the languages of that row. Since we consider all four languages, we have 3338 parallel sentences available.
Languages
Number of Sentences
en, es
2302527
en, ja
264259
en, fr
1682477
en, fr 23200
en, es, ja 147665
en, fr,
20990
en, es, fr, ja
3338
Table 14: Timm transformers used in Section 5.3
Version
Timm model name
Encoding Dim
Training data
VIT
base waceeeaaaahl...1 16_244 6_224
768
JFT-300M, ImageNet
VIT
sitssmlll_pachh 224
384
ImageNet
VIT
384
768
ImageNet
RexNet
rexnet_10
1280
ImageNet
23

---

Published as a conference paper at ICLR 2023
Table 15: Stitching performance comparison with different encodings techniques. The table reports the mean weighted Fl (# std) and MAE classification performance on Amazon Revi ews coarse- grained, across different seeds, All the language pairs are shown.
Absolute
Relative
Translated
Wikipodin
Decoder
Encoder
FScore
MAE
FScore
MAE
FSeore
MAE
91.54
0.58
0.08
0.01
90.06 + 0.60
0.10 0.01
90.45
0.52
0.10 + 0.01
13.67 = 1.09
0.56 + 0.01
82.78 + 0.81
0.17 = 0.01
78.53 0.30
0.21 + 0.00
ja
48.72 + 0.90
0.51 + 0.01
65.72 1: 0.55
0.34 1 0.01
66.31
1: 0.80
0.34 1 0.01
en
33.23 + 1.00
0.66 0.01
78.68 +: 2.74
0.21 + 0.03
76.65
3.23
0.23 + 0.03
91.64 1: 1.02
0.08
0.01
89.96 1. 0.77
0.10 1: 0.01
89.62
1: 0.94
0.10 1 0.01
47.66 + 0:70
0.52 + 0.01
78.57 + 1.80
0.21 + 0.02
0.76 75.25
0.25 + 0.01
53. 2.27
0.02
67.69 0.24
0.32 0.00
61.84 0.61
0.38 0.01
51.00 1: 2.63
0.49 1. 0.03
83.32 1: 1.80
0.17 0.02
75.55
0.37
0.24 1 0.00
fr
+ 2.81
0.03
82.50
0.17 0.01
0.23
88. 22 = 0.75
0.12 0.01
85.68 + 1.37
0.11
86.45 : 0.96
0,13 0.01
ja
50.32 4.16
0.50 0.04
69.38 + 0.73
0,31 + 0.01
0.27 62.79
0.37 + 0.00
53.82 2.62
0.408
08.66
0.31 0.04
20.26+3.100
0.29 0.03
11.91 + 2.21
0.55 + 0.02
70.37
0.29 0.06
58.51 1.21
0.41 0.01
0.36
83.30 ja
1 0.67
0.17 1 0.01
81.04 1: 0.82
0.19 1: 0.01
80.80 1: 1.25
0.19 1 0.01
Stitching performance comparison with different encodings techniques. The table reports the mean weighted F1 (# std) and MAE classification performance on Amazon Rev ews fine- grained, across different seeds. All the language pairs are shown.
Table 16:
Absolute
Relative
Translated
Wikipodia
Decoder
Encoder
FScore
MAE
FScore
MAE
EScore
MAE
65.46 +: 2.89
0.38
0.02
61.18 + 1.92
0.44 + 0.02
62.36
2.23
0.43 + 0.02
22. 70 + 0.41
1.39 + 0.03
51.67
+ 1.20
0.62 = 0.01
45.40
1::00:00 0.68
0.76 + 0.01
0.90
24.85
0.91
1.37
+ 0.07
37.34
1.49
0.99 + 0.02
37.73
+ 0.70
1.01 + 0.02
21.24 0.81
1.43
0.07
51.02 + 2.54
0.68 + 0.05
47.70
5.08
0.73 + 0.10
29.29 1 3.04
0.43 + 0.02
57.89
+ 3,80
0.48 + 0.03
57.96
4.40
0.48 + 0.03
29.02
1 0.85
1.26 0.05
48.40
1 1.02
0.71 1 0.02
44.92
1.83
0.77 1 0.01
ja
1.56
0.01
1,08
27. 39
4: 1.22
1.23 + 0.06
45.55 1: 3.55
0.76 + 0.09
39.01 1: 1.25
0.88 1 0.06
1
fr
56.40 40
+ 1.89
0.51 0.01
53.58
+ 0.70
0.57 + 0.01
54.23
0.95
0.56 + 0.01
ja
25.92 1 1.31
1.25 1 0.05
38.60 1: 1.03
0.96 1 0.02
35.22 1: 0.56
1.08 1 0.02
29.36
0.59
1.17 0.04
38.19 2.28
0.88 0.03
36.57 1.72
0.98 1 0.02
25. + 1.77
1.28 4 0.04
34.23
2.62
1.00 + 0.05
33.16 +:166:000 2.28
1.06 + 0.03
36.68
1.00
54.09 + 1.35
0.60 + 0.02
50.89 + 1.70
0.65 + 0.02
51.64 + 1.47
0.65 + 0.02
24

---

Published as a conference paper at ICLR 2023
Table 17: Stitching performance comparison on XLM-R, multilingual model by design. The table reports the mean weighted fine-grained, across 5 different seeds.
Fl (# std) and MAE classification performance on Amazon Reviews
Absolute
Relative
Decoder
Encoder
FScore
MAE
FScore
MAE
en
65.27 0.94
0.41
0.01
58.21 +: 1.92
0.51 + 0.03
59.55 + 0.76
0.48 + 0.01
52.81 + 1.57
0.62 # 0.02
en
fr
58.58 1.04
0.49
54.01 + 1.34
0.59 = 0.02
ja
57.98 0.77
0.00
48.47 2.67
0.71 0.04
60.32 1.50
0.01
45.69 2.19
0,87 = 0.07
es
61.25 1.74
0.01
57.61 0.73
0.51 0.01
59.50 1.41
0.01
45.16 3.30
0.83 0.09
ja
58.24 + 1.31
0.51 0.51
41.14 1.76
0.99 = 0.05
en
58.00 - 4.21
0.49
52.37 + 1.66
0.66 0.03
es
56.87 + 3.79
0.03
54.99 + 0.46
0.57 = 0.01
fr
57.99 +: 3.88
0.02
57.00 0.90
0.52 = 0.01
ja
55.83 3.32
0.53 0.03
39.15 1.21
1.02 = 0.03
em
59.53 + 1.73
0.48 +
39.46 + 2.34
1.04 = 0.07
57.02 1.36
4
0.97 + 0.09
ja
57.48 1.06
0.01
3.70
0.89
ja
61.43 0.97
0.45
57.67 1.17
0.51 0.01
Table Stitching performance comparison with different encodings techniques. The table reports the mean weighted F1 (+ std) classification performance on CIFAR-100 fineggrained, across different seeds.
Decoder
Encoder
Absolute
Relative
rexnet-100
72.77 + 0.19
71.39 + 0.18
httbbasc-patch16-222
40.68 0.50
rexnet-100
vil-base-resne155-3844
38.18 0.24
vil-small-patchhl1-2244
44.11 0.84
rexnet-100
57.81 0.39
bit-base-patchh1-222
88.69 + 0.14
87.05 0.34
vit- vit-base-patee 16-224
vit- ti-base-rresnet0--
1.08 + 0.19
66.65 + 1.79
vit- stt-small-pathh6-222
73.73 0.60
rexnet-100 100
66.91 + 0.79
vit-base-patchl6-22
1.10 = 0.09
75.70 : 0.68
vit-base-resnee50--844
85.85 - 0.18
85.04 0.38
nit-small-patch 16-224
75.52 0.36
rexnet-
56.60 0.39
vit- baasc-pacch16 6-224
70.14 0.46
vit-small-patchl6-22 16-24
vit- hit-base-resnet50-15
62.85 + 1.22
vil-small-patch
16-222 16-224
81.11 + 0.14
83.24 1: 0.13
25

---

Published as a conference paper at ICLR 2023
ViT-base ViT-small 11
ViT-base ViT-small 11
ViT-base
ViT-small
Absolute
Relative
Figure 11: Different dimensionality reduction techniques applied to absolute and relative spaces on CIFAR-10. From left right: PCA (Principal Component Analysis), (SNE, and visualizing only their first two dimensions. Only 800 randomly sampled points are shown, belonging to the classes "bird". "ship", "cat", and "frog".
26


Reviews:
[{"review_id": "5xuWMgh_bt", "reviewer": "Reviewer_LNFQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the opinion that the relative representation, the representation of data described by a fixed set of anchor representations, is invariant among different randomized training factors like seeds, optimization strategies, training steps, and even architectures. To prove their opinion, the authors then prevent extensive studies across a broad range of domains in modern machine learning. The results are impressive and very interesting. The authors show that 1) model trained on datasets sampled from similar domains (like English language) with different distributions may share the same relative representations; 2) the similarity of the relative representations of networks has a positive correlation with their performance; 3) training with relative representations shows comparable performance against the absolute representations; 4) relative representations can be shared by networks of different architectures, training distributions, and seeds, to achieve a valid effect.", "summary_of_the_review": "Very impressive results in the general area of machine learning, combined with very extensive and strong empirical arguments. Deeper analysis of the sources and reasons of this phenomenon will be appreciated.", "strength_and_weaknesses": "Strength:\n\n1. The empirical findings in this paper are very impressive and interesting.\n\n2. The paper is extremely well-written and easy to follow.\n\n3. The experiments involve broad domains of machine learning and are of interest to a general audience.\n\n4. The experiment arguments involve rich aspects of analysis and thus are convincing and impressive.\n\n\nWeakness:\n\n1. The experiments in sec. 4.1 are somewhat weak compared with those in the remained sections; only one case is considered. Adding an extra case of computer vision task may further enhance it.\n\n2. The existence of this phenomenon is well proven, but the source and deep reason for it to happen are not presented as strongly as the former.\n\n3. In Sec.4.1 and Tab. 1, it will be better if the authors can add references and more detailed explanations for the two metrics Jaccard and Mrr. It is a bit hard for me to understand how to calculate them and their roles directly from the current context.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3ObZH3YcnbZ", "reviewer": "Reviewer_LWWZ", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces the concept and the use of relative representations for machine learning models, which consists in representing data samples in terms of their embedding distance to a fixed set of \"anchor data samples\". The motivation behind proposing this representation is that, under some regularity and symmetry assumptions, it would allow for representing data in a way that is invariant across training run, initializations and even architectures. In fact, the paper demonstrates in simulations on various datasets and tasks that this hypothesis is empirically borne out. Interestingly, data similarity in this representation is observed to be highly correlated with model performance. A tantalizing application of the invariance afforded by relative representations is that they allow for zero-shot model-stitching, i.e. \"transferring\" representations across models, possibly trained on different datasets or with differing architectures. The paper for instance demonstrates combining together encoder and decoder models that were not originally trained together, which allow for instance for things like cross-lingual stitching: combining encoders and decoders trained on different languages.", "summary_of_the_review": "The paper introduces the concept of relative representations, a simple idea with straightforward realization but far reaching conceptual and practical consequences. The applications that this conceptual innovation affords are wide reaching and include potential theoretical advances in the study of generalization and invariance in representation learning, and a host of practical applications such as zero-shot stitching of models.", "strength_and_weaknesses": "Strengths:\n- Simple and at the same time very innovative idea. Its simplicity makes it extremely relevant and applicable across settings, domains, architectures and training paradigms, suggesting great potential for impact, both conceptual as well as practical.\n- Thorough examination and empirical demonstration of some of the potential of this new technique.\n \nThe paper does not have any apparent major weaknesses of relevance.\nThere are however technical details that could benefit from some clarification. For instance, the paper considers the training modality of training models with relative representation, but while doing that it is not readily clear whether this happens by also backpropagating through the anchors. It would be beneficial to clarify that, and whether and if there are any differences between doing that (backpropagating also through the anchors) vs only backpropagating through the data.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-OH8leuHFVY", "reviewer": "Reviewer_hU9r", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper highlights the difficulty in using latent representations which were obtained by training on different data, with different architectures, or, indeed, simply as a consequence of stochasticity in the training process itself. They introduce the notion of a \"relative embedding\", which is obtained by projecting the latent representations onto a (fixed, shared) set of representations of \"anchor entities\". The authors provide empirical evidence for the validity of their approach in several settings:\n1. Aligning latent representations allows for greater preservation of similarity in a shallow word embedding setting.\n2. Using relative embeddings during training does not significantly decrease performance on a collection of image and graph node classification tasks.\n3. Relative embeddings allow for zero-shot model stitching, wherein models with different architectures and potentially trained on different tasks can be combined when using these relative embeddings.\n", "summary_of_the_review": "This paper presents a simple idea that seems to work very well. The practical limitation of the proposed method is that it only works if the networks were trained with these relative representations in the first place. The strongest direct outcome of this paper would be that people would start using relative representations by default, thus facilitating the \"stitching\" mentioned in section 5, however I think this would require much greater exploration (eg. heuristic methods to choose the anchor elements with theoretical bounds on the potential impact) before being widely adopted.", "strength_and_weaknesses": "### Strengths\n1. The fundamental idea is pleasingly simple\n2. The authors included experiments to validate their motivation and intuition\n3. A comprehensive suite of empirical results on the ultimate task (stitching), including multiple datasets, domains, and variations (eg. differences in architecture vs. data) were included\n\n### Weaknesses\n1. For the main tasks (stitching) presented in section 5 I do not think the authors made it sufficiently clear which models were *trained* using relative embeddings, and which models used the relative embeddings purely as a post-hoc adjustment step. The emphasis on zero-shot (in the title, section header, and throughout), and the fact that earlier experiments (eg. section 4.1) were performed by creating the relative embeddings post-hoc may leave the reader with the impression that the relative embeddings can allow existing pre-trained networks to be stitched together. To my understanding, this is fundamentally impossible. At a minimum, the parts of the model downstream of the \"stiching\" must always be trained using relative embeddings, because the function which takes an absolute embedding to a relative one is not (in general) invertible. Of course, this also opens two opportunities for future work: (a) For instances where the mapping from absolute to relative embeddings is not strictly invertible, perhaps it is still essentially invertible on the data manifold. For example, a large component which contributes to the lack of invertibility is the vector normalization operation, however it has previously been argued that regularization results in embeddings which are essentially on a sphere. In such a setting, we could \"learn\" the inverse via gradient descent, and attempt to use this to stitch together networks without fine-tuning either part. (b) Can we choose the anchor nodes to improve the extent to which the transformation is essentially invertible?\n\n2. I have a few issues with Section 4.1 - Word Embeddings. First, using 300 randomly drawn parallel anchors seems very weak - if the words were drawn uniformly randomly then they are very likely all rare words, and as such would seem to serve as a very poor choice for anchors. This also may explain why the Jaccard similarity did not increase as substantially as might have been expected. I also take issue with this sentence: \"The average Jaccard distance reported in Table 1 (right), says that the neighborhoods of the relative representations are matched exactly 34% of the time in one direction, and 39% of the time in the other one.\" At least to my understanding, that is not what the average of Jaccard similarity would show. In fact, one could obtain those metrics with neighborhoods which *never* exactly match. Perhaps what was meant is that the *words* in the neighborhoods matched exactly, as the authors then go on to mention the discrepancies are likely due to semantic differences. I agree, and as such I wonder why such a course-grained evaluation was used in the first place? The setting we are in is as follows: given some embeddings A and B, find some new embeddings C and D such that the all-pairs similarities between elements of A are proportional to those between elements of C (and similarly with B and D), however the similarities between equal elements of C and D are minimized. One could simply measure and report on that metric.\n\n3. No theoretical analysis is included. It is, of course, increasingly common for papers to rely fundamentally on empirical results, however it seems possible to prove results relating the number of anchor entities, variation in the training data, and potential accuracy on the task. Even making minor preliminary theoretical statements on these aspects would strengthen the paper.\n\n4. I was not able to find details on how the 2-dimensional representation in on the left half of Table 1 was created. Presumably, some dimensionality reduction technique (eg. tSNE) is used, however that raises into question the usefulness of drawing conclusions from such a picture as the dimensionality reduction introduces a number of factors which may impact the resulting representation. (One could imagine various dimensionality reduction techniques with noise or hyperparameters which, for the same exact embedding, result in different representations.) At the very least, the authors should explain how these pictures were derived.\n\n\n### Typos / Suggestions\n1. Abstract: I would recommend including something about the anchor entities in the abstract, as (to my mind) that is essential to understanding the high-level idea. Perhaps this sentence: \"In this work, we propose to adopt pairwise similarities as an alternative data representation, that can be used to enforce the desired invariance without any additional training.\" could be changed to: \"In this work, we propose to use the similarity between each representation and a fixed set of anchor representations, and demonstrate that this can enforce the desired invariances without any additional training.\"\n2. Page 1: \"The underlying assumption is that the learned latent spaces should be the best encoding given the data distribution, the downstream task, and the network constraints. In practice, however, the learned latent spaces are subject to changes even when the above assumptions remain fixed.\" The writing here presents this as though this is contradictory, but of course it is not - the latent representation which is best for the downstream task given network constraints is not unique.\n3. Page 2: \"more in general\" -> \"more generally\"\n4. Page 7, Figure 4: I don't think this figure highlights the results as well as it could. Consider making the figure 5 rows tall, and group based on absolute vs. relative first (i.e. rows would be original, abs ae, abs vae, rel ae, rel vae). This would make it very clear that absolute struggles, while relative works well. (I would make similar suggestions for Table 3 and 5)\n5. Page 7: \"prove that representations are invariant to training stochasticity\"is a bit too strong, as I believe any mathematically rigorous interpretation of this statement would be provably false. Something like \"support our claim that relative representations are more robust to training stochasticity\" seems more accurate, given the results presented.\n6. Page 8: \"obtaine\" -> \"obtaining\"\n7. Page 9: \"allow to stitch modules\" -> \"allow stitching modules\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%