From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication
Authors: Irene Cannistraci
Luca Moschella
Marco Fumero
Valentino Maiorca
Emanuele Rodolà
Venue: ICLR 2024 spotlight (2024)
Keywords: invariance
latent space
latent comunication
zero-shot stitching
representation learning
relative representation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Incorporate invariances into neural representations to construct a product space of invariant components and unlock applications, such as merging, stitching, and reusing different neural modules.

Abstract: It has been observed that representations learned by distinct neural networks conceal structural similarities when the models are trained under similar inductive biases. From a geometric perspective, identifying the classes of transformations and the related invariances that connect these representations is fundamental to unlocking applications, such as merging, stitching, and reusing different neural modules. However, estimating task-specific transformations a priori can be challenging and expensive due to several factors (e.g., weights initialization, training hyperparameters, or data modality). To this end, we introduce a versatile method to directly incorporate a set of invariances into the representations, constructing a product space of invariant components on top of the latent representations without requiring prior knowledge about the optimal invariance to infuse. We validate our solution on classification and reconstruction tasks, observing consistent latent similarity and downstream performance improvements in a zero-shot stitching setting. The experimental analysis comprises three modalities (vision, text, and graphs), twelve pretrained foundational models, nine benchmarks, and several architectures trained from scratch.

Published as a conference paper at ICLR 2024
FROM BRICKS TO BRIDGES: PRODUCT OF INVARIANCES TO ENHANCE LATENT SPACE COMMUNICATION
Irene Cannistraci!
Luca Moschellas1
Marco Fumero
Valentino Maiorcal
Emanuele Rodolà
Sapienza University of Rome, 'Equal Contribution
ABSTRACT
It has been observed that representations learned by distinct neural networks con- ceal structural similarities when the models are trained under similar inductive biases. From geometric perspective, identifying the classes of transformations and the related invariances that connect these representations is fundamental to unlocking applications, such as merging, stitching, and reusing different neural modules. However, estimating task-spe transformations a priori can be chal- lenging and expensive due to several factors (e.g., weights initialization, training hyperparameters, data modality). To this end, introduce versatile method to directly incorporate a set invariances into the representations, constructing a product space of invariant components top of the latent representations without requiring prior knowledge about the optimal invariance to infuse. We validate our solution on classification and reconstruction tasks, observing consistent latent similarity and downstream performance improvements in a zero-shot stitching setting. The experimental analysis comprises three modalities (vision, text, and graphs), twelve pretrained foundational models, nine benchmarks, and several architectures trained from scratch.
INTRODUCTION
Discovering symmetries and conserved quantities core step for extracting meaningful repre- sentations from raw data in biological and artificial systems (Higgins et al.) 2022 Benton et al.
2020: Lyle et al.] 2020; Marchetti et al., (2023). Achieving invariance to specific groups of transformations within neural models holds sig- nificant utility in wide range of real-world applications, such comparing similar latent spaces across multiple training instances, facili- tating communication, and enabling model reuse
0.90
Invariance
Conformal
E(n)
(n)
Space Similarlly
(Cohen & Welling, 2016
Fawzi et al.
2016
Salamon & Bello 2017 Klabunde
al.
2023
0.75
Lähner & Moeller 2023 Maiorca et al.
2023).
These desired invariances be defined with re-
spect to transformations in the input space (Ben-
et al. 2020 [mmer al.] [2022] Cohen & Welling 2016) Cohen et al. 2019). in relation to the latent space, as explored by Moschella et al. (2023). Such properties can arise implicitly from architectural choices (Cohen & Welling al.) 2019 Worrall et al. 2017; Zhou et al. 2017) or be explicitly enforced us- ing methods like loss penalties (Arjovsky et al.)
RexNlet
VIT
CVT B/32
Figure 1: CKA similarity between pretrained mod- els on F-MNIST measured infusing invariances specific classes of transformations (Conforma? Euclidean, Orthogonal). Each bar reports the dis- tribution of similarity the other models. The score diversity highlights the absence of a univer- sal transformation connecting all latent spaces.
2016; Cohen Gohen
2019). A recent study introduced the concept of Relative Representation (RR) (Moschella al.
2023). In its original formulation, this framework enforces invariance to anglc-preserving transformations of the latent space. This approach enhances
Global orthogonal transformations composed with local rescalings

---

Published as a conference paper at ICLR 2024
communication between latent spaces by projecting them into shared relative space determined by distances between data points. However, as shown in Figure [ the transformations relating different neural representations are not always consistent with a single class of transformations, such as the one considered in Moschella et al. (2023). Determining a priori which class of transformations relates distinct latent spaces challenging due complex interactions in the data, and multiple nuisance factors that are typically irrelevant but can nevertheless affect the representation (e.g. random ini- tialization, neural architecture, and data modality). To address this challenge, we expand upon the method of RR, presenting a framework to efficiently incorporate set of invariances into the learned latent space. This is achieved by constructing product space of invariant components on top of the latent representations of, possibly pretrained, neural models. Each component of this product space is obtained by projecting samples as a function of fixed data points, denoted as anchors. Using different similarity functions for each subspace, we can infuse invariances to specific transformations into each component of the product space. Our main contributions can be summarized as follows:
We show that the class of transformation that relates representations learned by distinct Neural Networks (NNs)-trained on semantically similar day vary and depends on multiple factors;
We introduce a framework for infusing multiple invariances into a single latent representation, constructing product space of invariant components to enhance latent communication;
We validate our findings stitching tasks across various data modalities, including images, text, and graphs: product of invariances can capture complex transformations of the latent space in single representation, achieving the best performance without any prior knowledge of the transformation or the factors that may affect it.
2 RELATED WORK
Representation Similarity. Several metrics have been proposed to compare latent spaces generated by independent NNs, capturing their inherent similarity up to transformations that correlate the spaces. A classical statistical method is Canonical Correlation Analysis (CCA) (Hotelling, 1992), which is invariant to linear transformations; variations of CCA seek to improve robustness through techniques like Singular Value Decomposition (SVD) and Singular Value CCA (SVCCA) (Raghu et al. 2017) or to reduce sensitivity to perturbations using methods such as Projection Weighted CCA (PWCCA)
MMoroos et al., 2018). Closely related to these metrics, the Centered Kernel Aligmment (CKA)
Kornblith et al. 2019) measures the similarity between latent spaces while disregarding orthogonal transformations. However, recent research (Davari et al., 2022) demonstrates its sensitivity to shifts in the latent space. Finally, Limbeck et al. (2023) proposes a novel family of magnitude-bart measures to quantify the intrinsic diversity of latent spaces, while Wayland et al. (2024) employs persistent homology to measure the (dis)similarity of different representations.
Learning and Incorporating Invariance and Equivariance into Representations. Invariances in NNs can be enforced through various techniques operating at different levels, including adjustments to model architecture, training constraints, or input manipulation (Lyle et al. 22220). [Benton et al. (2020) proposes method to learn invariances and equivariances, [mmer et al. (2022) introduces gradient- based approach that effectively captures inherent invariances in the data. Meanwhile, van der Ouderaa & van der Wilk (2022) enables training of NNs with invariance specific transformations by learning weight-space equivalents instead of modifying the input data. Other works directly incorporate invariances into the model through specific constraints. RRth & Condurache (2023) enforces multi- stream architecture to exhibit invariance to various symmetry transformations without relying on data-driven learning; Kandi et al. (2019) propose an improved Convolutional Neural Network (CNN) architecture for better rotation invariance; Gandikota et al. (2021) introduces method for designing architectures that are invariant or equivariant to structured transformations; [Sanborn & Miolane (2023) propose a general method to achieve robust -nuuriinvariance in Group-Esqivariial CNNs; and Fumeer et al. (202I) proposes to recover the factors of variation underlying the data distribution by modeling them as product manifold. Finally, Moschella et al. (2023) proposes an alternative representation of the latent space that guarantees invariance to angle-preserving transformation without requiring additional training but only set of anchors, possibly very small (Cannistraci et al.. 2023), or very large (Norelli et al. 2023). This approach has shown efficiency in different settings (Crisostomi al. 2023 Riceiardi et al. 2023 Chen et al. 2023).
2

---

Published as a conference paper at ICLR 2024
Our work leverages the RR framework to directly incorporate set of invariances into the learned latent space. creating a product space of invariant components which, combined, can capture complex transformations of the latent space.
3 INFUSING INVARIANCES
Z
Relative representations
E
T 1
TM
M = M
r TRTM
E
Relative representations
Figure 2: Framework description. Given two latent spaces 2, Z' related by unknown transforma- tion T (resp. T-1), we assume that there exist a manifold . where samples in 2, 2' coincides when projected into M, via т.M- We approximate M building a product space M, where each space is RR computed using similarity function d; invariant to specific, known class of transformations. Combining the resulting spaces, we recover a representation which should approximate x TM-
Setting. We consider NNs parametric functions Fg compositions of encoding and decoding maps Fg = Da, o Eg,, where the encoder Eg,, is responsible for computing a latent representation Es, (z), a X' for some domain X, with dim << (x'(:);; and the decoder Da, is responsible for solving the task at hand (e.g., reconstruction, classification). In the following, we will drop the dependence on parameters for notational convenience. For a single module E (equivalently for D), we indicate with Exx if the module E was trained on the domain X. In the upcoming, we will summarize the necessary background to introduce our method.
Background. The RR framework (Moschella et al. 2023) provides straightforward approach to represent each sample in the latent space according its similarity to set of fixed training samples, denoted anchors. Representing samples in the latent space as a function of the anchors corresponds to transitioning from an absolute coordinate frame into a relative one defined by the anchors and the similarity function. Given domain X' an encoding function Ex X 2, set of anchors Ax C X, and a similarity or distance function d : Z x + R. The RR for each sample x E X is:
RR(2;Ax,d
(((a))
a.E.Ax
where = [x), and + denotes row-wise concatenation. Moschella et al. (023), ). d was set as Cosine similarity. This choice induces representation invariant to angle-preserving transformations. In this work, our focus to leverage different choices of the similarity function to induce a set of invariances into the representations capture complex transformations between latent spaces.
Overview. When considering different NNs F, F', we are interested in modeling the class of transformations T that relates their latent spaces Z, z'. T could be something known, e.g., rotations, or nontrivial, complex class transformations. The two networks could differ by their initialization seeds (i.c., training dynamics), by architectural changes, which could affect the latent space in a different way (as observed in Figure T). The fundamental assumption of this work is that these variations induce changes in the latent representations of the models, but there exists an underlying manifold M where the representations are the same (see Figure D. Formally:
even domain changes, i.c., X' * X',
Assumption. Given multiple models Fi..Fa assume that there exists manifold M which identifies an equivalence class of encoders Er induced by the class of transformation T fe.g. rota- tions), defined as er {E TMTE TMEE VT T, where πM represent the projection on M. M is equipped with a metric d which is preserved under the action of elements of T, i.e. (M (MMTTMMMMM)))))) (
VT ET.
3

---

Published as a conference paper at ICLR 2024
What we look for is function r which independently projects the latent spaces 21. into M and invariant to T. i.c. r(z) = r(Tz), for each T T, and for each 2 E 2n.. Generalizing the framework of Moschella et al. (2225) to arbitrary similarity functions, distance metrics, gives straightforward way to define representations r invariant to specific classes of transformations.
However, d is typically unknown priori, and in general, it is challenging to capture T with single class of transformations (as observed in Figure | and empirical demonstrated Section 4.1).
To overcome this challenge, in this work, we upproximate M with product space M := II M,
where each component is obtained by projecting samples of 2 in a space equipped with a different similarity function d, Each M, will have properties induced by similarity function d; invariant to
specific, known, class of transformations T; (e.g. dilations). By combining this set of invariances, we want to recover the representation such that approximates well T (see Figure 2). We define r formally as the projection from 2 M:
Definition (Product projection). Given a set of latent spaces Z.. related to one another by an unknown class of transformation T, a set of similarity functions DD each one invariant to a specific known class of transformations T: (e.g. rotations), i.e. RR(z,d. = RR(TT2 2, d.), VT E T; We define the producl projection r 2 M as:
r(z) = ø RR(Z;;x;d) Id.ED
where de is an segregation,functionn (e.g. concatenation) responsible for merging the relative spaces induced by each d; E D.
We give more details different strategies how implement & in section 3
Distance-induced invariances. We leverage the RR framework considering the following similarity functions d: Cosine, Euclidean, Manhattan (L1), Chebyshev (Loo), each one inducing invariances to a specific, known class of transformations. For formal definitions, synthetic examples, and visualizations, please refer to the Appendix A.2
Aggregation functions. This section summarizes different strategies to construct the product space M, directly integrating a set of invariances into the representations. Consider a latent space image of encoder E X' + Z, and set of similarity functions D. For each D. produce [D] relative latent spaces. Every space is produced via a similarity function (i.e., Cosine, Euclidean, L1. I,o), enforcing invariance to specific class of transformations.
These spaces can be aggregated using diverse strategies, corresponding to different choices of di
Concatenation (Concat): the subspaces are independently normalized and concatenated, giving to M the structure of a cartesian product space.
Aggregation by sum (MLP+Sum): similar to DeepSet Zhang al. 2019). the spaces are independently normalized and non- non-linearly projected. The resulting components are summed.
Self-Attention (SelfAttention) the spaces are independently normalized and aggregated via a self-attention layer.
When not specified, all the results are obtained using the Aggregation by swm strategy. For the implementation details of each strategy, please refer the Appendix A.3 The product space M yields robust latent representation, made of invariant components which are combined to capture nontrivial, complex transformations, boosting the performance on downstream tasks.
EXPERIMENTS
In this section, we perform qualitative and quantitative experiments to analyze the effectiveness of our framework in constructing representations that can capture complex transformations of the latent space. In Section 4.1 we empirically motivate our study by analyzing the similarity between latent spaces generated by trained from scratch AutoEncoder (AE) and Variational AutoEncoder (VAE) architectures. and pretrained foundation models. We analyze the emerging class of transformations between different latent spaces by enforcing invariances into the representations. Additionally, in
4

---

Published as a conference paper at ICLR 2024
Section assess the zero-shot stitching performance of framework across text, vision, and graph modalities. Finally, in Section 4.3 we conduct an ablation study on different aggregation functions, and in Section we investigate attention weights and their significance in selecting the optimal relative space.
4.1 LATENT SPACE ANALYSIS
TRAINING FROM SCRATCH
Experimental setting. We perform image reconstruction on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), MNIST (Deng 2012). and Fashion MNIST (Xiao et al. 2017) datasets (refer to Table for additional information). Using comparable convolutional architectures, we focus on AEs and VAEs. We consider models with unflattened latent image bottleneck (referred to AE and VAE) that preserve the image spatial structure, and models with linear projections into and out of a flattened latent space (referred as Linearized AutoEncoder (LinAR) and Linearized Variational AutoEncoder (LinVAE)). Using different random seeds, we train five instances of each model until convergence. Then we project each latent space into its relative counterpart using distinct projection functions to infuse different invariances in each representation. Finally, for each combination, we measure the cross-seed latent space similarity.
Result Analysis. In Figures 12 and 13 we report the Pearson and Spearman cross-seed correlations for various architectures on diverse datasets. These outcomes illustrate that it impossible unify the latent spaces obtained with different initializations by means of single invariance. For instance, Figure 3 we observe that the highest cross-seed similarity is achieved using different projection types when considering the VAE or Lin VAE architecture, even when keeping fixed all the other parameters. Additionally, dataset variations significantly alter the trends in the behavior all the architectures. This discovery challenges the assumption in Moschella et al. (2023) that angle- preserving transformations are the primary drivers of correlation among the latent spaces of models trained with different seeds. Please refer Figures 12 and
13 for additional results.
MNIST
CIFAR100
1.0
0.58
Plarson Condlation
Corrleaion
0.96
0.91
0.55
0.92
Projections
Projections
Cosine
Euslidear
L1
0:76
Cosine
Euciidea
Figure 3: Latent Spaces Cross-Seed Similarity. Cross-seed pearson correlation of latent spaces for AEs trained on MNIST (left) and CIFAR-100 (Iright)) until convergence. Notably, no single projection consistently outperforms others across all settings. The Los projection is not displayed to improve visualization.
PRETRAINED MODELS
Experimental setting. We analyze the similarity of latent spaces produced by pretrained foundational models in both the vision and text domains. For the vision domain, we evaluate five distinct foun- dational models (either convolutional or transformer-based) using the CIFAR-10) 10, CIFAR- 00, MNIST, and F-MNIS" STT datasets. Meanwhile, in the text domain, we assess seven different founda- tional models using the DBpedia (Zhang al.) 2015). TREC (Coarse) (Hovy al. [000)) and 4news (Text) (Wang et al. 2022) datasets. Refer to Table for details on the pretrained models and Table 7 for the datasets.
Result Analysis. In Figure we report the Linear CKA correlations for various pretrained models for vision and text modalities. This analysis highlights the absence of universally shared transformation class that connects latent spaces foundation models across distinct conditions. For example, on CTFAR-100 0, the highest similarity achieved with different projection functions when using different
5

---

Published as a conference paper at ICLR 2024
Cifar10
0.7
0.6
Unear CKA
0.4
00.5
0.4
Invariance
0.3
Cosine
Cosine,
RoxNot
VIRBB116 VIT-S/16
RVIT-8116
CVIT-832
BERTIC BERTFU ELECTRAROBERTO ALBERT
XLMR DVITB222
Figure 4: Latent Spaces Cross-Architecture Similarity. Linear CKA similarity of latent spaces across several pretrained models on CIFAR-1C (left) and TREC (right). In each bar, we report the space similarities distribution to the other models while infusing a specific invariance. There is no singular projection that consistently outperforms others across all configurations.
architectures. Moreover, it is possible to see that similar architectures (i.e., ViT- based models) exhibit similar trends when tested on the same dataset. Refer to Figure 14 for further results.
Takeaway. The transformation class that correlates different latent spaces produced by both pretrained and sained-from--cratchh models depends on the dataset, architecture, task, and possibly other factors.
4.2 DOWNSTREAM TASK: ZERO-SHOP STITCHING
Experimental setting. We perform zero-shot stitching classification using text, vision, and graph modalities with various models and datasets. For the vision and text domains, we used the same datasets and pretrained models employed in Section For the Graph domain, we employed the CORA dataset (Sen et al. 2008) and a Graph Convolutional Network (GCN) trained from scratch (refer to Tables 6 shot stitching reconstruction using AEs on CIFAR-100 Relative decoders are trained with three different seed values, and the resulting representations projecting the encodings onto 1280 randomly selected but fixed anchors. It is important to highlight, as demonstrated in Table and Figure I0] that varying the number of anchors results in different emerging transformations, indicating that a single projection function cannot capture the desired invariance. Nevertheless, our method achieves the highest score, regardless of the number of anchors.
and forr additional details of models and datasets). Additionally,
perform zero-
transformed into relative representations by
Zero-Shot Model Stitching. The stitching methodology allows combining components of different NNs obtain a new model. In this paper, we adopted the same setting proposed by Moschella et al.) (2023), where each element of the stitched model functions an autonomous frozen module: the encoder handles data embedding, while the dedicated relative decoder manages the downstream task. Refer to Figure for a visual depiction of the procedure. The approach is termed zero-shot because the stitching procedure is executed without any further training or fine-tuning.
Table 1: Graph and Text Classification Stitching Performance. Zero-shot accuracy scores across various decoders, seeds, and datasets. For the text domain, we use pretrained models, while for the graph domain, we train GCN models from scratch and evaluate the stitching across seeds. Additionally, we calculate the stitching index for the graph modality, showing that composing different projections using the Aggregation by sum enables zero-shot stitching without any performance drop in this setting, ensuring competitive end-to-end performance. Refer to Tables
and [4] for complete results.
Text
Graph
ALBERI
GCN
Projection
DBpediaf
TREC1
CORA 1
Stitching Index
Cosine
0.50 + 0.02
0.54 + 0.03
0.53 + 0.06
0.71
Euclidean
0.50 + 0.00
0.60 +: 0.03
0.27 107-100 0.06
0.58
L1
0,52 = 0.01
0.65 + 0.02
0.26 0.06
0.58
Loo
0.18 0.02
0.06
0.12 0.003
Cosine. Euclidean, Li, L.v
0.53 + 0.01
0.65 + 0.02
0.77 0.00
1.00

---

Published as a conference paper at ICLR 2024
Table Image Classification Stitching Performance Cross-Architecture and Cross-Seed. Zero- shot accuracy score across different pretrained models, seeds, and datasets. The proposed method using the Aggregation by sum consistently achieves the highest accuracy score or comparable results, without prior knowledge of the optimal projection to employ. See Table T0 for complete results.
Accuracy
Encoder
Projection
C1FAR-100
ClEAR-10
MNIST
F F-XN_ST
T-B/32
Cosine
0.52 0.03
0.02
0.61 0.06
Eaclidean
0.53 0.02
0.87
0000
0,66 0.05
0.70 = 0.03
0.53 0.04
0.87
0.02
0.66 0.05
0,70 0.03
Loe
0.27 0.04
0.52 + 0.04
0.57 1 0.03
0.55 + 0.01
Cosine, Euclidean, Lı, Lox
0.58 :: 0.03
0.88 0.02
0.68 0.05
0,70 = 0.01
RV1 RV11-8/16
Cosine
0.79 0.03
0.94 0.01
0.69
0.04
0.76 1: 0.03
Enclidean
0.79 0.03
0.94 1 0.01
0,71 0.04
0.77 1: 0.03
I4
0.77 0.04
0.95
0.01
0.71
0.04
0.79 :: 0.03
Loo
0.31 0.03
0.75 0.044
0.61 0.05
0.00 + 0.03
Cosine, Faclidean, Lg. Fro
0.81 0.04
0.95 0.01
0.72 0.04
0.76 + 0.04
Results Analysis. Tables and present the performance for the classification downstream task of various projection functions for different modalities. While Figure shows the qualitative results for the reconstruction task (see Table 23 for quantitative results). As previously observed in Section [. and Figure 3 the experiments reveal the absence of single optimal projection function across archi- tectures, modalities, and even within individual datasets. Our proposed framework, which leverages product space harness multiple invariances, followed by trainable aggregation mechanism, consistently achieves superior accuracy across most scenarios. It is important to emphasize that the dimensionality of each independent projection and the aggregated product space remains constant, and and 11 moreover, in Tables 15 to 21 show the performance for each pair of encoder and decoder without averaging over the architectures. The reference end-to-ens performance are reported in Tables to 33 better interpret the performance of the stitched models on the downstream tasks. To this end, we propose an additional evaluation metric named the Stütchig Index computed as the ratio between the stitching score and the end-to-end score. measures how closely the stitching accuracy aligns with the original score, i.c., stitching score of one indicates there is drop in performance when stitching modules. Results Table Tighhihhgh without any performance drop in this setting while still ensuring competitive end-to-end performance.
ensuring fair comparison. Additional stitching results on graphs and text in Appendix Table
highlight that our method enables zero-shot stitching
D1 F2
D2 Ex
D Fa
D2 Ea
Cos, Elce L1. Loc
Cos, Euc,
Cosine
Fuclidean
Frou
Absolute
Figure 5: Reconstruction Qualitative Results on CIFAR-100 using AEs. The last two sets of columns (D, o Eg and D2 E2) are the end-to-end AEs with unique initialization seeds, while the first two (D, o Feg and D2 FT) illustrate the outputs of the zero-shot stitching these independently trained models. The first row displays the source images, the two subsequent rows show the results from distinct combinations of projection aggregated through the Aggregation by sum, while the last one shows the outputs from baseline model that does not incorporate interesting to see that when using the Lax projection (second to last row), the reconstructed images blurred, but when removing the Lxc space from the aggregation (third row) the reconstruction drops in performance, meaning that it captures information not captured by the others.
methodology. It is
Takeaway. product space of invariant components can improve the zero-shot stitching performance without any prior knowledge of the class of transformation that relates different spaces.
7

---

Published as a conference paper at ICLR 2024
4.3 AGGREGATION FUNCTIONS: ABLATION
Experimental setup. We perform an ablation study on the merging strategies presented in Section 3 In these experiments, we conduct zero-shot stitching classification on the three modalities using the same datasets and models described in previous sections (refer to Tables 6 [nnd] [oor additional details of models and datasets). The relative decoders are trained using three different seeds, and the accuracy score is assessed each assembled model.
Table 3: Ablation Study on the Aggregation Functions. Zero-shot classification accuracy score different architectures, seeds, and datasets. The Aggregation by sum (third row) obtains consistently the best accuracy score. See Appendix Tables 12 to for the complete results.
Vision
Text
Graph
Aggregation
RViT-B/16
BERT-C
GCN
Concar
0,81 + 0,00
0.54 + 0.03
0.75 + 0.02
MLP+SelAAttention
0.84 + 0.01
0.51 = 0.03
0.63 = 0.13
MI.P+Sum
0.85 0.00
0.55 0.04
0.77 0.01
SelfAttention
0.76 + 0.03
0.36 + 0.22
0.76 + 0.02
Result Analysis. Tablee]present the results of the ablation study conducted using various aggregation methodologies (complete results can be found in Tables 14). Among the different methodologies, MLP+Sum outperforms the others consistently. This method preprocesses each subspace with an independent MultiLayer Perceptron (MLP). which includes normalization layer, linear layer, and tanh activation, and then sums the resulting representations. It is essential to highlight that the Concat aggregation method is not directly comparable to the others since it increases the dimensionality of the space linearly with the number of subspaces (refer to Appendix AA.4 for more details).
Takeaway. The aggregation by sum methodology consistently guarantees the highest performance between merging strategies without increasing the dimensionality of the space.
4.4 SPACE SELECTION
In the preceding sections, we discussed integrating individual and multiple invariances into the representation through various projection functions and appropriate aggregation strategies. In this section, we aim analyze and understand if tuning only the aggregation strategy at stitching time reasonable cost for selecting the optimal space. We focus on the Self-attention aggregation, which is a single self-attention layer as described in Section 3 and finetune only the Query, Key, Value (QKV) parameters (i.e., the ones responsible for space blending). Each space is generated by its own projection function. We remark that stitching-ti time fine-tuning is exclusive to this experiment.
Experimental setup. We identify two crucial components within the stitched model: (1) the linear projections associated with QKV in the attention mechanism, which is responsible for selecting and blending the spaces, and (2) the ML.P in the classification head following the attention mechanism, which classifies the aggregated embeddings. We examine two distinct approaches: the first approach fine-tunes only the first component (QKV opt), while the second one fine-tunes the second one (MLP opt). All the experiments in this section are conducted on the CIFAR 10C dataset using the RexNet as encoder and the ViT-B/1 1 6 as decoder.
Result Analysis. Table 4 summarizes downstream classification accuracy for the stitched model using various projection functions and aggregation strategies. Incorporating multiple invariances and aggregating them via Self-attention (fifth row) does not perform well; meanwhile, using the MLP+Sum or the Cosine projection alone is more effective. This is expected, considering the attention mechanism is trained to improve end-to-end performance rather than to maximize compatibility between different spaces. Incorporating the adaptation strategies at stitching time significantly boosts performance, either focusing on the optimal space selection and blending (QKV opt) the classification head (MLP opt). We find that an informed fine-tuning of the parameters responsible for the space blending (i.e., only the QKV projections) significantly impacts performances. even more than tuning the whole classifier. Figure 66 illustrates the attention weights averaged over the test dataset, the attention weights of the zero-shot stitched model (left) remain unchanged when
8

---

Euclidean Published as a conference paper at ICLR 2024
fine-tuning only the classifier (right). Meanwhile, fine-tuning the QKV projections (center) shifts the attention weights to allocate less importance to worse- worse-performing projections (i.e., Looo).
Table 4: Optimal Space Selection. Classification accuracy for the stitched model with RexNet encoder and ViT-B/10 as decoder on CIF AR-100, 100, using different projection functions and aggregation strategies. Fine-tuning the space selection and blending module (OKV opt) has more significant effect on performance improvement than tie-ttunng the MLP head (MLP opt).
Projection
Aggregation
Accuracy
Cosine
0.50
0.38
La
0.24
Loc
0.21
Cosine, Euclidean, L1. Lose Cosine, Euclidean, L1, Lio
SelfAttention MLP+Sum
0.17
0.45
Cosine, Euclidean, L1. Lox Cosine, Euclidean, L1. Lio
SelfAttention QKV opt
0.75
SelfAttention MLP opt
0.52
Original
QKV
opt
MLP
opt
0.99
0.00
0.00
0.00
0.86
0.01
0.02
0.10
0.99
0.00
0.00
0.00
Euclidean
0.01
0.80
0.09
0.10
Euclideem
0.00
0.99
0.01
0.00
Buclidean
0.01
0.80
0.09
0.10
0.01
0.65
0.28
0.06
111
0.00
0.92
0.08
0.00
0.01
0.65
0.28
0.06
Linf
0.38
0.03
Linf
0.00
0.07
LinE
0.05
0.03
0.54
Cosine
Eaclidean
1.1
Luif
Cosine
Raciddam
Loii
Cosine
Eadiddam
Linf
Figure 6: Attention Weights Visualization. Comparison of attention weights for the stitched model with RexNet encoder and B1TBB/16 decoder CIFAR-100, before and after fine-tuning. (left) the attention weights of the initial zero-shot stitched model, which remain unchanged when fine-tuning the classifier (MLP opt) (right). Conversely, fine-tuning the QKV projections (OKV opt) (center) leads to a notable shift attention weights, assigning lower importance to the space that performs worst individually.
Takeaway. Appropriate selection and aggregation of the optimal invariant spaces are crucial in enhancing latent communication between neural models.
CONCLUSION
In this paper, we introduced a framework incorporate invariances into neural representations to enhance latent space communication without prior knowledge of the optimal invariance to be enforced. Constructing a product space with invariant components, we showed that it possible to capture large class of complex transformations between latent spaces within single representation that is robust to multiple changing factors such dataset, architecture, and training hyperparameter variations.
Limitations and Future Works. Our framework allows to incorporate multiple invariances single latent representation, induced by specific similarity functions. However, this can become limiting when a similarity function cannot be modeled analytically, or expressed closed form. In such cases, an interesting direction would be to learn the similarity functions, paving the way to an even more extensive set of possible invariances to be enforced in the representation. Furthermore, while our method achieves good results on the tested benchmarks, optimal invariant spaces can yield equivalent performance using fewer components. Therefore, integrating better space selection strategy holds promise for future research.
observed that selecting the

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The authors gratefully acknowledge the anonymous reviewers and the AC for the thoughtful remarks, and Luigi Gresele and Emanuele Marconato for the insightful discussions. This work is supported by the ERC grant no. 80.540255 (SPECGEO), PRIN 2020 project 2020TA23NNN (LEGO.AI), and PNRR MUR project PE00000L3-FAIR.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: (007.0299000293 2019.
Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew Wilson. Learning invariances in neural networks from training data. Advances in neural information processing systems, 33:17605-17616. 2020.
Irene Cannistraci, Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, and Emanuele Rodolà. Bootstrapping parallel anchors for relative representations. In Krystal Maughan, Rosanne Liu, and Thomas F. Burns (eds.), The First Tiny Papers Track at ICLR 2023, Tiny Papers @ ICLR 2023, Kigali, Rwanda, May 5, 2023. OpenReview.net, 2023. URL https:
net met/pdffic=vBulte
Chi Chen, Peng Li, Maosong Sun, and Yang Liu. Weakly supervised vision-and-langun lanlauage pre-training with relative representations. In Anna Rogers, Jordan Boyd-Graher, and Naoaki Okazaki (eds.), Proceedings of the sl Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8341-8355. Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.11665330 10.11866/vv22/ 20//5///1/222..co www.:/22...lllo...4666..
URL https: http//aaaannttooooooooooo org/
2023.acl-lomm 464
Kevin Clark, Minh-Thany Luong, Quoc V. Le, and Christopher D. Manning, ELECTRA: pre-training text encoders discriminators rather than generators. 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpt:tt/tppseeeeeeet
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999, PMLR, 2016
Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homoge- neous spaces. Advances neural information processing systems, 32, 2019.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv: /91 1,02,22 2019.
Keenan Crane, Clarisse Weischedel, and Max Wardetzky. The heat method for distance computation. Commun. ACM, 60(11):90-99, October 2017. ISSN 0001-0782. doi: 10.1145/3131280. URL
httpp://dii acm. http.//..gg/1111153311111222
Donato Crisostomi, Irene Cannistraci, Luca Moschella. Pietro Barbiero, Marco Ciccone, Pietro Lio. and Emanuele Rodolà. From charts to atlas: Merging latent spaces into one. In NewrIPS 2023 Workshop on Symmetry and Geometry in Neural Representations, 2023. URL https //openreview, wttp:/iwww.eeeffummttt.oooo forum?id=2F ut
MohammadReza Davari, Stefan Horoi, Amine Natik, Guillaume Lajoie, Guy Wolf, and Eu- gene Belilovsky. Reliability of cka as a similarity measure in deep learning. arXiv preprint arXiv:2210.16156 2022.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 1916):1111442 2012.
10

---

Published as a conference paper at ICLR 2024
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171 Minneapolis, Minnesota, 2019. URL https:/7
Association for Computational Linguisties. doi: 10. 10.18653/1//N19-1423.
wwwnthhooog..rrg/MM... 1423,
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OvenReview.net, 2021. URL YicbFaNTTy
Alhussein Fawzi, Horst Samulowitz, Deepak Turaga, and Pascal Frossard. Adaptive data augmenta- tion for image classification. In 2016 IEEE international conference on image processing (ICIP), pp. 3688-3692. leee, 2016.
Marco Fumero, Luca Cosmo, Simone Melzi, and Emanuele Rodola. Learning disentangled representations via product manifold projection. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Pro- ceedings of Machine Learning Research, pp. 3530-3540. PMLR, 18-24 Jul 2021. URL
/ttps::/prrceeddnng wttpeeediiing...........e. www.s/pesssss.eeeeeeeeeeess wttpss///w///uerree/e/2e fumero21a, 1ttmm
Kanchana Vaishnavi Gandikota, Jonas Geiping, Zorah Lähner, Adam Czapliński, and Michael Moeller. Training or architecture? how incorporate invariance neural networks. arXiv preprint 200044 2021.
Dongyoon Han, Sangdoo Yun, Byeongho Heo, and YoungJoon Yoo. Rethinking channel dimensions for efficient model design. ArXiv preprint, abs/2007.00992. 2020. httplattpp///xi.....// org/
abs 2007. 00992
Irina Higgins, Sébastien Racanière, and Danilo Rezende. Symmetry-based representations for artificial and biological general intelligence, 2022.
Harold Hotelling. Relations between two sets of variates. Breakthroughs in statistics: methodology and distribution, pp. 162-190. 1992.
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. To- ward semantics-basee answer pinpointing. In Proceedings of the First International Conference on Human Language Technology Research, 2001. URL https::///// https////ccaaathooooo.ooo org/
H01 11-10
Alexander Immer, Tycho van der Ouderaa, Gunnar Rätsch, Vincent Fortuin, and Mark van der Wilk. Invariance learning in deep neural networks with differentiable laplace approximations. Advances in Neural Information Processing Systems, 35:12449 1244444-2443,,, 2022.
Haribabu Kandi, Ayushi Jain, Swetha Velluva Chathoth, Deepak Mishra, and Gorthi RK Sai Subrah- manyam. Incorporating rotational invariance in convolutional neural network architecture. Pattern Analysis and Applications, 22:935-948, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
7/arxiv. www./rg.bbb....... hbtp://bb//44... /1412. 6980
Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: survey of functional and representational measures. arXiv preprint arXiv:2305.06322 063699 2023.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
11

---

Published as a conference paper at ICLR 2024
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.
Ruslan Kuprieiev, skshetry, Dmitry Petrov, Pawet Redzyússii Peter Rowlands, Casper da Costa- Luis, Alexander Schepanovski, Ivan Shcheklein, Batuhan Taskaya, Gao, Jorge Orpinel, David de la Iglesia Castro, Fábio Santos, Aman Sharma, Dave Berenbaum, Zhanibek, Dani Hodovic, daniele, Nikita Kodenko, Andrew Grigorev, Earl, Nabanita Dash, George Vyshnya, Ronan Lamy, maykulkarni, Max Hora, Vera, and Samidhya Mangal. Dve: Data version control git for data & models, 2022. URL URL httph: org 10. 5281/ / zenodo. 7083378
Zorah Lähner and Michael Moeller. On the direct alignment of latent spaces. In UniReps: the First Workshop on Unifying Representations in Neural Modely, 2023. URL
fort/forum?ino wtt.orrmmmdnnoottrF...
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self sulf-supervised learning of language representations. arXiv preprint arXiv:1909. 1942, 2019.
Katharina Limbeck, Rayna Andreeva, Rik Sarkar, and Bastian Rieck. Metric space magnitude for evaluating unsupervised representation learning. arXiv preprint arXiv:2311. 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11666 2019. URL https:
/ttpstts: aawiivaiivvv///////////a///aaaia ozg/ abs / 1907 .1692
Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Reddy. On the benefits of invariance in neural networks. arXiv preprint arXiv:2005.00/78 2020.
Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodolà. Latent space translation via semantic alignment. In A. Oh, Neu- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 55394-55414. Curran Associates, Inc., 2023. URL httpps://prroceeedgggg..eeerrrre wwwpeurriip.ccppppppeeeeeepeeeee httpreerrffeeeeepae/////// file/ F
Giovanni Luca Marchetti, Christopher Hillar, Danica Kragic, and Sophia Sanborn. Harmonics of learning: Universal fourier features emerge invariant networks, arXiv preprint arXiv:2312.08550, 2023.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances Neural Information Processing Systems, 31, 2018.
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In The Eleventh International Conference on Learning Representations, 2023. URL https://
openreview.nor wtt.:eeteeooommmaddd----eeee
Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, and Francesco Locatello. Asif: Coupled data turns unimodal models multimodal without train- ing. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 15303-15319, Curran Asso- ciates, Inc., 2023, htt///ile/31181
files/paper
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Maithra Raghu, Justin Gilmer, Jason Yosinski. and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2024
Matthias Rath and Alexandru Paul Condurache. Deep neural networks with efficient guaranteed invariances. In International Conference on Artificial Intelligence and Statistics, pp. 2460-2480.
PMLR. 2023.
Antonio Pio Ricciardi, Valentino Maiorea, Luca Moschella, and Emanuele Rodolà. Zero-shot stitching in reinforcement learning using relative representations. In Sixteemth European Workshop on Rein- forcement Learning. 2023. URL httptss////peeeeeiiee nwt. http://wwwne.eeeoommmmeeeeeeeeee Inf
Justin Salamon and Juan Pablo Bello. Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal processing letters, 24(3):279-283, 2017.
Sophia Sanborn and Nina Miolane. A general framework for robust g-invariance in g-equivariant networks. In Sevetty Conference on Neural Information Processing Systems, 2023. URL
hittss://openrevile wtt.://reee/eetree/////e
net/ Forum? 2i 611Mixx.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective Classification in Network Data. AlMag., 29(3):93, 2008. ISSN 2371-9621. doi: (tt.9///amaa..293.2125..
Hang Shao, Abhishek Kumar, and P. Thomas Fletcher. The riemannian geometry of deep generative models, 2017.
Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 29065500):2319-23888 2000.
Tycho FA van der Ouderaa and Mark van der Wilk. Learning invariant weights in neural networks. In Uncertainty Artificial Intelligence, pp. 1992-2001. PMLR, 2022.
Zhen Wang, Xu Shan, Xiangxie Zhang, and Jie Yang. N24News: A new dataset for multimodal news classification. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 6766-677777 Marseille, France, June 2022, European Language Resources Association. URL
1rec-1.729
Jeremy Wayland, Corinna Coupette, and Bastian Rieck. Mapping the multiverse of latent representa- tions. arXiv preprint arXiv:2402.01514 2024.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5028-50337 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. mashion-mmist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 2017.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 649 657, 2015. URL wtt.sttttpps/ppcceediineee neurip wctpppppp/.../////e.c.c 22BCBB611c73355GG63ee33372----2-
html
Yan Zhang, Jonathon Hare, and Adam Prugel-Bennett. Bennett. Deep set prediction networks. Advances in Neural Information Processing Systems, 32, 2019.
Yanzhao Zhou, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. Oriented response networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. pp. 519-528, 2017.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A. REPRODUCIBILITY STATEMENT
we provide detailed description of the proposed framework and the experimen- tal settings for the various scenarios. In the following sections, we present all implementation details that are not described in the main manuscript. Additionally, we are releasing a modular Py PyTorch implementation
In Section Section 4
A.2
DISTANCE-INDUCED INVARIANCES DETAILS
A.2.1 DISTANCES DEFINITION
This section provides additional details about the metrics described in section Section Cosine. Given vectors 44, V, the cosine similarity is defined as:
cos(u, (u, v) ||u|| |||||
(2)
Euclidean. Given two vectors u, the Euclidean distance is defined as:
d(u, v) (n u;)2
(3)
Manhattan (T,). Given two vectors v, the Ta distance is defined as:
d(u, v) Σ
(us - v,)l
(4)
Chebyshev (Loc). Given two vectors u, v, the Lxx distance is defined as:
d(u, v) max(Iu yil) lim p-000
Σ
x;
Vil P
(5)
and can be approximated in differentiable way employing high values for p.
Geodesic distance. Given manifold M and its parametrization g ZZ + we can represent the Riemannian metric as symmetric, positive definite matrix G(2) defined at each point in Z. G(2z) can be obtained as G(z) = J=(()) where (2)2) indicates the Jacobian of g evaluated at z. This metric enables to define inner product tangent spaces M. Considering smooth curve y [a, b] + 2, this corresponds to a curve on M via g (t). Its are length is defined as:
I(w) = S. V
(6)
A geodesic curve is curve that locally minimizes the are length, corresponding to minimizing the following energy functional:
El7) 2 [. ((1) G()()a)
2
In Figure 7 we show how geodesic distance is preserved under several classes of transformations, including manifold isometries, i.e., possibly nonlinear transformations that preserve the metric on M. In the synthetic experiment, geodesic distances are computed using the heat method Crane et al.
(2017). and the manifold isometry is calculated using Isomap (Tenenbaum et al.] 2000). Possible approaches to extend geodesic computation to real cases when dim(22 3 include Shao et al. (2017). We leave this promising direction for future work.
14

---

Published as a conference paper at ICLR 2024
Original
boomety
Gooleic
40
14 Sotete
Euchuken
Figure 7: Qualitative Synthetic Results. We show invariances induced using geodesic distance- based representation. We plot geodesic distances (top row) from the violet star point with values going from blue (closer) to red (farther). On the bottom row, we compare with Euclidean distances, showing that the latter does not estimate nor preserve well the metric information under transformations the manifold.
A.2.2 INFUSED INVARIANCES
In 5 summarize the invariances guaranteed by different distance metrics concerning the following standard classes of transformations: Isotropic Scaling (IS), Orthogonal Transformation (OT), Translation (TR), Permutation (PT), Affine Transformation (AT), Linear Transformation (LT), and Manifold Isometry (MIS). Where MIS is an isometric deformation of the manifold that preserves the geodesic distances between points, see Figure 7 for a synthetic example. general, capturing the set of invariances induced by similarity function is not straightforward. For example, the Lox. distance does not enforce isometry invariance the representation but, simultaneously, induces an invariance to perturbations in dimensions other than the maximum one. Formalizing and analyzing such types of invariances presents challenges since these transformations cannot neatly classified into a specific simple class of transformations.
Table 5: Invariances. Overview of the different distance-induced invariances.
Similarity
Isotropic
Orthogonal Transf.
Affine Transf.
Linear Transf.
Manifold Isometry
Function
Scaling
Translation
Permutation
Absolute
Cosine
X
X
Euclidean
X
V
X
X
Manhattan
X
X
Chebyshev
X
X
X
Geodesic
X
V
A.3 AGGREGATION FUNCTIONS
In this section, we report the implementation details of the aggregation functions & described in Section There are two possible preprocessing strategies applied each subspace independently:
Normalization layer: independent LayerNorm for each subspace.
MLP: a compact, independent, fully connected network defined for each subspace, comprised of LayerNorm, Linear layer, and Tanh activation function.
These preprocessing modules can be applied before either the (Sum) or attenittention aggregation strategies.
15

---

Published as a conference paper at ICLR 2024
Original
Isotropic Scale
Orthogonal
Translation
Linear Transformation
Absolute
0.5
0.0 0
0
0
-05
0
0.5 0.0 0.5
0
2
Cosine
0.95
0
0
0
0
0
0.95
1.00
Eaclidean
60
2
2
L1
2.5
0.0
0
5
Linf
0
0
5
Figure 8: Qualitative synthetic experiments using grid initialization. We consider a synthetic absolute latent space by initializing points in grid shape (top left). We then apply various transfor- mations the absolute space, converting it into different transformed spaces (first row). We convert the entire first row into the corresponding relative space for all the different similarity functions con- sidered (i.e. Cosine, Euclidean, L, and I. Observing which transformation does not change the original relative space (left column), shows which projections induce an invariance to each considered transformation.
A.4 IMPLEMENTATION DETAILS
This section details the experiments conducted in Section 4 Table 6 contains the full list of the pretrained models, while Table 7 contains dataset information.
A.4.1 TOOLS & TECHNOLOGIES
All the experiments presented in this work employ the following tools:
PyTorch Lightning, to ensure reproducible results while also getting clean and modular codebase;
NN-Template GrokAI (2021), easily bootstrap the project and enforce best practices;
Transformers by HuggingFace, to get ready-to-uss transformers for both text and images;
Datasets by HuggingFace, to access most of the datasets;
DVC (Kuprieiev et al. 2022), for data versioning:
16

---

Published as a conference paper at ICLR 2024
Table Pretrained models details. Details of the pretrained feature extractors with their Hugging- Face key, their alias, and their latent space dimensionality.
Modality
HuggingFace model name
Alias
Enc. Dim
bert-base-cased wurt-base-uncasel
BERT-C
Devlin et al.
2019
768
BERT Devlin cl al.
2019
768
ELECTRA CClark a
2020
768
Language
coberta-base
RoBERTa (Luu et 2019)
768
albert-buse-v2
ALBERT Lan cl al.
2019
768
xim-roberta-bass
XLM-R (Conneau et al. 2019
768
openai/elip-vit-base--pp
CViT-B732 (Kadford et
2021
768
rexnet_100
RexNet (Han et al. 2020
1280
vit att.small-patchh..22
ViT-S/ (Dosovisskiy et 2021
384
Vision
vit brasc_patchl 66_34 6_384
ViT-B/16
Dosovitskiy et al.
202
768
RViT-B/16 (Dosovitskiy et al.
2021
768
apenai/elip-vit-base-
CViT-B/32 (Radford 2021
768
Graph
GCN
GCN
300
Table 7: Dataset details. Details of the HuggingFace datasets used in the classification and recon-
struction experiments, with the associated number of classes.
Modality
Name
Number of Classes
MN ST
10
Fashi
10
Image
CIFAR-10
10
CIFAR-100
20 (coarse) - 100 (fine)
ImageNetlk
1000
TREC
(coarse) - 50 (fine)
Text
DBpedia
14
424news (Text)
24
Graph
CORA
7
A.5
LATENT SPACE ANALYSIS
RECONSTRUCTION
This experiment adopts convolution-baaee AE and VAE. The design variations encompass 2D- bottleneck architectures with dimensionality of 16 x 7 x and ID-bottleneck architectures with a dimensionality of 784 16 x 7 x for fair comparison. The models with 2D bottlenecks are endowed with approximately 50k parameters (AE) and 60k parameters (VAE), while their linearized counterparts possess 1.3 million parameters (LinAE) and 1.9 million parameters (Ain Intrigu- ingly, the variants preserving spatial structure in the latent space demonstrate superior performance despite having significantly fewer parameters. All models undergo training using the Adam optimizer (Kingma& Ba 2015) with learning rate set 1e-3. Early stopping employed based validation reconstruction eтror, quantified by mean squared error.
A.6 SPACE SELECTION
In Table 8 and Figure we present similar results to the ones reported Section 4.4, but using a more expressive classifier. This choice establishes less favorable scenario since fine-tuning more parameters in the PrffAtttntionnMM.. opt setting. It illustrates that is still better to fine-tune the QKV rather than the whole classifier.
17

---

Published as a conference paper at ICLR 2024
Table 8: Optimal Space Selection. Classification accuracy for the stitched model with RexNet as encoder and ViT-B/16 as decoder on CIFAR-100, using different projection functions and aggregation strategies. Fine-tuning the subspace selection and blending part (QKV opt) has more significant effect on performance improvement than fine-tuning only the larger MLP (MLP opt).
Projection
Aggregation
Accuracy
Cosine
0.52
Euclidean
0.42
L1
0.34
Loo
0.22
Cosine, Euclidean, L1, Los Cosine, Euclidean, L1, Loo MLP+Sum
SelfAtention
0.25
0.43
Cosine, Euclidean, L1, Los Cosine, Euclidean, L1, Loo
SelfAttention QKV opt
0.75
SelfAttention MLP opt
0.65
Original
QKV
opt
MLP
opt
1.00
0.00
0.00
0.00
0.96
0.00
0.01
0.03
1.00
0.00
0.00
0.00
Eucisdoan
0.00
0.76
0.16
0.07
Buclidess
0.00
0.86
0.14
0.00
Buclidean
0.00
0.76
0.16
0.07
11
0.01
0.55
0.39
0.06
L1
0.00
0.59
0.41
0.00
L1
0.01
0.55
0.39
0.06
Lind
0.32
0.03
0.63
0.00
0.88
0.11
0.01
Lint
0.02
0.03
0.63
Cosine
Enchidean
L1
Lunt
Cosine
Eachidoon
1.1
Lini
Cosine
Eaciddmm
Linf
Figure 9: Attention Weights Visualization. Comparison of attention weights for the stitched model between RexNet and FiT-B/16 on CIFAR- 10PAR100, 00, before and after fine-tumi veeettunigg (left) the attention weights of the initial zero-shot stitched model, which remain unchanged when fine-tuning the classifier (MLP opt) (right). Conversely, fine-tuning the QKV projections (WKV opt) (center) leads to notable shift attention weights, assigning greater importance to the subspace that performs better individually.
A.7 ANCHOR SELECTION ANALYSTS
when referring random but fixed anchors, we mean that the anchors are uniformly randomly sampled from the training set, but with fixed seed value. Thus, we conducted an analysis on the number of randomly selected anchors for the stitching task using CIFAR-109 across three different anchor selection seeds. The results in Table 9 and Figure 10 reveal that varying the number of anchors leads to different transformations, indicating that a single projection function cannot incorporate the desired invariance. In contrast, method enables the attainment of the highest score regardless of the number anchors or the seed value employed for the uniform random sampling of anchors.
In this section, we present the analysis performed on the anchor choice. In Section 4.2
A.8 STITCHING PROCEDURE
In Figure 11 illustrate in detail the stitching procedure introduced in Section 4.2
18

---

Published as a conference paper at ICLR 2024
Projections
Acunay Sore
Narbon
Figure 10: Accuracy Number of Anchors. Each box represents stitched model using CViT-B/32 and ViT-B/16 on CIFAR-100. The proposed methodology using the Aggregation by sum consistently outperforms other results regardless of the number of anchors.
Encoding
E
E
E
L
Projection(s)
RR
RR
Decoding Aguregation
D
Figure 11: Stitching Procedure Description. Given two trained models (lefi and center columns), the zero-shot stitched model (right column) is formed by combining the encoder of the first model with the decoder the second one. The zero-shot stitched model does not require additional training or fine-tuning. Although certain components of our module have learnable parameters (c.g., Φ), these parameters are exclusively trained during their respective network training phase and are subsequently utilized in their trained, frozen state in the stitched model.
19

---

Published as a conference paper at ICLR 2024
Table 9: Number of Anchors Ablation. sero-shot stitching classification performance results using CViT-B/32 32 and ViT-B/16 CIFAR-100. The proposed methodologies utilizing the Aggregation by sum consistently outperform the other results regardless of the number anchors.
Number of Anchors
Projection
Accuracy t
Cosine
0.05 + 0.02
Euclidean
0.05 + 0.02
T
0.06 + 0.02
Lec
0.04 = 0.02
Cosine, Euclidean, L1, Loo
0.07 0.02
Cosine
0.24 0.04
Euclidean
0.19 0.02
L1
0.16 = 0.02
Loc
0.14 0.02
Cosine, Euclidean, L1, Loo
0.28 0.02
Cosine
0.51 = 0.02
0.02
Fuclidean
0.46 + 0.04
50
L
0.44 + 0.03
Loc
0.25 = 0.03
Cosine, Euclidean, L,, Loe
0.58 + 0.05
Cosine
0.54
Euclidean
0.55 + 0.05
70
L1
0.54 = 0.05
Loc
0.27 = 0.04
Cosine, Euclidean, Lt, Loo
0.63 + 0.06
Cosine
0.59 + 0.02
Euclidean
0.58 + 0.05
100
T
0.59 + 0.05
Loc
0.28 = 0.03
Cosine, Euclidean, L,, Loe
0.67 + 0.04
Cosine
0.72 + 0.03
Euclidean
0.67 = 0.08
300
L1
0.70 = 0.06
Loc
0.34 = 0.04
Cosine, Euclidean, L1, Loo
0.76 0.05
Cosine
0.78 + 0.02
Fuclidean
0.73 + 0.08
800
L
0.77 + 0.05
Too
0.38 + 0.05
Cosine, Euclidean, L1, Loo
0.82 + 0.04
Cosine
0.80 + 0.02
Euclidean
0.75 = 0.09
1280
L1
0.79 + 0.05
Loe
0.38 = 0.05
Cosine, Euclidean, L1, Loo
0.83 0.04
Cosine
0.80 0.02
Euclidean
0.76 = 0.09
2000
L
0.79 + 0.06
Thee
0.39 + 0.05
Cosine, Euclidean, L1, Loo
0.83 + 0.05
20

---

Published as a conference paper at ICLR 2024
A.9
ADDITIONAL RESULTS
MNIST
F-NNIST
0.38
Pearson Conslation
0.38
Pearson Conreattion
0.50
0.91
0.90
Projection
Projections
Cosino
Euciidean
Costre
Eudiidean
LiNÆE
LiVAE
CIFARIO
CIFARI00
1.00
1.00
0.55
Pearson Corrlatton
0.90
0.00
0.85
0.65
0.88
0.73
Projections
Projections
Cosino
Euslidean
0.76
Cosine
Euclidean
L1
0.70
VAE
Figure 12: Latent Spaces Cross-Seed Similarity. Pearson correlation, measuring the similarity of latent spaces across different seeds for various architectures and datasets.
MNIST
F-MNIST
Spaarmen Conrlation
0.96
Spearman Carrelation
0.53
0.94
0.80
0.88
Projections
Cosine
Eusideen
Coeine
Euciddean
CIFAR10
CIFAR100
1.00
1.00
0.38
Sparrnn Carelatton
0.98
0.94
0.38
0.30
Projections
Projections
Cosine
Euslidean
Cosine
Eucidean
0.58
AE
VAE
LiVEE
LINVAE
VAE
LinÆE
LnnWE
Figure 13: Latent Spaces Cross-Seed Similarity. Spearman correlation, measuring the similarity of latent spaces across different seeds for various architectures and datasets.
21

---

Published as a conference paper at ICLR 2024
Mrist
Fashion
0.7
0.9
0.6
077
Linear CKA
Linear CKA
0.6
0.4
0.
0.4
Inyariance
0a
Inveriance
Casino
Cosine
Euclidaa
02
Eucidean
0.1
RecNet
VIT-B/18
VIT:S/16
RVIT-8/18 8
CV/T-8/22
Roxhot
VIT8115
VIT-5/16
RVIT
CVTT-8/22
Depadia
0.5
0.7
0.6
UhearCKA
0.8
ALS
0.4
0.4
0.3
0.2
Invariance
Cosino
01
Coaine
0.1
Hudioean
RecNet
VIT-B118
VIT-S/16
RVIT-8/18
CVT-BB322
BERT-C BERFU
ELECTRARUBERTA ALBERT
XLMRR CVIT-8/22
Figure 14: Latent Spaces latent spaces of pretrained models across different architectures and datasets.
Similarity. Linear CKA, measuring the similarity of
Table 10: Image Stitching Performance Aress-Architecture and Cross-Seed. Zero-shot accuracy
score for image classification task across different pretrained models, seeds, and datasets. The proposed method using the Aggregation by sum consistently achieves the highest accuracy score or comparable results, even without prior knowledge of the optimal projection employ.
Accuracy
Encoder
Projection
CIFAR-10 100
CIFAR-11 10
MNIST
Fashion MNIST
CV1T-3/32 Cosine Euclidean
0.52 0.03
0.87 + 0.02
0,61 0.06
0.68 0.02
0.53 0.02
0.87
0.02
0.66 0.05
0.70 0.03
L1
0.53 0.04
0.87
0.02
0.66 0.05
0.70 0.03
L.0
0.27 0.04
0.52 0.04
0.57 0.03
0.55 0.01
Cosine, Fuclidean, la I.
0.58 + 0.03
0.88 0.02
0.68 0.05
0.70 0.01
RVIT 3/16
Cosine Euclidean
0.79 = 0.03
0.01
0.69 0.04
0.76 + 0.03
0.79 = 0.03
0.94
0.71 = 0.04
0.77 0.03
0.77 0.04
0.95+00.0 0.01
0.71 = 0.04
0.79 0.03
I...
0.31 0.03
0.75 + 0.04
0.61 + 0.05
0.60 0.03
Cosine, Euclidean, L,, Loo
0.81 + 0.04
0.01
0.72 0.04
0.76
RexNet
Cosine Euclidean
0.50 + 0.02
0.79 - 0.01
0.01
0.71 = 0.02
0.74 1: 0.01
0.43 0.06
0.72 - 0.02
0.69 = 0.04
0.76 - 0.01
L1
0.33 0.06
0.70 + 0.01
0.69 = 0.04
0.76 + 0.02 0.02
Loo
0.19 0.03
0.48 + 0.03
0.48 = 0.02
0.60 + 0.05
Cosine, Euclidean, Loo
0.52 + 0.05
0.75 0.01
0.70 0.03
0.75 0.02
ViT-B/16
Cosine
0.75 = 0.05
0.96 - 0.01
0.59 = 0.05
0.79 - 0.03
Euclidean
0.76 = 0.05
0.96 - 0.01
0.65 = 0.06
0.81 + 0.02 0.02
L1
0.76 + 0.06
0.96 - 0.01
0.66 = 0.07
0.81 + 0.02
Loo
0.42 0.02
0.70 + 0.05
0.42 = 0.05
0.52 + 0.04
Cosine, Fuclidean, In, I.x.
0.81 + 0.05
0.96 + 0.01
0.66 0.04
0.80 0.04
ViT S/32
Cosine Euclidean
0.73 + 0.04
0.93 0.01
0.68 0.04
0.77 + 0.02
0.64 0.03
0.93 0.01
0.70 = 0.02
0.77 + 0.02
L1
0.58 0.09
0.93 0.01
0.70 0.03
0.78 0.02
0.33 0.04
0.69
0.03
0.48 0.02
0.53 + 0.02
Cosine, Euclidean, L,, Loo
0.73 + 0.05
0.93 + 0.01
0.69 = 0.02
0.76 +: 0.02
22

---

Published as a conference paper at ICLR 2024
Table Text Stitching Performance Cross-Architecture and Cross-Seed. Zero-shot accuracy score for text classification task across different pretrained models, seeds, and datasets. The aggre- gation function is the Aggregation by sum and the results are obtained using a linear classifier as decoder, instead of MLP in Moschella al. (2023).
Accuracy
Encoder
Projection
DBpedia
TREC
N24news (Text)
ALBERT
Cosine
0.48 + 0.05
0.49 + 0.08
0.09 + 0.02
Euclidean
0.49 + 0.05
0.54 + 0.06
0.09 + 0.03
L1
0.51 + 0.04
0.59 + 0.06
0.09 + 0.03
Loo
0.17 + 0.02
0.32 + 0.07
0.07 + 0.01
Cosine, Euclidean, L1. Loo
0.50 + 0.06
0.53 + 0.06
0.09 + 0.03
BERT-C
Cosine
0.46 + 0.07
0.440.00
0.21 + 0.09
Euclidean
0.48 = 0.09
0.57
0.22 + 0.09 0.21 = 0.09
L1
0.50 = 0.10
0.59 + 0.06
Loo
0.13 + 0.04
0.21 + 0.04
0.11 + 0.04
Cosine, Euclidean, L1, Loo
0.50 + 0.13
0.54 + 0.07
0.21 + 0.09
BERT-U
Cosine
0.51 = 0.05
0.46 + 0.12
0.15 + 0.06
Fuclidean
0.36 + 0.05
0.56 + 0.07
0.17 + 0.06
T
0.36 + 0.06
0.58 0.09
0.16 + 0.06
Loo
0.12 + 0.02
0.28 + 0.07
0.06 + 0.01
Cosine, Euclidean, L1, Loo
0.43 + 0.07
0.51 0.10
0.17 + 0.07
CViT-B/32
Cosine
0.20 + 0.02
0.50 + 0.04
0.08 + 0.03
Euclidean
0.22 + 0.02
0.48 + 0.10
0.12 + 0.05
L1
0.22 + 0.02
0.57 0.07
0.11 + 0.03
Loo
0.12 + 0.02
0.25 + 0.09
0.07 + 0.02
Cosine, Euclidean, L1 Loo
0.22 + 0.01
0.50 + 0.09
0.10 + 0.03
ELECTRA
Cosine
0.27 + 0.05
0.39 + 0.11
0.04 + 0.01
Euclidean
0.25 + 0.05
0.53 0.05
0.04 + 0.01
L
0.33 + 0.06
0.50 0.10
0.05 + 0.01
0.11 + 0.02
0.26 + 0.11
0.05 + 0.01
Cosine, Euclidean, L1, Loo
0.32 + 0.07
0.52 + 0.14
0.04 + 0.01
RoBERTa
Cosine
0.31 + 0.07
0.43 + 0.05
0.41 + 0.18
Euclidean
0.26 + 0.05
0.52 - 0.11
0.39 + 0.18
LI
0.30 + 0.06
0.56 + 0.07
0,45 + 0.20
Loo
0.13 + 0.01
0.20 + 0.07
0.08 + 0.02
Cosine, Euclidean, L1, Loo
0.34 + 0.08
0.53 0.09
0.43 0.20
XLM-R
Cosine
0.13 + 0.03
0.39 1.10
0.20 + 0.10
Euclidean
0.16 + 0.03
0.53 0.06
0.26 + 0.11
L
0.32 + 0.06
0.61 + 0.10
0.32 + 0.14
Loo
0.08 + 0.01
0.30 + 0.08
0.07 + 0.01
Cosine. Euclidean, L1, Loo
0.21 + 0.03
0.55 + 0.07
0.27 + 0.14
23

---

Published as a conference paper at ICLR 2024
Table 12: Image Classification Ablation on the Aggregation Functions. Zero-shot accuracy score across different architectures, seeds, and datasets. The proposed aggregation function obtains the best accuracy score. It is essential to highlight that the Concat aggregation method is not directly comparable the others because increases the dimensionality of the space linearly with the number of subspaces. The standard deviation is high because it accounts for the average across all possible image datasets.
Encoder
Aggregation Function Accuracy
CViT-B/32
Concat
0.68 +'2 0.13
MLP+SelfAttentio
0.66 + 0.13
MLP+Sum
0.71 + 0.11
SelfAttention
0.54 + 0.18
RViT-B/16
Concat
0.72 + 0.18
MLP+Scl[Attention
0.72 0.18
MLP+Sum
0.74
SelfAttention
0.61 0.24
RexNet
Concat
0.58 + 0.20
MI.P+SelfAttention
0 0.21
MLP+Sum
0.61 + 0.20
SelfAttention
0.40 0.22
ViT-B/16
Concat
0.72 + 0.19
MLLFteefAAttention
0.70
MLP+Sum
0.74 0.19
SelfAttention
0.60 + 0.24
ViT-S/16
Concat
0.69 + 0.18
MLP+4elfAttentton
0.67 0.19
MLP+Sum
0.71 0.18
SelfAttention
0.57 + 0.22
24

---

Published as a conference paper at ICLR 2024
Table 13: Text Classification Ablation the Aggregation Functions. Zero-shot accuracy score across different architectures, seeds, and datasets. The proposed aggregation function obtains the best accuracy score. It is essential highlight that the Concat aggregation method is directly comparable to the others because it increases the dimensionality of the space linearly with the number of subspaces. The standard deviation is high because it accounts for the average across all possible text datasets.
Encoder
Aggregation Function Accuracy
ALBERT
Concat
0.38 = 0,22
MLP+SelfAttention
0.35 + 0.21
MLP+Sum
0.38 + 0.21
SelfAttention
0.21 + 0.17
BERT-C
Concat
0.41 + 0.17
MLP+SelfAAtention
0.36 + 0.17
MLP+Sum
0.42 + 0.18
SelfAttention
0.20 + 0.16
BERT-U
Concat
0.36 + 0.17
Ml.PeSelfAttention
0.31 + 0.15
MI.++Sum
0.37 + 0.17
SellAttention
0.19 + 0.15
CViT-B/32
Concat
0.23 + 0.18
MLP+SelfAAttentio
0.22 0.18
MLP+Sum
0.26 + 0.21
SelfAttention
0.15 0.13
ELECTRA
Concat
0.30 + 0.18
Ml.P4SelfAttentio
0.27 + 0.16
MLP+Sum
0.30 0.18
SelfAttention
0
ROBERTA
Concat
0.41 + 0.15
MLP+SelfAttention
0.36 +: 0.14
MLP+Sum
0.44 0.15
SelfAttention
0.23 + 0.16
XLM-R
Concat
0.35 +: 0.17
ML.P+SelfAttenton
0.29 + 0.17
MLP+Sum
0.35 + 0.17
SelfAttention
0.18 - 0.16
25

---

Published as a conference paper at ICLR 2024
Table 14: Graph Classification Ablation Study. Zero-shot accuracy score across different ar- chitectures and seeds. We analyzed the effect of selecting only specific spaces and including all available spaces. Furthermore, we assessed all the possible aggregation functions and calculated the stitching index validate stitching performance. Our proposed aggregation function achieved the highest accuracy score. It is important to note that the Concat aggregation method cannot directly compared to others as it linearly increases the dimensionality of the space with the number of spaces.
Aggregation
Projection
Accuracy t
Stitching index
Absolute
0.14 + 0.04
0.18
Cosine
0.53 + 0.06
0.71
Euclidean
0.27 + 0.06
0.58
L1
0.26 + 0.06
0.58
Loc
0.12 + 0.03
1.00
Concat
Euclidan
0.69 + 0.04
0.90
Cosine, L1
0.69 + 0.04
0.90
Cosine,
0.65 - 0.05
0.87
Euclidean,
0.40 + 0.07
0.72
Euclidean,
0.30 + 0.09
0.65
Li,Lo
0.32 + 0.09
0.67
Cosine, Euclidean, I1, Loo
0.75 + 0.02
0.97
SelfAttention
Cosine,Euclidens
0.72 + 0.04
0.96
Losine,Ly
0.72 + 0.04
0.96
Cosine,
0.76
1.00
Euclidean,
0.68 + 0.03
0.92
Euclidean,
0.69 + 0.04
0.93
L,
0.68 + 0.04
0.91
Cosine, Euclidean, L1, Loo
0.63 + 0.13
0.85
MLP4SelfAttention
Eusine,En
0.76 0.01
1.00
Cosine.L1
0.76
1.00
Cosine.
0.76
1.00
Euclidean, J_
0.68 0.06
0.92
Euclidean,
0.71 0.03
0.95
LI
0.69 + 0.03
0.93
Cosine, Euclidean, L1, Loc
0.76 + 0.02
0.99
MLP+Sum
CusinenE
0.78 + 0.02
1.00
L
0.78 + 0.01
1.00
Cosine, Euclidean, L1
0.77 = 0.01
1.00
0.72 0.03
0.97
Euclidean,
0.68 + 0.04
0.94
L L
0.67 + 0.01
0.94
Cosine, Euclidean, L1, Loo
0.77 + 0.01
1.00
26

---

Published as a conference paper at ICLR 2024
Table 15: Zero-Shot Stitching Results on CIFAR-14 10. The table shows the mean and standard deviation of the test accuracy for the different projection methods sorted by maximum difference between projections, reported in the first column.
Encoder
Decoder
MaxDiff
Cosine
Fuclidean
Ta
T
RexNet
RViT-B/16
0.11 + 0.01
0.80 + 0.01
0.74 + 0.02
0.69 + 0.02
0.46 0.02
ViT-B/16
0.11 0.00
0.79 0.01
0.69 + 0.02
0.69 0.01
0.51 0.03
CViT-B/32
0.09:0001
0.79 0.01
0.73 0.01
0.70
0.48 0.03
ViT-S/16
0.09 1 0.02
0.79
0.72 0.02
0.70
0.45 0.02
ViT-B/16
RexNet
0.02 = 0.01
0.94 0.01
0.95 0.00
0.96 + 0.00
0.63 = 0.01
CViT-B/32
ViT-B/16
0.02 0.01
0.87 0.01
0.87 0.01
0.87 + 0.01
0.56 + 0.00
RViT-B/16
0.02 + 0.00
0.84 + 0.01
0.84 0.01
0.81 0.01
0.54 0.02
RViT-B/16
RexNet
0.01 0.00
0.93 + 0.00
0.94 + 0.00
0.94 + 0.00
0.70 + 0.02
CViT-B/32
RexNet
0.01 + 0.00
0.87 1 0.00
0.88
0.00
0.87 + 0.00
0.47 + 0.01
ViT-S/16
0.01 + 0.01
0.89 1 0.01
0.89 + 0.01
0.89 + 0.01
0.52 1 0.01
ViT-8/16
RViT-B/16
0.01 1 0.00
0.93 T 0.00
0.93
0.00
0.92 1 0.00
0.73 1 0.00
RViT-B/16
ViT-B/16
0.01 0.00
0.95 T. 0.00
0.95
0.00
0.95 0.00
0.74 T 0.01
VII-S/16
ViT-B/16
0.01 = 0.00
0.94 + 0.00
0.93
0.00
0.94 + 0.00
0.69 = 0.01
RVIT-B/16
CViT-B/32
0.01 = 0.01
0.93 + 0.00
0.94 = 0.00
0.94 + 0.00
0.78 = 0.03
ViI-S/16
CViT-B/32
0.01 = 0.00
0.93 + 0.00
0.93 0.00
0.92 + 0.00
0.66 + 0.03
RexNet
0.01 = 0.00
0.92 0.00
0.91 = 0.00
0.92 + 0.00
0.66 = 0.02
ViT-B/16
RViT-B/16
0.01 0.00
0.90
0.96 0.00
0.97 + 0.00
0.76 0.00
CViT-B/32
0.01 = 0.00
0.95
0.96 + 0.00
0.96 + 0.00
0.67 + 0.02
RVIT-B/16
ViT-S/16
0.00 = 0.00
0.95 0.00
0.95 = 0.00
0.95 + 0.00
0.79 = 0.02
VIT-B/16
ViT-S/16
0.00 0.00
0.97 + 0.00
0.97 0.00
0.97 + 0.00
0.73 = 0.03
Table 16: ZSro-Shot Stitching Results on CIFAR-100. The table shows the mean and standard deviation of the test accuracy for the different projection methods sorted by maximum difference between projections, reported in the first column.
Encoder
Decoder
MaxDiff
Cosine
Euclidcan
L1
Loo
RexNet
ViT-B/16
0.21 + 0.01
0.50 + 0.01
0.40 0.02
0.25 + 0.02
0.21 + 0.02
ViT-S/16
RexNet
0.23 + 0.01
0.73 + 0.00
0.63 0.01
0.50 + 0.01
0.31 + 0.00
RexNet
RViT-B/16
0.20 + 0.02
0.52 + 0.01
0.42 0.02
0.31 + 0.02
0.22 + 0.01
ViT-S/16
CViT-B/32
0.17 1 0.03
0.68 1 0.00
0.61 1 0.01
0.51 1 0.03
0.30 1 0.00
ViT-B/16
0.14 1 0.00
0.77 T 0.00
0.64 0.00
0.71 0.01
0.33 1 0.01
RViT-B/16
0.13 0.01
0.75 T 0.00
0.67 0.01
0.62 0.01
0.39 1 0.00
RexNet
ViT-8/16
0.12 1 0.01
0.47 - 0.01
0.40 0.02
0.34 1. 0.01
0.17 0.00
CViT-B/32
0.11 + 0.01
0.50 1 0.01
0.52 0.02
0.41 4: 0.02
0.17 + 0.02
CVIT-B/32
RViT-B/16
0.07 = 0.03
0.48 + 0.04
0.52 0.02
0.49 + 0.03
0.32 = 0.01
RVIT-B/16
RexNet
0.06 = 0.00
0.79 1 0.00
0.77 + 0.00
0.72 + 0.00
0.33 = 0.00
CVIT-B/32
ViT-S/16
0.05 + 0.01
0.53 1 0.02
0.54 0.02
0.57 + 0.02
0.22 + 0.01
ViT-B/16
0.05 + 0.02
0.53 0.02
0.52 0.02
0.56 1 0.01
0.26 1 0.01
ViT-B/16
RexNet
0.04 0.01
0.72 4.00
0.72
0.68 0.00
0.40 0.02
CViT-B/32
0.04 +
0.70
0.71
0.73 0.01
0.39 0.01
ViT-S/16
0.03 = 0.00
0.80 0.00
0.83 0.00
0.84 0.00
0.44 0.01
CVIT-B/32
RexNet
0.03 = 0.02
0.52 0.01
0.55 - 0.02
0.53 + 0.00
0.27 = 0.01
RViT-B/16
ViT-S/16
0.03 + 0.01
0.79 + 0.01
0.82 0.00
0.81 + 0.01
0.34 + 0.00
ViT-B/16
RViT-B/16
0.02 + 0.01
0.78 0.01
0.80 0.01
0.79 + 0.00
0.44 0.01
RViT-B/16
ViT-B/16
0.01 0.00
0.82+
0.81 + 0.00
0.82 + 0.00
0.26 + 0.01
CViT-B/32
0.01 + 0.01
0.75 + 0.00
0.75 + 0.01
0.75 + 0.00
0.30+0000
27

---

Published as a conference paper at ICLR 2024
Table 17: Zero-Shot Stitching Results on Fashion MNIST. The table shows the mean and standard deviation of the test accuracy for the different projection methods sorted by maximum difference between projections, reported the first column.
Encoder
Decoder
Max
Cosine
Euclidean
L
Loo
CViI-B/32
RexNet
0.05 = 0.00
0.67 + 0.00
0.72 = 0.01
0.72 + 0.00
0.54 = 0.00
RVIT-B/16
RexNet
0.04 0.01
0.76 + 0.01
0.79 0.01
0.80
0.59 0.01
ViT-B/16 6
CViT-8/38
0.03 + 0.00
0.76 + 0.01
0.78 + 0.00
0.79 + 0.00
0.50 + 0.01
CViT-B/32
RViT-B/16
0.03 + 0.02
0.68 + 0.01
0.72 + 0.01
0.71 + 0.01
0.56 + 0.00
RViT-B/16
ViT-B/16
0.03 + 0.01
0.78 + 0.01
0.79 0.01
0.81 + 0.00
0.61 + 0.01
ViT-S/16
RViT-8/16
0.03 1 0.00
0.76 T 0.00
0.77 0.00
0.78 1 0.00
0.52 T 0.01
RViT-B/16
CViT-B/32
0.03 1 0.01
0.72 11: 0.01
0.72 +: 0.01
0.74 4: 0.01
0.57 11: 0.01
ViT-B/16
RexNet
0.03 + 0.01
0.79 + 0.00
0.81 0.00
0.81 + 0.00
0.48 + 0.01
RexNet
RViT-B/16
0.03 = 0.01
0.74 + 0.01
0.77 0.00
0.76 + 0.00
0.53 0.02
CVII-B/32
ViT-S/16
0.03 :: 0.01
0.66 1 0.02
0.65 0.01
0.65 + 0.01
0.56 + 0.00
RexNet
ViT-S/16
0.02 1 0.01
0.74 1 0.00
0.77 1: 0.00
0.76 1 0.01
0.64 1 0.01
ViT-8/16
RexNet
0.02 = 0.01
0.78 + 0.01
0.78 0.01
0.78 + 0.01
0.51 = 0.01
VIT-B/16
ViT-S/16
0.02 0.01
0.79 + 0.01
0.80 0.01
0.80 + 0.01
0.57 0.01
CVIT-B/32
ViT-B/16
0.02 = 0.01
0.70 + 0.01
0.71 = 0.01
0.72 = 0.00
0.55 = 0.01
RexNet
C-1/3B/32
0.01 + 0.01
0.72 + 0.01
0.74 0.01
0.73 + 0.01
0.63 + 0.00
RViT-B/16
ViT-S/16
0.01 0.01
0.79 + 0.00
0.79 + 0.00
0.80 + 0.01
0.63 + 0.00
ViT-3/16
ViT-B/16
0.01 0.00
0.79 + 0.00
0.80 0.00
0.80 + 0.01
0.54 0.01
RexNet
ViT-B/16
0.01 1 0.01
0.76 1 0.00
0.77 1 0.01
0.77 1. 0.00
0.60 1 0.02
ViT-8/16
CViT-B/32
0.01 + 0.00
0.75 4 0.00
0.75 0.00
0.75 + 0.01
0.55 = 0.02
ViT-B/16
RViT-B/16
0.01 0.00
0.84 + 0.00
0.84 0.00
0.84 0.00
0.52 0.00
28

---

Published as a conference paper at ICLR 2024
Table 18: Zero-Shot Stitching Results on MNIST. The table shows the mean and standard deviation of the test accuracy for the different projection methods sorted by maximum difference between projections, reported in the first column.
Encoder
Decoder
MaxDiff
Cosine
Euclidean
L
Loo
CViT-B/32
ViT-B/16
0.09 = 0.01
0.57 + 0.00
0.65 = 0.01
0.66 + 0.01
0.55 0.01
ViT-8/16
RexNet
0.08 + 0.01
0.62 + 0.01
0.69 0.01
0.71 + 0.00
0.48 + 0.01
ViT-B/16
RexNet
0.08 0.01
0.56 + 0.00
0.62 0.01
0.64 + 0.00
0.43 0.01
CViT-B/38
0.07 + 0.02
0.62 + 0.02
0.69 + 0.01
0.69 + 0.01
0.47 + 0.00
RexNet
ViT-S/16
0.07 + 0.01
0.71 + 0.00
0.65 0.01
0.64 + 0.01
0.51 + 0.00
ViT-B/16
ViT-S/16
0.07 1 0.01
0.66 T 0.01
0.71 0.00
0.73 1 0.00
0.42 TL 0.00
CViT-B/32
RViT-B/16
0.05 1 0.03
0.57 +1 0.01
0.62 0.02
0.60 4: 0.02
0.54 +:: 0.01
ViT-B/16
RViT-B/16
0.05 + 0.03
0.53 + 0.02
0.58 0.02
0.56 + 0.02
0.33 + 0.00
ViI-S/16
RVII-B/16
0.05 0.02
0.69 + 0.01
0.73 0.01
0.74 +
0.49 0.02
CVIT-B/32
ViT-S/16
0.04 = 0.01
0.60 + 0.01
0.65
0.01
0.65 + 0.01
0.57 = 0.01
ViT-S/16
CViT-B/32
0.04 1 0.01
0.71 1 0.01
0.69
0.01
0.67 1 0.00
0.49 11: 0.01
RVIT-B/16
RexNet
0.04 = 0.02
0.63 + 0.00
0.66 0.01
0.67 + 0.01
0.65 - 0.01
RexNet
CViT-B/32
0.04 0.01
0.71 + 0.01
0.74 0.01
0.74 + 0.00
0.48 0.00
RVII-B/16
ViT-S/16
0.03 = 0.01
0.74 0.01
0.77 0.00
0.77 + 0.00
0.63 = 0.01
ViT-B/16
0.03 - 0.01
0.01
0.71 0.00
0.72 +
0.53 0.00
CViT-B/32
RexNet
0.02+0.00
0.71 + 0.00
4.73+
0.73 + 0.02
0.61 + 0.00
RViT-B/16
CViT-B/32
0.02
0.69 0.00
0.68 + 0.01
0.68 + 0.01
0.61 + 0.01
RexNet
RViT-B/16
0.02 + 0.01
0.67 + 0.01
0.67 0.01
0.66 + 0.01
0.47 = 0.01
ViT-8/16
ViT-B/16
0.01 1 0.01
0.68 1 0.01
0.69 0.00
0.68 1 0.00
0.46 1 0.02
RexNet
ViT-B/16
0.01 + 0.01
0.73 + 0.01
0.72 4: 0.01
0.72 4: 0.00
0.46 0.01
29

---

Published as a conference paper at ICLR 2024
Table 19: Zero-Shot Stitching Results DRpedia. The table shows the mean and standard deviation of the test accuracy for the different projection methods sorted by maximum difference between projections, reported in the first column.
Encoder
Decoder
Maxfiff
Cosine
Euclidean
L1
Txo
XLM-R
BERT-C
0.25 0.01
1 0.00
0.19 0.01
0.39
0.01
0.09
0.01
ALBERT
0.22 1 0.03
0.15 1 0.03
0.17 1 0.02
0.36 1 0.02
0.10 1 0.00
ELECTRA
0.22 1 0.02
0.08 1 0.00
0.13 1 0.01
0.30 1 0.02
0.06 11 0.01
BERT-U
RoBERTa
0.20 + 0.01
0.53 4 0.01
0.36 4: 0.01
0.33 4 0.01
0.14 0.01
XLM-R
RoBERTa
0.19 = 0.02
0.16 + 0.02
0.19 0.01
0.35 + 0.01
0.08 0.01
BERT-U
ELECTRA
0.19 = 0.02
0.50 + 0.01
0.31
0.01
0.35 + 0.00
0.14 0.01
XLM-R
BERI-U
0.16 + 0.01
0.12 1 0.00
0.14 0.01
0.27 +: 0.01
0.10 - 0.00
BERT-U
BERT-C
0.15
0.57 + 0.00
0.43
0.42 0.01
0.13 0.01
CViT-B/32
0.03
+
0.30
0.26 0.01
0.11 0.00
XLM-R
0.05
0.50
0.35
0.36+0.00
0.08 0.01
RoBERTa
BERT-C
0.02
0.40 + 0.03
0.30 + 0.02
0.28 + 0.01
0.13 0.01
BERI-C
ELECTRA
0.12 = 0.02
0.49 + 0.02
0.46 = 0.01
0.58 + 0.02
0.11 0.01
XLM-R
CViT-B/32
0.12 + 0.02
0.12 + 0.01
0.12 + 0.02
0.23 + 0.00
0.08 + 0.00
RoBERTa
BERT-U
0.11 + 0.05
0.31 + 0.06
0.27 0.06
0.31 + 0.06
0.15 + 0.01
ELECTRA
ALBERT
0.11 + 0.01
0.25 + 0.01
0.27 + 0.02
0.36 + 0.02
0.11 + 0.00
BERT-U
ALBERT
0.11 0.04
0.53 T. 0.01
0.42 0.03
0.43 0.02
0.11 T 0.00
ELECTRA
BERT-C
0.11 + 0.01
0.31 + 0.01
0.27 0.01
0.38 +: 0.03
0.10 + 0.00
ALBERT
XIM-R
0.09 + 0.01
0.40 + 0.00
0.42 + 0.00
0.49 + 0.01
0.14 + 0.00
ROBERTA
ELECTRA
0.09 = 0.02
0.36 1 0.03
0.29 0.01
0.37 + 0.02
0.13 + 0.01
BERT-C
XLM-R
0.09 0.05
0.49 1 0.03
0.57
0.04
0.51 0.03
0.12 0.01
RoBERTa
0.08 1 0.05
0.52 1 0.03
0.56 1 0.02
0.60 1 0.02
0.14 0.00
ELECTRA
BERT-U
0.08 = 0.03
0.24
0.21 0.01
0.29 + 0.02
0.14 0.00
RoBERTA
0.08 = 0.01
0.35 0.02
0.34
0.41 + 0.01
0.10 0.01
RoBERTA
CViI-B/32
0.07 0.02
+ 0.02
0.21 0.01
0.24 0.06
0.11 0.01
BERT-C
ALBERT
0.07 = 0.03
0.43 + 0.04
0.43 0.05
0.42
0.13 = 0.00
BERI-U
0.06 0.06
0.04
0.53 0.04
0.54 0.04
0,21 = 0.01
ELECTRA
CVi T-2/32
0.06 0.01
0.22 + 0.01
0.19
0.25 + 0.01
0.10 + 0.01
BERT-C
CViT-B/32
0.06 + 0.05
0.33 0.02
0.34 0.03
0.35 + 0.07
0.09 + 0.01
RoBERTa
ALBERT
0.06 = 0.00
0.32 + 0.04
0.31 0.01
0.36 + 0.01
0.14 0.00
ELECTRA
XLM-R
0.05 1 0.01
0.25 1 0.00
0.25 0.00
0.30 1 0.01
0.13 T 0.01
RoBERTa
XLM-R
0.05 + 0.01
0.21 + 0.02
0.21 +: 0.01
0.26 + 0.00
0.12 + 0.01
ALBERT
ELECTRA
0.04 + 0.01
0.50 + 0.00
0.50
0.01
0.53 + 0.01
0.17 + 0.01
CVIT-B/32
ELECTRA
0.04 = 0.03 : 0.00
0.20 1 0.00
0,23 = 0.00
0.21 0.00
0.11 0.01
RoBERTa
0.21 1 0.00
0.24
0.24 1 0.01
0.15 0.01
BERT-U
0.03 - 0.01
0.19 1 0.01
0.22 0.01
0.21 + 0.00
0.12 0.00
ALBERT
BERT-U
0.03 1 0.02
0.54 1 0.01
0.55
0.00
0.57 1 0.01
0.18 1 0.00
CV1T-B/32
ALBERT
0.03 - 0.01
0.17 + 0.01
0.19 0.00
0.20 + 0.01
0.09 0.01
ALBERT
RoBERTa
0.02 0.02
0.50
0.50 0.00
0.52 0.01
0.18 - 0.02
BERI-C
0.02 0.02
0.51 1.11+000
0.53
0.01
0.18 0.00
CViT-B/32
0.02 0.00
0.00
0.43 0.00
0.15 0.00
CVII-B/32
BERI-C
0.02
0.20
0.21
0.22
0.14
XLM-R
0.000
0.22 0.00
0.00
0.24 0.00
0.13 0.00
30

---

Published as a conference paper at ICLR 2024
Table 20: Zero-Shot Stitching Results on TREC. The table shows the mean and standard deviation of the test accuracy for the different projection methods sorted by maximum difference between projections, reported in the first column.
Encoder
Decoder
MaxDiff
Cosine
Euclidcan
L
Loo
XLM-R
RoBERTa
0.32 + 0.11
0.40 + 0.10
0.46 0.04
0.69 + 0.06
0.33 : 0.09
BERT-U
0.30 + 0.13
0.36 + 0.18
0.53 + 0.04
0.61 4 0.01
0.31 + 0.04
BERT-C
XIM-R
0.30 + 0.14
0.34 + 0.08
0.55
0.03
0.64 + 0.06
0.17 + 0.03
XLM-R
BERT-C
0.26 + 0.08
0.42 + 0.09
0.56 0.02
0.68 + 0.02
0.26 + 0.11
CVIT-B/32
BERT-C
0.25 1 0.16
0.24 1 0.12
0.45 0.03
0.45 + 0.12
0.33 = 0.01
RoBERTa
XLM-R
0.25 1 0.03
0.37 T 0.03
0.36 1 0.01
0.60 1 0.02
0.27 T 0.08
XLM-R
CViT-B/32
0.25 1 0.12
0.30 T 0.06
0.47 T 0.04
0.55 1 0.07
0.34 1 0.03
ALBERT
XLM-R
0.24 = 0.12
0.37 + 0.09
0.50 0.02
0.61 + 0.03
0.28 0.04
BERT-U
XLM-R
0.24 = 0.08
0.34 0.08
0.53
0.03
0.58 + 0.00
0.27 = 0.10
ELECTRA
0.23 = 0.09
+ 0.06
0.57 0.06
0.43 + 0.07
0.32 = 0.01
RoBERTa
ALBERT
0.23 0.04
0.42 - 0.01
0.65 # 0.03
0.62 + 0.03
0.14 0.04
ELECTRA
ALBERT
0.21 0.11
0.51 + 0.04
0.48 + 0.11
0.63 + 0.04
0.16 + 0.04
CViT-B/32
RoBERTa
0.20 0.02
0.45 + 0.02
0.54
0.65 - 0.01
0.32 0.09
ELECTRA
0.20
0.33 + 0.07
0.52 0.01
0.39 + 0.01
0.31 - 0.01
BERT-C
ELECTRA
0.20 + 0.06
0.40 + 0.04
0.60 0.02
0.53 0.03
0.20 + 0.02
ELECTRA
ROBERTa
0.20 + 0.00
0.52 + 0.02
0.33 0.01
0.45 + 0.05
0.32 + 0.07
XLM-R
ALBERT
0.19 + 0.12
0.46 + 0.08
0.58 + 0.07
0.64 + 0.07
0.21 + 0.01
RoBERTa
BERT-U
0.19 + 0.02
0.44 1 0.02
0.60
0.02
0.62 + 0.03
0.21 + 0.01
XLM-R
ELECTRA
0.19 0.07
0.37 T 0.06
0.56 0.02
0.45 0.02
0.38 1 0.03
CVIT-B/32
ALBERT
0.18 1 0.08
0.45 T 0.10
0.60
0.04
0.48 1 0.03
0.11 T 0.01
XIM-R
0.17 1 0.06
0.36 1 0.06
0.51
0.04
0.53 1. 0.03
0.34 1 0.01
ELECTRA
BERI-U
0.16 = 0.05
0.52 + 0.02
0.44 0.10
0.59 + 0.05
0.20 0.04
BERI-U
RoBERTa
0.14 = 0.02
0.56 + 0.00
0.64 = 0.03
0.70 + 0.02
0.29 = 0.08
CViT-B/32
0.14 0.06
0.46 + 0.03
0.52 0.01
0.60 + 0.04
0.25 = 0.08
BERT-C
BERI-U
0.14:10 0.18
0.53 1 0.13
0.61 + 0.03
0.65 + 0.06
0.23 : 0.04
RoBERTa
CViT-B/32
0.13 = 0.05
0.42 + 0.04
0.51 0.07
0.54 + 0.08
0.27 + 0.10
ALBERT
CViT-B/32
0.12 0.04
0.51 + 0.08
0.47 0.02
0.56 + 0.02
0.30 - 0.02
BERT-C
ALBERT
0.12 = 0.10
0.53 + 0.10
0.54 0.06
0.59 + 0.08
0.23 = 0.03
BERT-U
BERT-C
0.11 + 0.02
0.41 + 0.00
0.48 0.01
0.53 + 0.02
0.35 + 0.02
BRRT-C
CViT-B/32
0.11 0.09
0.43 + 0.07
0.53 0.01
0.55 + 0.03
0.18 + 0.06
ALBERT
RoBERTa
0.11 + 0.06
0.54 + 0.03
0.60
0.03
0.65 + 0.02
0.29 + 0.06
ELECTRA
CViT-B/32
0.10 1 0.08
0.48 1 0.05
0.50 1 0.04
0.57 1 0.05
0.33 1 0.05
XLM-R
0.10 1 0.08
0.48 1 0.08
0.55 1 0.02
0.57 1 0.02
0.30 1 0.07
BERT-C
ROBERTa
0.10 1 0.09
0.56 1 0.11
0.59 1 0.04
0.60 1. 0.02
0.24 1 0.01
BERT-U
ALBERT
0.09 = 0.05
0.63 + 0.07
0.64 # 0.05
0.64 + 0.05
0.22 + 0.06
ALBERT
BERI-U
0.08 0.05
0.54 + 0.04
0.57 0.03
0.60 + 0.04
0.47 = 0.01
CVIT-B/32
BERI-U
0.08 + 0.01
0.51 + 0.00
0.58 0.02
0.53 + 0.05
0.12 = 0.00
RoBERTa
BERT-C
0.07 + 0.03
0.52 + 0.03
0.59 + 0.00
0.57 + 0.01
0.18 + 0.03
ELECTRA
BERT-C
0.07 = 0.02
0.49 + 0.02
0.56 0.02
0.56 + 0.01
0.17 0.05
ALBERT
BERI-C
0.07 0.01
0.43 0.00
0.49 0.02
0.49 4.0 0.01
0.32 0.01
ELECTRA
0.06 = 0.01
0.57 + 0.02
0.61 = 0.02
0.62 + 0.01
0.29 = 0.01
RoBERTa
ELECTRA
0.02 + 0.01
0.41 + 0.00
0.40 + 0.01
0.42 + 0.01
0.16 0.01
31

---

Published as a conference paper at ICLR 2024
Table 21: Zero-Shot Stitching Results on 4nwws (Text). The table shows the mean and standard deviation of the test accuracy for the different projection methods sorted by maximum difference between projections, reported the first column.
Encoder
Decoder
MaxDiff
Cosine
Faclidean
T
XLM-R
ALBERT
0,18 + 0.01
0.16 1 0.01
0.29 + 0.02
0.34 + 0.01
0.07 + 0.00
BERI-C
0.17 + 0.01
0.22 1 0.01
0.33 +: 0.02
0.39 +: 0.02
0.07 : 0.00
BERI-U
0.15 + 0.00
0.21 1 0.01
0.29 i 0.01
0.36 0.01
0.08 - 0.00
ELECTRA
0.15 0.01
0.18 1 0.01
0.26 0.01
0.33 1 0.01
0.06 1 0.00
RoBERTa
XLM-R
0.11 1 0.00
0.53 T 0.00
0.55 1 0.00
0.64 0.00
0.08 T 0.00
XLM-R
ROBERTa
0.09 = 0.02
0.37 + 0.01
0.37 = 0.02
0.45 + 0.02
0.06 = 0.01
ROBERTA
ELECTRA
0.09: 0.01
0.38 + 0.01
0.35 0.00
0.44 0.00
0.11 = 0.01
ALBERT
0.08 = 0.01
0.38 + 0.02
0.39
0.46 0.01
0.08 = 0.00
BERI-U
0.08 = 0.01
0.50
0.44
0.61
0.08 = 0.01
ELECTRA
ROBERTa
0.08 = 0.01
0.11
0.19
0.13
0.06 = 0.01
BERI-U
0.07 177:0000 0.04
0.05
0.10
0.11
0.09 = 0.01
BERT-C
0.06 106::0000 0.02
0.01
0.13
0.11 0.01
0.09 + 0.01
BERT-C
BERT-U
0.04
0.21 + 0.02
0.23
0.23 + 0.04
0.13 0.01
ELECTRA
XLM-R
0.04
0.09 + 0.00
0.12 0.01
0.11 + 0.01
0.07 = 0.00
BERT-C
ELECTRA
0.03 0.01
0.18 + 0.01
0.20 0.01
0.21 0.01
0.10 + 0.01
ROBERTa
BERT-C
0.03 + 0.00
0.59 + 0.01
0.57 + 0.00
0.60 + 0.01
0.08 + 0.00
BERT-C
ROBERTA
0.03+ 0.01
0.26 + 0.00
0.25 + 0.01
0.23 + 0.01
0.14 + 0.01
BERT-U
ELECTRA
0.03 1 0.02
0.14 1 0.02
0.16 + 0.01
0.17 + 0.01
0.07 1 0.01
ALBERT
0.03 1 0.01
0.15 1 0.01
0.18 1 0.00
0.18 1 0.01
0.07 1 0.00
BERT-C
CViT-B/32
0.03 1 0.01
0.05 T 0.02
0.04 0.02
0.04 1 0.02
0.04 1 0.01
CViT-B/32
RoBERTa
0.03 0.01
0.04 1. 0.01
0.04 0.02
0.05 0.00
0.05 11 0.00
RoBERTA
CVIT-B/32
0.03 = 0.03
0.06 + 0.02
0.04 = 0.01
0.04 + 0.01
0.04 0.01
ELECTRA
CVIT-B/32
0.02 = 0.02
0.05 + 0.02
0.04 + 0.02
0.04 + 0.02
0.05 = 0.01
BERI-U
XLM-R
0.02 + 0.01
0.18 1 0.01
0.19 0.01
0.18 + 0.02
0.07 = 0.00
ALBERT
CViT-B/32
0.02 0.00
0.05 + 0.02
0.04 0.01
0.04 + 0.01
0.05 0.01
BERT-U
ROBERTA
0.02 = 0.00
0.20 0.01
0.17 0.01
0.07 = 0.01
BERT-C
0.02 = 0.02
0.20 0.01
0.22 0.01
0.22 + 0.01
0.06 = 0.00
XLM-R
CVIT-B/32
0.02 = 0.01
0.06 + 0.01
0.05 - 0.01
0.04 0.01
0.05 = 0.01
ELECTRA
ALBERT
0.02 + 0.02
0.11 + 0.02
0.13 + 0.01
0.13 + 0.01
0.06 + 0.00
BERT-U
CViT-B/32
0.01 0.01
0.04 + 0.01
0.05 + 0.01
0.05 + 0.01
0.04 + 0.00
ALBERT
RoBERTa
0.01 + 0.00
0.08 + 0.00
0.09
0.00
0.07 + 0.01
0.08 + 0.00
BERT-C
XLM-R
0.01 1 0.01
0.29 0.00
0.29 1 0.00
0.28 1 0.01
0.14 T 0.00
ALBERT
0.01 1 0.01
0.28 1 0.01
0.29 1 0.00
0.29 1 0.01
0.10 T 0.01
ALBERT
BERT-C
0.01 0.01
0.08 0.00
0.09 0.01
0.09 0.01
0.06 11 0.00
ELECTRA
0.01 0.01
0.11 1 0.00
0.12
0.12 0.01
0.08 1 0.01
BERT-U
0.01 0.01
0.11 + 0.01
0.11 0.00
0.11 4: 0.00
0.07 + 0.00
CVIT-B/32
XLM-R
0.01 #.01
0.04 + 0.01
0.05
0.00
0.05 + 0.00
0.05 = 0.00
ALBERT
XLM-R
0.01 = 0.01
0.09 1 0.00
0.10 + 0.00
0.09 + 0.01
0.07 = 0.00
CViT-B/32
ELECTRA
0.00 + 0.00
0.05 0.00
1.000
0.05 + 0.00
0.05 1 0.00
BERT-U
0.00 0.00
0.05
1..00
0.00
0.050000
0.05 1 0.00
BERT-C
0.00
0.00
0.00
0.03 0.0 0.00
ALBERT
0.00
0.00
0.05
0.05 + 0.00
0.05
32

---

Published as a conference paper at ICLR 2024
Table 22: Zero-Shot Stitching Results on ImageNet Zero-shot accuracy score across different architectures and seeds. The standard deviation is provided for each accuracy score. The proposed methodology utilizing the Aggregation by sum consistently outperforms the other results.
Encoder
Aggregation Function
Accuracy t
RViT-B/14
Cosine
0.50 0.01
Fuclidean
0.52 0.01
L1
0,45 + 0.01
Loo
0.01 0.00
Cosine, Euclidean, L1, Los
0.67 0.01
ViT-B/16
Cosine
0.27 0.03
Euclidean
0.25 + 0.02
L1
0.43 + 0.02
Loo
0.02 = 0.01
Cosine, Euclidean, L1, Loo
0.56 0.02
ViT-S/16
Cosine
0.34 0.02
Euclidean
0.33 + 0.03
L11
0.35 0.02
Loc
0.02 = 0.01
Cosine, Fuclidean, L1, Loz0
0.51 + 0.02
Table 23: Zero-Shot Stitching Reconstruction Performance. Zero-shot L₁ and MSE score for the reconstruction task across different seeds on CIFAR-100 using AEs. The proposed methodology utilizing the Aggregation by sum consistently matches outperforms the other results.
Aggregation
Projection
L
MSE
MuP+SSm
Cosine, Euclidean, L,, Loo
0.01 + 0.00
0.06 + 0.00
MLP+Sum
0.02 = 0.00
0.10 0.01
Cosine
0.06 = 0.02
0.20 0.03
Fuclidean
0.01 + 0.00
0.06 0.00
L
0.01 + 0.00
0.09 0.00
Loo
0.03 0.01
0.14 0.01
Absolute
0.26 0.06
0.43 - 0.06
33

---

Published as a conference paper at ICLR 2024
Table 24: End-to-end Performance Results on CIFAR-10, The classifier head a simple Linear layer.
Model
Aggregation
Projection
CViT-B/32
Ahsolute
0.92 + 0.05
Cosine
0.90
Euclidean
- 0.90 0.06
L1
0.90 + 0.06
Loo
0.79 + 0.13
MLP+Sum
Cosine, Euclidean, L1. Loo
0.91 + 0.05
SelfAttention
Cosine, Euclidean, Lr, Loo
0.90 + 0.06
RViT-B/16
Absolute
0.85 0.19
Cosine
0.83 0.18
Euclidean
0.83 - 0.118
L1
0.83 0.19
T.oo
0.79 + 0.20
MLP+Sum
Cosine, Enclidean, L1, Too
0.84 + 0.19
SelfAttention
Cosine, Euclidean, L1. Loo
0.82 + 0.20
RexNet
Absolute
0.80 0.21
Cosine
0.76
Euclidean
0.76
L1
0.77 0.20
Loo
0.72 + 0.20
MLP+Sum
Cosine, Fuclidean, T., Too
0.79 + 0.21
SelfAttention
Cosine, Faclidean, T1, Too
0.76 + 0.21
ViT-B/16
Absolute
0.86 + 0.18
Cosine
0.84 0.18
Euclidean
0.84 1 0.18
0.85 0.18
L
0.73 1 0.19
MLP+Sum
Cosine, Euclidean, L1, Loo
0.86 0.18
SelfAttention
Cosine, Euclidean, L1, Loo
0.81 0.19
ViT-S/16
Ahsolute
0.83 0.19
Cosine
= 0.82 0.19
Euclidean
0.82 0.18
L1
0.82 1 0.19
Loo
0.74 0.19
MLP+Sum
Cosine, Euclidean, L1. Loo
0.83 1 0.18
SelfAttention
Cosine, Euclidean, L1, Loo
0.82 0.20
34

---

Published as a conference paper at ICLR 2024
Table 25: End-to-end Performance Results on CIFAR-100. The classifier head is simple Linear layer.
Model
Aggregation
Projection
CViT-B/32
Ahsolute
0.92 0.05
Cosine
0.90
Euclidean
0.90 - 0.06
L1
0.90 + 0.06
Loo
0.79 + 0.13
MLP+Sum
Cosine, Euclidean, L1. Loo
0.91 + 0.05
SelfAttention
Cosine, Euclidean, Lr, Loo
0.90 + 0.06
RViT-B/16
Absolute
0.85 0.19
Cosine
0.83 0.18
Euclidean
0.83 - 0.118
L1
0.83 0.19
T.oo
0.79 + 0.20
MLP+Sum
Cosine, Enclidean, L1, Too
0.84 + 0.19
SelfAttention
Cosine, Euclidean, L1. Loo
0.82 + 0.20
RexNet
Absolute
0.80 0.21
Cosine
0.76
Euclidean
0.76
L1
0.77 0.20
Loo
0.72 + 0.20
MLP+Sum
Cosine, Fuclidean, T., Too
0.79 + 0.21
SelfAttention
Cosine, Faclidean, T1, Too
0.76 + 0.21
ViT-B/16
Absolute
0.86 + 0.18
Cosine
0.84 0.18
Euclidean
0.84 1 0.18
0.85 0.18
L
0.73 1 0.19
MLP+Sum
Cosine, Euclidean, L1, Loo
0.86 0.18
SelfAttention
Cosine, Euclidean, L1, Loo
0.81 0.19
ViT-S/16
Ahsolute
0.83 0.19
Cosine
= 0.82 0.19
Euclidean
0.82 0.18
L1
0.82 1 0.19
Loo
0.74 0.19
MLP+Sum
Cosine, Euclidean, L1. Loo
0.83 1 0.18
SelfAttention
Cosine, Euclidean, L1, Loo
0.82 0.20
35

---

Published as a conference paper at ICLR 2024
Table 26: End-to-end Performance Results on Fashion MNIST. The classifier head is simple
Linear layer.
Model
Aggregation
Projection
CViT-B/32
Ahsolute
0.92 + 0.05
Cosine
0.06
Euclidean
0.90 - 0.06
L1
0.90 + 0.06
Loo
0.79 + 0.13
MLP+Sum
Cosine, Euclidean, L1. Loo
0.91 + 0.05
SelfAttention
Cosine, Euclidean, Lr, Loo
0.90 + 0.06
RViT-B/16
Absolute
0.85 0.19
Cosine
0.83 0.18
Euclidean
0.83 - 0.118
L1
0.83 0.19
T.oo
0.79 + 0.20
MLP+Sum
Cosine, Enclidean, L1, Too
0.84 + 0.19
SelfAttention
Cosine, Euclidean, L1. Loo
0.82 + 0.20
RexNet
Absolute
0.80 0.21
Cosine
0.76
Euclidean
0.76
L1
0.77 0.20
Loo
0.72 + 0.20
MLP+Sum
Cosine, Fuclidean, T., Too
0.79 + 0.21
SelfAttention
Cosine, Faclidean, T1, Too
0.76 + 0.21
ViT-B/16
Absolute
0.86 + 0.18
Cosine
0.84 0.18
Euclidean
0.84 1 0.18
0.85 0.18
L
0.73 1 0.19
MLP+Sum
Cosine, Euclidean, L1, Loo
0.86 0.18
SelfAttention
Cosine, Euclidean, L1, Loo
0.81 0.19
ViT-S/16
Ahsolute
0.83 0.19
Cosine
= 0.82 0.19
Euclidean
0.82 0.18
L1
0.82 1 0.19
Loo
0.74 0.19
MLP+Sum
Cosine, Euclidean, L1. Loo
0.83 1 0.18
SelfAttention
Cosine, Euclidean, L1, Loo
0.82 0.20
36

---

Published as a conference paper at ICLR 2024
Table 27: End-to-end Performance Results on MNIST. The classifier head is simple Linear layer.
Model
Aggregation
Projection
CViT-B/32
Absolute
0.92 + 0.05
Cosine
0.90 + 0.06
Euclidean
0.90 + 0.06
L1
0.90 = 0.06
Loo
0.79 0.13
MLP+Sum
Cosine, Euclidean, L1, Loo
0.91 = 0.05
SelfAttention
Cosine, Euclidean, L1. Loo
0.90 = 0.06
RViT-B/16
Absolute
0.85 - 0.19
Cosine
0.83 : 0.18
Euclidean
0.83 = 0.18
I11
0.83
Too0
0.79 0.20
MLP+Sum
Cosine, Euclidean, Lt, Loo
0.84 0.19
SelfAttention
Cosine, Euclidean, L1, Loo
0.82 0.20
RexNet
Absolute
0.80 0.21
Cosine
0.76 0.20
Euclidean
0.76 - 0.20
0.77 0.20
Loo
0.72 + 0.20
ML.++Sum
Cosine, Fuclidean, I,, Too
0.79 + 0.21
SelfAttention
Cosine, Fuclidean, T., Too
0.76 + 0.21
ViT-B/16
Absolute
0.86 + 0.18
Cosine
0.84 0.18
Euclidean
0.84 0.18
L1
0.85 1 0.18
Loo
0.73 1 0.19
MLP+Sum
Cosine, Euclidean, Lr Loo
0.86 + 0.18
SelfAttention
Cosine, Euclidean, LA, Too
0.84 0.19
ViT-S/16
Ahsolute
0.83 0.19
Cosine
0.82 0.19
Euclidean
0.82 1 0.18
L1
0.82 1: 0.19
Loo
0.74
MLP+Sum
Cosine, Euclidean, L1, Loo
0.83 0.18
SelfAttention
Cosine, Euclidean, L1, Loo
0.82 0.82 0.20
37

---

Published as a conference paper at ICLR 2024
Table 28: End-to-end Performance Results DBpedia. a. The classifier head is simple Linear layer.
Model
Aggregation
Projection
ALBERT
Absolute
0.71
Cosine
0.61 0.23
Euclidean
L1
0.63 1 0.23
Loo
0.49 1 0.19
MLP+Sum
Cosine, Euclidean, L1, Loo
0.63 + 0.22
SelfAttention
Cosine, Euclidean, L1, Too
0.61 + 0.23
HERT-C
Absolute
0.84 + 0.11
Cosine
0.78 + 0.13
Euclidean
0.80 0.13
L1
0.81 + 0.13
Loo
0.56 1 0.12
MLP+Sum
Cosine, Euclidean, L1. Loo
0.81 1 0.11
SelfAttention
Cosine, Euclidean, Lr, Loo
0.77 1 0.13
BERT-U
Absolute
0.77
Cosine
0.73
Euclidean
0.74
L1
0.74
T.oo
0.46 0.08
MLP+Sum
Cosine, Enclidean, L1, Too
0.74
SelfAttention
Cosine, Euclidean, L1. Loo
0.66 + 0.18
CViT-B/32
Absolute
0.33 0.24
Cosine
0
Euclidean
0.33 0.26
L1
0.33 = 0.26
Loo
0.23 + 0.14
MLP+Sum
Cosine, Fuclidean, T., Too
0.33 + 0.25
SelfAttention
Cosine, Faclidean, T4, Too
0.20 + 0.16
ELECTRA
Absolute
0.76 + 0.14
Cosine
0.15 0.59 1 0.09
Euclidean
0.64
L
0.32 1 0.07
MLP+Sum
Cosine, Euclidean, L1, Loo
0.65
SelfAttention
Cosine, Euclidean, L1, Loo
0.61 + 0.14
RoBERTa
Ahsolute
0.81 + 0.10
Cosine
0.75 0.04
Euclidean
0.77
L1
0.81 1.81100 0.02
Loo
0.36 + 0.07
MLP+Sum
Cosine, Euclidean, L1. Loo
0.80 1 0.04
SelfAttention
Cosine, Euclidean, L1, Loo
0.61 0.27
XLM-R
Absolute
0.74 0.12
Cosine
0.58 0.07
Euclidean
0.64 0.07
L1
0.78 + 0.05
T.oo
0.26 + 0.09
MLP+Sum
Cosine, Enclidean, La. Too
0.78 + 0.06
SelfAttention
Cosine, Euclidean, L1, Loo
0.71 + 0.22
38

---

Published as a conference paper at ICLR 2024
Table 29: End-to-end Performance Results on TREC. The classifier head is simple Linear layer.
Model
Aggregation
Projection
ALBERT
Absolute
0.71 + 0.21
Cosine
0.61 = 0.23
Euclidean
0.62 1 0.22
L1
0.63 + 0.23
Loo
0.49 =: 0.19
MLP+Sum
Cosine, Eaclidean, L1, Loo
0.63 + 0.22
SelfAttention
Cosine, Fuclidean, L1, Tes
0.61 + 0.23
BERT-C
Absolute
0.84 + 0.11
Cosine
0.78 + 0.13
Euclidean
0.80 - 0.13
L1
0.81 1 0.13
Loo
0.56 0.12
MLP+Sum
Cosine, Euclidean, L1, Loo
0.81
SelfAttention
Cosine, Euclidean, L1. Loo
0.77
BERT-U
Absolute
0.77
Cosine
0.73
Euclidean
0.74
I11
0.74
T
MLP+Sum
Cosine, Euclidean, Lt, Loo
0.74
SelfAttention
Cosine, Euclidean, L1, Loo
0.66
CViT-B/32
Absolute
0.33 0.24
Cosine
0.31 = 0.23
Euclidean L1
0.33 0.26
0.33 + 0.26
Loo
0.23 + 0.14
ML.++Sm
Cosine, Fuclidean, I,, Too
0.33 + 0.25
SelfAttention
Cosine, Fuclidean, T., Too
0.20 + 0.16
ELECTRA
Absolute
0.76 + 0.14
Cosine
0.58 + 0.15
Euclidean
0.59 0.09
L1
0.64 0.13
Loo
0.32 1 0.07
MLP+Sum
Cosine, Euclidean, Lr, Loo
0.65 + 0.10
SelfAttention
Cosine, Euclidean, LA, Too
0.61 + 0.14
ROBERTa
Ahsolute
0.81 0.10
Cosine
0.75 0.04
Euclidean
0.77 0.01
L1
0.81 1: 0.02
Loo
0.36 + 0.07
MLP+Sum
Cosine, Euclidean, L1, Loo
0.80 0.04
SelfAttention
Cosine, Euclidean, L1, Loo
0.61 - 0.27
XLM-R
Absolute
0.74
Cosine
0.58 0.07
Euclidean
0.64 0.07
I,1
0.78 0.05
T.oo
0.26 + 0.09
MIP++um
Cosine, Fnclidean, La. Too
0.78 + 0.06
SelfAttention
Cosine, Euclidean, L1, Loo
0.71 + 0.22
39

---

Published as a conference paper at ICLR 2024
Table 30: End-to-end Performance Results N24news (Text). The classifier head is simple Linear layer.
Model
Aggregation
Projection
ALBERT
Absolute
0.71
Cosine
0.61 0.23
Euclidean
0.62 0.22
L1
0.63 1 0.23
Loo
0.49 1 0.19
MLP+Sum
Cosine, Euclidean, L1, Loo
0.63 + 0.22
SelfAttention
Cosine, Euclidean, L1, Too
0.61 + 0.23
HERT-C
Absolute
0.84 + 0.11
Cosine
0.78 + 0.13
Euclidean
0.80 0.13
L1
0.81 + 0.13
Loo
0.56 1 0.12
MLP+Sum
Cosine, Euclidean, L1. Loo
0.81 1 0.11
SelfAttention
Cosine, Euclidean, Lr, Loo
0.77 1 0.13
BERT-U
Absolute
0.77
Cosine
0.73
Euclidean
0.74
L1
0.74
T.oo
0.46 0.08
MLP+Sum
Cosine, Enclidean, L1, Too
0.74
SelfAttention
Cosine, Euclidean, L1. Loo
0.66 + 0.18
CViT-B/32
Absolute
0.33 0.24
Cosine
0
Euclidean
0.33 0.26
L1
0.33 = 0.26
Loo
0.23 + 0.14
MLP+Sum
Cosine, Fuclidean, T., Too
0.33 + 0.25
SelfAttention
Cosine, Faclidean, T4, Too
0.20 + 0.16
ELECTRA
Absolute
0.76 + 0.14
Cosine
0.15 0.59 1 0.09
Euclidean
0.64
L
0.32 1 0.07
MLP+Sum
Cosine, Euclidean, L1, Loo
0.65
SelfAttention
Cosine, Euclidean, L1, Loo
0.61 + 0.14
RoBERTa
Ahsolute
0.81 + 0.10
Cosine
0.75 0.04
Euclidean
0.77
L1
0.81 1.81100 0.02
Loo
0.36 + 0.07
MLP+Sum
Cosine, Euclidean, L1. Loo
0.80 1 0.04
SelfAttention
Cosine, Euclidean, L1, Loo
0.61 0.27
XLM-R
Absolute
0.74 0.12
Cosine
0.58 0.07
Euclidean
0.64 0.07
L1
0.78 + 0.05
T.oo
0.26 + 0.09
MLP+Sum
Cosine, Enclidean, La. Too
0.78 + 0.06
SelfAttention
Cosine, Euclidean, L1, Loo
0.71 + 0.22
40

---

Published as a conference paper at ICLR 2024
Table 31: Graph End-to-End Classification Score. Accuracy across differentt architectures
and seeds. The aggregation function is the Aggregation by sum.
Aggregation
Projection
Accuracy
Absolute
0.79 1 0.01
Cosine
0.74 1 0.01
Euclidean
0.46 + 0.06
L1
0.44 0.06
Loo
0.12 0.03
Concat
Dosine,Euclident
0.76 + 0.01
Cosine, L1
0.77 = 0.01
Los
0.75 = 0.01
Euclidean,Li
0.55
Faclidean, To
0.46 0.11
I1,
0.48 + 0.11
Cosine, Euclidean, Lt, Loo
0.77 +
SelfAttention
Cuclieann Lo
0.75 0.01
0.75 0.01
Loo
0.76 0.02
Euclidcan,Ly
0.74
Euclidean,
0.75 + 0.02
L1,L.
0.75 + 0.02
Cosine, Fuclidean, TA, Too
0.74 + 0.02
MLP+SelfAttention
Cosine.Euclidean
0.76 + 0.02
Tosine,
0.76 0.02
Cosine, T
0.76 + 0.02
Euclidcan, LA
0.73 + 0.01
Euclidean, L.
0.74 1 0.01
Li.Lsc
0.74 1 0.01
Cosine, Euclidean, L1, Loe
0.77 + 0.01
MLP+Sum
Euclidenn
0.77 + 0.01
Losine, Cosine, Loo
0.77 + 0.01
0.77 + 0.02
Faclidean, La Euclidean,
0.75 + 0.02
0,72 + 0.02
Lr
0.71 + 0.03
Cosine, Euclidean, L1, Lewe
0.76 + 0.01
41


Reviews:
[{"review_id": "W8uXzoSDcb", "reviewer": "Reviewer_ay1h", "summary": "This paper expands on Relative Representations by allowing several distances to be combined, which allows incorporating additional invariances in the resulting representation.\n\nThey first present evidence that single distances aren’t sufficient as they are data/model dependent (the original work used Cosine distance), they then explore adding 3 new distances (Euclidean, L1, L_\\infty) in Text, Image and Graph domains.\n\nThis is a rather simple yet very clear and well-executed paper, which brought back Relative Representations to my attention, a nice idea which got washed up in the recent wave of LLM excitement. It might have limited scope, but currently I lean favorably.", "questions": "1. Do you really need to spend that much space and energy on presenting the failures of single distance Relative Representations? \n   1. Figure 1, 3 and 4 are all making a similar point, and Section 4.1 does not feel as crucial as its length suggests it.\n   2. I would probably recommend re-balancing this down and using the extra space to expand on the other Results sections.\n   3. I would recommend keeping either 3 or 4 in the main text but not both.\n   4. I am not sure that Figure 1 is the best framing figure to open the paper with, I might prefer to start with Figure 2.\n2. The aggregations functions are presented well in Section 3, but it would have been useful to present implications for the choices of Sum and SelfAttention, in a similar manner to Concat (“giving to M the structure of a cartesian product space”).\n   1. The Sum aggregation is actually a DeepSet by implementation. I would have liked having this spelt out explicitly and discussed?\n3. The choice of Anchor points A_X and their implications on the invariances or properties of the relative representations are not discussed.\n   1. Section 4.2 mentions using 1280 randomly selected fixed anchors. Did you try changing it? Does it affect distances differently?\n4. I could not find which aggregation function was used for results in Table 1, 2 and 3. This should be specified clearly.\n5. It feels like showing other combinations of distances (instead of “single” vs “all”) would have been helpful, especially if different domains require different distances.\n   1. Section 4.4 tries to go in that direction, but the Transformer aggregation is not the best one and combined with my issue 4, I wasn’t sure what you used, so it muddles the results.\n6. Section 4.4 would benefit from being extended, I do not think it contains enough details currently.\n   1. The experimental setup needs more details, there is no description of the transformer aggregation anywhere I could find.\n   2. Table 5 should contain the value for the best other aggregation (e.g. MLP+sum?), as currently it makes it harder to see if QKV opt is sufficiently accurate or not.\n   3. It is unfortunate that the Transformer aggregation performs poorly. It would be good to bring the MLP+Transformer one to the main text, or at least present more clearly what model is used. It is not my expectation that a DeepSet should outperform a Transformer if it has enough layers?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The paper has a clear focus, presents the problem well, and is overall extremely well executed. \n2. It was easy to follow and the extensions to the math were very well brought up.\n3. It explores appropriate choices of distances and aggregations. Good details and interpretations were provided for what one should expect from them (e.g. Table 6 in the Appendix was exactly what I was looking for)\n4. Results are clear and do improve in predictable fashion over baselines.", "weaknesses": "1. I feel like too much of the paper is spent on presenting evidence for the sub-optimality of single distance relative representations. I did not really understand why that point was made so repeatedly (Figure 1, Figure 3, Figure 4, Appendix Figure 8 and 9), instead of spending more time presenting different *combinations* of distances and their benefits/implications. In effect Table 1 is the first time a clear combination of distances is shown, and it is clearly better than the rest, so I would have wanted more of that.\n2. Equally, as a result, less emphasis and space was spent explaining the results in 4.2, 4.3 and 4.4. I had to go back to the original paper to remember/understand what “zero-shot stitching” meant and how it was implemented.\n3. Details were lacking in a few places, for example which aggregation function was used for most of the results. I assume MLP+sum given it performed the best in Section 4.3, but this isn’t spelt out?\n4. Section 4.4 is also lacking in details and could benefit from some improvements, see below.\n5. It is potentially of limited scope, but I would defer to the majority vote to see if that is a blocker or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZyzP7Cvpb3", "reviewer": "Reviewer_NofE", "summary": "This paper extends the work on Relative Representation by ensembling multiple relative representations obtained by different distances. The combination of four distances cos, Euc, L1, $L_\\infty$ and three ensemble methods concat, sum, and attention are explored in the text. Extensive experiments across text, graph, and vision domains demonstrated that the ensembled version can improve the performance of zero-shot stitching.", "questions": "See weakness section.\n\ntypo:\n\n1. page 4, invriances\n2. page 9, fourth row -> fifth row", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Ensembling multiple relative representations is a reasonable idea and it enhances the power of the original cosine relative representation.\n2. The experiments are extensive. There are 28 tables including the appendix.\n3. The writing style is formal.", "weaknesses": "1. The selection of distances seems arbitrary.\n    - (a) While the Euclidean distance is invariant under the Euclidean isometries and is a reasonable candidate beyond the Cosine distance. What is the rationale of the rest of the distances? Any geometric intuitions?\n    - (b) The Euclidean isometry is a special case of conformal (angle-preserving) map. For experiments that show better performance on Euclidean distances than on Cosine distances, what can we say about the underlying symmetries of the neural representation? Does it mean that that latent space contain less invarinace? I am asking this question because I want to see what extra understanding on neural representations we can get from this new formulation.\n    - (c) Page 2 \"... which, combined, can capture arbitrary complex transformations of the latent space\". It seems an overstatement to claim that the four chosen distances can capture \"arbitrary complex transformations\".\n2. The Assumption in page 3 does not read smoothly. \n    - (a) The equivalence class of encoders is defined as the set of E such that $ \\pi_\\mathcal{M}TE=\\pi_M E, \\forall T\\in\\mathcal{T}$. This definition is confusing. I fail to see why it is an equivalence class. For example, say, $\\mathcal{T}_1$ is scalings, and $\\mathcal{T}_2$ is rotations, and $E$ is a constant mapping to the origin. Does $\\mathcal{T}_1$ and  $\\mathcal{T}_2$ induces two different equivalence classes of transformations? But clearly $E$ belongs to both classes of transformations.\n    - (b) Suppose $\\mathcal{M}$ is a single point. Then $\\pi_{\\mathcal{M}}TE=\\pi_{\\mathcal{M}}E$ for all $E$ and all $T$. This definition does not contain any useful info then.\n3. Page 5 \"it is not possible to connect latent spaces of models with different initializations ...\" It seems that the Pearson correlation for Cosine is higher than 0.94 in the left subfigure of Fig. 3 and higher than 0.8 in the right subfigure. What is the criterion for the statement of \"no connection\"? Any reference for the choice of criterion? I do not see this as \"challenges the assumption in Moschella et al.\".\n4. Please clarify the aggregation used in Tab. 1 to Tab. 3, since there are multiple possibilities.\n5. Sec. 4.4 leads confusion. What is the difference between SelfAttention and SelfAttention + MLP opt? Isn't the SelfAttention trained (finetuned)? If not, what is the exact computation formula for the SelfAttention aggregation? Where is the initial values of the attention weights come from? Also, the numbers in Tab. 5 does not match that of in Tab. 15.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9rWxldKVkd", "reviewer": "Reviewer_8YvH", "summary": "This paper presents a product projection mechanism to generalize the framework of relative representation. In particular, the authors incorporate a set of invariances into the representation by constructing a production space of invariances. The findings are intuitive that multiple projections behave differently across different choices on initialization, model architecture, etc. Experimental results proved the effectiveness of the proposed method.", "questions": "No.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The motivation is well presented of infusing multiple invariances into relative representation.\n\n(2) The explanations and illustrations are mostly clear and intuitive of the manifold assumption and the product projection mechanism.", "weaknesses": "(1) On Page 5, the result analysis presents the discovery challenges the assumption in Moschella et al. (2022). More explanations are required to make this point clear. Besides, wondering if the experimental results are just a normal fluctuation due to different runs.\n\n(2) On Page 6, the authors used 1280 randomly selected but fixed anchors. This is also a kind of randomness that is not explained away. In fact, for different choice of anchors, the sensitivity is different of the projection and measure function.\n\n(3) On the experiments, the employed datasets and models are in small-scale and probably prone to overfitting issues. Do the analysis conclusions hold for large-scale models such as Stable Diffusion and GPT? There should be large-scale results to support the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "soeIhxPpej", "reviewer": "Reviewer_pGUF", "summary": "The paper is about enhancing the relative representation. Relative representation is determined with dissimilarity measure between target data and anchor that is invariant to angle transformation. The former work of Moschella et al. (2022) uses cosine angle as this dissimilarity, but in this paper, it aggregates other dissmilarity to enhance latent communication. The results of this aggregation is assessed by accuracy of zero-shot classification using stiching models.", "questions": "1. In experiment in Section 4.3 ~ 4.4, is the MLP and self-attention aggregation trained (fine-tuned) in end-to-end fashion? \n\n2. Does the downstream task with relative representation presented also enhance the performance of other tasks? (e.g. generation)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper gives evidences that why the relative representation only using cosine angle can be inappropriate.", "weaknesses": "1. The definition of the RR framework is strange. It is stated that RR is concatenation of $d(z, a_i)$, but $z$ and $a_i$ should be in different domain $(\\mathcal{Z},$ and $\\mathcal{X})$. I am assuming that the anchors are also encoded with $E_\\theta$ so that $a_i$'s in the latent space $\\mathcal{Z}$\n\n2. The experiments setting in the section 4 is unclear. I am having trouble figuring out what is a stiching model for this downstream task and how it is trained. I am assuming it is the same definition as the stiching model defined in Moschella et al. (2022), but I am having trouble how the decoder for this down-stream task is (pre-)tained.\n\n3. The enhancement of relative representation through aggregating is not convincing for me. In Table 2, the aggreagated accuracies closely matches with using $L_1$ encoder. Using MLP or SelfAttention in aggreagation does not seems to be fair in that it requires an addtional training to get the additional parameters for these layers (correct me if I am wrong.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)