Latent Space Translation via Semantic Alignment
Authors: Valentino Maiorca
Luca Moschella
Antonio Norelli
Marco Fumero
Francesco Locatello
Emanuele Rodolà
Venue: NeurIPS 2023 poster (2023)
Keywords: latent space translation
relative representation
Procrustes analysis
zero-shot
stitching
latent communication
representation learning
manifold alignment
multimodal
TL;DR: A semantic correspondence between latent spaces is enough to unlock translation between them, reaching zero-shot stitching of arbitrarly pre-trained neural components across various tasks, modalities and architectures.

Abstract: While different neural models often exhibit latent spaces that are alike when exposed to semantically related data, this intrinsic similarity is not always immediately discernible. Towards a better understanding of this phenomenon, our work shows how representations learned from these neural modules can be translated between different pre-trained networks via simpler transformations than previously thought. An advantage of this approach is the ability to estimate these transformations using standard, well-understood algebraic procedures that have closed-form solutions. Our method directly estimates a transformation between two given latent spaces, thereby enabling effective stitching of encoders and decoders without additional training. We extensively validate the adaptability of this translation procedure in different experimental settings: across various trainings, domains, architectures (e.g., ResNet, CNN, ViT), and in multiple downstream tasks (classification, reconstruction). Notably, we show how it is possible to zero-shot stitch text encoders and vision decoders, or vice-versa, yielding surprisingly good classification performance in this multimodal setting.

Latent Space Translation via Semantic Alignment
Valentino Maiorca
Luca Moschella'.
Antonio Norelli
Marco Fumero'
Francesco Locatello
Emanuele Rodolà !
Sapienza University of Rome
PIsstitute of Science and Technology Austria (ISTA)
Abstract
While different neural models often exhibit latent spaces that are alike when ex- posed to semantically related data, this intrinsic similarity is not always immediately discernible. Towards better understanding of this phenomenon, how representations learned from these neural modules can be translated between different pre-trained networks via simpler transformations than previously thought. An advantage of this approach is the ability to estimate these transformations using standard, well-understood algebraic procedures that have closed-form solutions. Our method directly estimates a transformation between two given latent spaces, thereby enabling effective stitching of encoders and decoders without additional training. We extensively validate the adaptability of this translation procedure in different experimental settings: across various trainings, domains, architec- tures (e.g.. ResNet, CNN, ViT), and in multiple downstream tasks (classification, reconstruction). Notably, we show how it is possible to zero-shot stitch text en- coders and vision decoders, or vice-versa, yielding surprisingly good classification performance in this multimodal setting.
work shows
Introduction
Representation learning [Bengio et al. 2014 is a fundamental paradigm in the field of artificial intelligence, aimed at uncovering the underlying structure of complex data. One of the main goals of representation learning is to discover a robust representation of the data that is insensitive to certain transformations of the input. The Manifold Hypothesis [Fefferman et al.] 2013) posits that real-world data lies on a low-dimensional non-linear manifold embedded in a high-dimensiona space. Yet, a complication arises in modeling these non-linear manifolds: the learning process is usually influenced by stochasticities in the training dynamics and extrinsic factors that do not pertain to the data's core attributes, resulting in different representations for samples expected be similar (e.g.. different views of the same object, multiple translations of the same sentence, or even the exact same input sample). This is critical hinders knowledge transfer between these networks. Recently, the concept of relative representations [Moschella et al. (2023) has been proposed as method for zero-shot communication between latent spaces that is invariant to these extrinsic factors. The idea is that latent spaces of neural networks trained on comparable data can be projected into the same relative space, derived from the distances between the data points. One of the main contributions of relative encoding is that it shows how the signal encoded in the angles with respect to reduced set of data points (called anchors) is enough to capture the intrinsic shape the latent space, reaching results on various benchmarks comparable to those using the original (absolute) encodings. As a consequence, they empirically demonstrate that different latent spaces that share the same data semantics (i.c., different representations of the same high-level concepts, such images and their captions), mostly differ only by an preserires transformation.
"qqual contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

T(X)
X
Y
X A
Z
Y Y
decx
decy
decz
Absolute
Relative
Figure 1: Zero-shot stitching of X and Y absolute spaces utilizing relative representations and our method (the estimation of T). Our approach does require decoder specifically trained relative representations (decz). Instead, we directly translate latent spaces, enabling the use of arbitrarily pre-trained decoders originally trained on absolute spaces.
Building on this intuition of the existence of relatively simple transformation, we show the effective- ness and applications of directly translating between different lutent spaces provided that partial (and possibly sparse) correspondence between data points given. Remarkably, the process of seam- lessly combining different neural networks, pre-trained on diverse datasets, modalities, architectures, domains and downstream tasks, proves unexpectedly straightforward. For instance, we show how enables the ability to effectively integrate any pre- pre-trained text encoder with any image classification head, and vice versa, without requiring any additional re-training or assumptions (indeed, Moschella al. [2023] assumes the decoders are trained on relative representations). The method difference is emphasized Figure I While zero-shot stitching with relative representations assumes the use of a single decoder specifically trained relative space, method permits the reuse of the decoders originally trained on the absolute spaces.
Our main contributions can be summarized as follows:
We explore the direct translation between latent spaces of distinct neural networks to enable latent communication. In particular, leveraging semantic correspondence in the data, we directly translate for the first time across different trainings, architectures, and modalities. Notably, we obtain excellent stitching performances even in cross-modal settings, where we apply arbitrary text classifiers on top of pre-trained image encodings (and vice-versa).
We show that different downstream tasks, namely classification and generation, require modeling different transformations to obtain the most out of the translation between their latent spaces;
2 Related Works
Representations similarity Recent years have witnessed a growing consensus among researchers in the deep learning community that effective neural networks tend learn similar representations
for semantically similar data, regardless of the architecture, task, or domain in which they are applied.
This idea is supported by a plethora of empirical studies [Moschella al., 2023 Norelli et al. 2023
Morcos et al., 2018 Li et al.) 2016 Kornblith et al. 2019] Bonheme and Grzes] 2022 T Titsulin
2020
Barannikov
al. 2022 Vulié al. [2020a] Lample et al. 2018 Lene and Vedaldi
2015
Mikolov et al. 20120033 3a Antonello et al. 2021 Bengio et al. 2012 Movshoviz-AAtiaas et al.
2017
Chang et al. | [022 and the phenomenon is particularly pronounced for large and wide models
[Somepalli et al. 2022 Mehta et al.) 2022]. Nevertheless, despite this intrinsic similarity, latent
spaces can still exhibit extrinsic variations. Our work analyzes the possibility of translating these spaces from one to another, linking these extrinsic variations different classes of transformation.
Manifold alignment Procrustes analysis has been instrumental the alignment of latent spaces in deep neural networks [Wang and Mahadevan, 2008 2009), particularly in Natural Language Processing (NLP) where well-known that latent spaces different languages are isomorphic Vulić et al.
2020b and can be effectively aligned (Mikolov et al., 2013b Xing al. 2015). Rooted

---

Normaiization
De-ocmaliiaaion
Source X
Centering
Scaling
De Sealing
De Cenering
Target Y
Figure 2: Method illustration synthetic example. Given source space X, the steps translate to a target Y are sequentially applied as described Section 3.2 Note that the translation is not perfect due to arbitrary distortion of the data.
in shape analysis, this method efficiently uncovers correspondences between latent spaces different models through the estimation of optimal orthogonal transformation [Gower 1975]. Previous works largely exploit Procrustes analysis to align latent spaces originating from models of the same architecture [Csiszarik et al. 2021), such multi-lingual Fast'Text embeddings (Bojanowski et al.) 2017 Smith et al. 2017]. Instead, in this work, we extend the application of Procrustes analysis to network stitching new domains, architectures, and even modalities for multiple downstream tasks.
Stitching and zero-shot Model stitching, which involves the combination of different neural net- works to create a new model, has been topic of active research in the field of representation learning. A key concept in this area is that of relative representations [Moschella et al. 2023 Norelli et al.) 2023]. which enables zero-shot stitching between different neural networks trained on semantically similar data. While this approach assumes the use of decoders trained on relative representations, our work removes this constraint by introducing a zero-shot mechanism for translating one absolute space to another without relying shared (relative) representation, enabling the stitching of arbitrarily trained models, further generalizable by assuming only the positive scale invariance their decoder part. Previously, trainable stitching layers [Lene and Vedaldi 2015 Bansal et al. 2021 Csiszariki jet al. [222]] have been introduced to allow for the combination of parts different networks or to
verify statements regarding latent space similarity. Other works [Gygli al. 2021 202 Yaman et al. 2022 an 2020) have proposed alternative methods for producing directly compatible and
reusable network components without specific stitching layers. Here, sidestep the need create new compatible representation and instead focus on obtaining a direct transformation to map from source space target one enable seamless network stitching. Concurrently this work, similar approach by [Lähner and Moeller [0033] also targeted the direct alignment of representational spaces, focusing on the compatibility of models trained end-to-end.
Method
3.1
Preliminaries
Relative representation a framework introduced in Moschella et al. [2023], which enables latent spaces of arbitrary neural models communicate with each other. This is obtained by projecting the latent spaces into a common one, transitioning from an absolute coordinate frame to a relative space: function of a set of fixed samples denoted anchor set. Specifically,
each sample is represented the new representation is computed by independently projecting each sample point x in the latent space X E gaxd into the anchor set Ax C. X. Formally, this is represented as
X,el = Xabs A
(I)
where X,et E RRnk Xabs E Raxd and Ax E Bexd Samples in X and in A are rescaled to unit norm, i.c. X ||x||2 Vx X and a [|al|a Va Ax.
We assume to have access to subsets Ax C X and Ay V. with X' and y being the data distributions, and that there exists a correspondence I Ax + Ay between these two sets of parallel anchors. Parallel anchors act as a "Rosetta stone" [Norelli al. [2023), meaning they establish semantic correspondence between their respective spaces: an anchor sample in the first set represents same high-level concept as its counterpart the second set. This allows stitching together components of different models: i.e., merging independently trained encoder and decoder modules from different networks. The relative projection will map latent spaces into the same one as long as the core assumption that they differ by angle- seeeleepeeervingg transformation is satisfied. However, in
3

---

order to perform the stitching procedure in Moschella et al. [2023], decoders must be trained from scratch least once to process samples in this shared relative space.
In this work, we overcome this need by substituting the costly retraining procedure with an efficient strategy directly estimate the transformation necessary map between spaces. Moreover, relax the preeeeeprerr constraint by allowing for a broader class of transformations obtained via robust, closed-form algorithms.
3.2 Latent Space Translation
Consider two latent spaces. E Rexd and Y E Raxda Our objective is to estimate the transfor- mation T that translates X into Y: Y T(X), exploiting the semantic alignment between the two spaces. Throughout this work, we identify two main steps in the translation process: pre-proe pre-processing the spaces and estimating the transformation TT, as outlined in Figure 2
Pre-processing Generally, the two spaces may have different dimensionalities - in those cases, zero-pad the smaller one to match the dimension of the other without changing its underlying structure WWillamms et al. [202]]. Moreover, we standardize each feature in the encoding to have zero mean and unit variance (standard scaling) if not otherwise specified, whose statistics are computed only on the anchor sets for both source and target space, to perform the necessary dc-normalization.
Estimating T In Moschella et al. 2023], it is empirically shown that the spaces mostly differ by an angle-preserving transformation. Nevertheless, we broaden our investigation by considering different ways of obtaining T to evaluate the robustness of that assumption and the versatility of our approach. Throughout our experiments, we primarily operate under the assumption that T can be constrained to encode, at most, affine transformation: T(x) RX + b
This general formulation, without additional constraints, corresponds to our a ine method in the experiments, and it is optimized via gradient descent. The other transformations are trivially obtained by progressively adding constraints on this one:
linear. To model linear transformation, we can just set the bias term zero b 0 and optimize via Least Square. Here we are both simplifying the class of transtormations and switching from a gradient descent optimization to closed-form procedure.
1-ortho. Additionally, we could require R to be orthogonal to encode an isometry. In this case, we obtain this by applying Singular Value Decomposition (SVD) on the corresponding R obtained by the linear solution. Through this, aim to understand the implications of enforcing orthogonality on a transformation that was originally not constrained to be so, in a setting similar to Xing et al. (2055).
* ortho. To obtain the optimal orthogonal R, we apply Procrustes analysis Gower 1975].
This methodology facilitates efficient and precise zero-shot translation between disparate latent spaces. The transformation T, derived solely from the subset of corresponding points, provides robust and versatile foundation for model reuse and interoperability in diverse machine learning contexts.
Latent Communication via Translation
In this section, we evaluate the capabilities and effectiveness of our translation method through various scenarios, highlighting applicability in diverse contexts. We present empirical results in three different novel settings: i) cross-architectur;; ii) cross-modality; iii) autoencoding. In each case, the translation performance of each method for obtaining the transformation T is evaluated against two baselines, the naive absolute one and the relative one.
Stitching Procedure In line with the zero-shot stitching concept outlined in Moschella et al. 2023 we combine independent encoders and decoders (e.g,, classifiers, generators) without subsequent training or fine-tuning. This study does not necessitate a decoder trained on relative representations; instead, we directly employ the original decoders trained on absolute spaces. Each benchmark we perform follows the same procedure unless otherwise specified: we measure the mean performance over all the possible combinations of (encoder, decoder) for each test set in different settings:

---

0.8
0.6
Accuracy
0.5
0.3
0.2
arinn
ortho
100
200
300
400
500
600
700
800
900
3000
11.00
1200
1300
1400
1500
Number anchors
Figure 3: Performance comparison of affine, linear, l-ortho, and ortho at varying number of anchors classification accuracy. Results CIFAR100 fine-grained. The same analysis for the generation case is in Figure in the Appendix.
Table 1: Cross-architecture stitching with various methods for estimating and applying standard scaling. The stitched decoders SVMs with linear kernel. 5 runs for each encoder-decoder pair. (C) and (F) next to CIFAR100 indicate, respectively, coarse- grained and fine-gu areegraiddd. Please refer to the Appendix in Table for additional results with MLPs classification heads.
Stitching
ralative
1-ortha
0.95 0.83
0.16 0.22
0.05
0.93 0.04
Viiin
CIPAR100-C
0.85 + 0.07
0.11 0:21
0.54
0.25
0.78 = 0.00
0.73
0.16
0.77 - 0.11
0.81 2 0.07
CIPAR1DO-P
0.76 + 0.00
0.07 0:21
0.30 + 0.24
0.68 0.11
0.62 + 0.19
0.64 = 0.16
0.71 0.09
R-MSIST
0.15
0.96 0:01
0.15 0:21
0.50
0.94 0.01
0.89 0.08
0.81 =
0.81 0.02
0.12
0:20 0.06
0.12
0.25
+
0.79
AC Novn
0.73 +:34-:100 0.99
0:25 0.02
0.39 +: 0.13
0.66 = 0.08
0.62 0.08
0.61 + 0.30
0.66 0.10
DBpedia
0.78 4 0.23
0.07 0:01
0.16 + 0.10
0.66 d 0.24
0.62 0.23
0.57 +: 0.23
0.68 0.22
TMDS
0.50 0.01
no-stitch. The end-to-end performance of the decoder applied to the original space it was trained on. This useful to establish un upper-bound in performances;
absolute. The result of using the encodings without any transformation, we consider this as probe for any pre-existing compatibility among encodings and, therefore, a lower-bound; translation. These are the results of the application of our latent translation method, with the estimation of T via affine, linear, 1 1-ortho and ortho.
In each instance, we use the same parallel anchors, that are uniformly chosen, quantity comparable with the dimensionality of the absolute representation.
4.1 Cross-Architecture
Firstly, we test our method in a cross-architecture setting, zero-shot stitching together encodings coming from variety of pre- pre-trained networks and their associated absolute decoders (classifiers). This scenario provides an extensive testing ground for our method and demonstrates its robustness across different architectures. Please refer to Table 8 in the Appendix for further results cross- architecture stitching in generation tasks.
Experimental setting We consider a variety Computer Vision (MNIST Fashion MNIST N24News, CIFAR10, CIFAR100) and Natural Language Processing (TREC Hovv et al. 2001 Li and Roth 2002). DBpedia [Auer al. 2007). N24News [Wang al.) 2022 AG News [Zhang al. 2013). IMDB MMas et al. 2011 datasets. For the text domain we consider 7 different language models as encoders (uncased and cased BERT [Devlin al.) 2019], Electra (Clark et al. 2020 RoBERTa base [Liu et al. 2019]. ALBERT [Lan et al. 2020 and the text encoder of [Radford et al. 202T)), and for the image domain 6 encoders (RexNet100 Han et al. 2020), 4 variations of ViT
Doosoviskki al. 2020 and the image encoder of [Radford et al. 202 all pre- pre-trained and frozen.
5

---

Table 2: Cross arosstcrchi stitching with various methods for estimating T and applying L2 normal- ization. The stitched decoders are SVMs with linear kernel. 5 runs for each encoder-decoder pair. (C) and (F) next to CIFAR100 indicate, respectively, coarse- -ranine and fine- iheeggriddd. Please refer to TableeOinn in the Appendix for additional results with MLPs as classification heads.
Datanet:
No Stitching
absoluse
relative
aff
1 linsar
1-ortho
artho
CIFAR1O
0.95 + 0.03
0.16 + 0.22
0.50 + 0.22
0.93 = 0.04
0.78 + 0.27
0.88 = 0.32
0.91 + 0.09
Vidim
CIPAR100-C
0.85 + 0.07
0.11 1111000 0.21
0.54 + 0.25
0.79 = 0.07
0.65 + 0.25
0.73 4 0.17
0.79 0.
CTPAR100-2
0.76 0.00
0.07 0.21
+
0.69 0.10
0.19
0.13
F-13137
D.88 0.01
0.15 0:20
0.63
0.86 0.01
0.63 0.23
0.83 = 0.06
0.84 0.05
WK187
0.96 + 0.01
0.15 1: 0:21
0.50 + 0.22
0.94 = 0.01
0.61 + 0.23
0.90 = 0.08
0.90 2 0.04
TRBC
0.87 0.12
0.20 0.06
0.36 1: 0.13
0.82 2: 0.12
0.44 1: 0.20
0.74 4: 0.23
0.77 0.12
AC Neva
0.73 + 0.99
0.25 + 0.02
0.39 + 0.13
0.00 = 0.08
0.06 + 0.10
0.02 + 0.08
0.64 + 0.10
DEpodia
0.78
0.23
0.07 0.01
+
0.06 0.24
0.44
+ 0.23
0.00 0.22
TMDS
0.61 0.04
0.50 + 0:01
0.51 + 0.02
0.50 = 0.04
0.55 0.03
0.58 - 0.04
0.59 0.64
The full encoder list can be found in Table in the Appendix. For each dataset and for each encoder, we train an SVM classification head (decoder) on top of their specific encodings. We then proceed with the standard stitching procedure outlined in Section 3 and collect the results. Please see Table in the Appendix for arssttrrcie stitching in generation tasks, where we extend this analysis by verifying that our method works even across autoencoders of different bottleneck sizes.
The stitching results are in Table T As expected, the absolute encodings obtain score comparable to random guessing while also considering fewer encoder combinations out of the possible ones due to the dimensionality mismatch between some of them. Notably, these results show that the transformation relating to these pre-trained encoders is indeed mostly orthogonal: i) ortho and anei the narrowest and the broadest transformation classes considered, are the better-performing translation methods. But while the former is obtained via a simple and efficient closed-form algorithm, the latter is SGD-optimized (Section 3.2). ii) the 1-ortho version improves or has small drops in performances over the linear transformation it is obtained from, confirming that the least squares procedure converges to an R which is almost orthogonal. Note that these results demonstrate the feasibility of combining pre-trained models without the need for retraining or fine-tuning, with negligible drops in performances across the board and without any additional assumption the decoders. Please refer to 5 in the Appendix for results with different decoders. In the Appendix (Figure Z we extend the archstacchiectture transfer to decoders trained on different domains (styles) of the same CIFAR10 dataset: the original one and grayscale one.
Result analysis
Sensibility to Anchor Quantity The number of anchors is an essential parameter in our approach. In Figure 3 we evaluate how the quantity of these anchors impacts the residual error and the overall performance of our method for this experimental setting. This analysis offers insights into the optimal number of anchors necessary for efficient latent space translation.
Role of Scaling Our approach is designed to accommodate generic (re (re)scaling methods as pre- processing steps. We advocate for the use of standard scaling, as it shows reliable performance in our experiments, indicating that the scale of the data points is useful in estimating the latent transformation T. However, for completeness, we also consider L2 normalization, which is the standard normalization in relative representations. This normalization method generalizes the class of transformations handled by our method and introduces an element of complete scale invariance. It's important to note that when this level of generalization introduced, a scale-invariant decoder is required since the norm information is effectively removed. In the relative representation work, this is implicitly accomplished by training a decoder on relative representations. In our setting, since we do not train the decoder, in this setting we just assume its scale invariance (more details in Appendix [..I]. This investigation exemplifies the flexibility of our approach, capable of adapting to different normalization and pre- pre-processing strategies based on the specific requirements of the task at hand. The results presented in Table 2 when compared with Table [ indicate a stronger reliance of the text modalities on the information encoded in the norm. This is aligned with existing literature in the NLP domain OOyma et al.) 2022]. which suggests that the scale of the encodings contains information (e.g., it correlated with the token frequency).
These results in diverse scenarios showcase the flexibility and adaptability of our method, especially its robustness in translating between latent spaces of different dimensionality and domains.

---

Vision
Language
Number of samples (log)
currd
huee caccoed
operial www.alttpooo-baaeeeeeee
Scale Figure 4: Scale distribution in encodings of different pre-trained encoders the N24News dataset.
Scale
VIT-base-22 224
0:40
0.33
0.41
0.38
0.40
0.35
0.40
0.39
0.29
0.55
0.17
0.55
RexNer
0.36
0.3
0.31
0.57
0.98
0.55
0.37
0.26
0.54
0.16
0.5
Score
Scale
ViT-buse-B 384 0:00
0.32
0:41
0.59
0.18
0.57
VIT-base-224
0.40
90.45
ReaNet
Decoder
ViT-ResNet50
VET huse 384
384
0.41
89.66
50.
DarkNet
WT-RRcsNer50
0.41
32.10
BEXT-oned
0:31 0.30
0.57
0.00
DarkNet
11.62
DERT-incased
0.34 0.29 0.35 0.33 0.31
0.42
0.36
0.65
0.24
0.64
BERT cased
0.61
15.43
uncased
Electra 0.34 0.29 0.35 0.52 0.31 0.25 0.43 0.34 0.30
0.66
0.20
0.64
Electra
0.50
11.94
RoBEKTS
0.75
11.06
ROBERTs 0.78 0:31 0.30 0.37 0.36 0.3 0.50 1.39 0.40
0.75
021
0.72
ALBERT
0.27
32.27
ALBERT 0.30 0.25 0.31 0.28 0.32 0.55 0.40 0.33 0.32 0.56
0.7
0.55
XLM-R
0.73
18.75
XLM-R
0.37 0.33 0.38 0.57 0.99 0.1
1.38
0.55
0.2
0.71
ViT 234
RodNei
VrEhase-- VTTmmll
VyF-ReNeei
Drak
BERT BERT-oRud Foried
RENT Famaard
Eleatra
RoWERTa
ALBURT
XLM--
Encoder
Figure 5: Performance comparison between different encoders and data modalities on the N24News multimodal dataset. On the right the accuracy of models trained end-to-end on single data modality (Score) and their average norm (Scale). On the left the stitching performance between pairs of encoders and decoder. This shows the importance of translating from good encoders, that can even improve unimodal decoder performances. Results obtained with 2000 anchors and ortho, with an SVM classification head. the Appendix Figure 9 additional results using MLPs as decoders.
4.2 Cross-Modality
This scenario illustrates the applicability of our method in cross-modality settings, where we aim to translate between latent spaces of different modalities: text and image.
Experimental setting We adopt N24News Waan et al. 2022
multimodal news classification
dataset that contains both text and associated pictures. We apply the standard encoding procedure to these two features separately, using different pre-trained uni-mod mni-modal encoders. Then, we train classification head (an SVM, please refer to Appendix Figure 9 for further results employing an MI.P classification head) on top of each one. Lastly, we zero-shot stitch each encoder with classification head different from its corresponding one, measuring its classification accuracy, without further training fine-tuning.

---

7
2
4
4
Aba
7
aff
7
2
/
0
4
7
7
2
/
0
4
7
1-ortho
7
2 2
/
6
4
4
ortho
7
4 4
Figure 6: Reconstruction examples grouped by dataset. Each column is a different image, from top to bottom: original image, absolute stitching, affine stitching linear stitching, 1-ortho stitching, and ortho stitching. No additional normalization applied on the decoder part. Please refer Figures 10 and in the Appendix for decoders trained with L2 normalization.
Scale distributions Figure 4 we present the scale distribution of the embeddings produced by several encoders on the N24News dataset. This empirical analysis shows a consistent pattern among encoders that the scale distribution of their embeddings follows Gaussian one with a single mode and well-defined mean, which are usually compatible with standard scaling. This consistent behavior across encoders likely attributed to their architectural choices, such as the normalization techniques, regularizations and the optimization problems they are designed to solve.
Result analysis The discrepancy in the mean accuracy represented by the marginal bar plots in Figure is signal that can be used to identify spaces more suited to be decoded into and the ones that are stronger in encoding from. In fact, the language models as source space for the translation exhibit stronger performance than the vision encoders. We relate this behavior to the higher generality of the text domain data used during pre- vee-traiiig with respect to the image domain one [Zhai et al.) 2022). A remarkable finding in this setting is the improvement in classification performance when a
classifier trained on images is fed zero-shot with corresponding text encodings translated to the image domain via encoder and demonstrates the broad applicability of our technique. In practice, this means we can seamlessly apply image classifiers on textual data, and vice- versa.
method. This result underlines the significance of good
These results show that our method: i) obtains effective zero-shot translation over different modalities; ii) improves unimodal decoders when translating from better encoder than the one it was trained on.
4.3 Autoencoding
In this setting, our method is applied align latent spaces of different trainings of the same autoen- coder. The novelty of this scenario lies in the generation setting itself, as most prior works (Section 2) primarily focus on classification tasks. One key observation of [Cannistraci al. 2023 is that the task at hand (e.g., classification, generation) defines certain class of transformations (e.g. rotations) which act among the latent spaces. Restricting the search for the transformation to the right class, fundamental in order to guarantee optimal performance and efficiency.
Experimental setting We utilize four datasets for these experiments: MNIST Leeun al.) 1998]. Fashion MNIST [Xiao el al. 2017 and CIFAR10 and CIFAR100 Krizhevsky 20099. For each dataset, we train two standard CNN-based autoencoder, with convolutions in the encoder and deconvolutions in the decoder, please refer to the supplementary material for further implementation details. The two autoencoders are identical in structure. differing only the random seed used for weight initialization and data shuffling. To perform zero-shot stitching, we first translate each data point from the latent space of the first encoder to the latent space of the second using 1000 anchors. We then apply the second decoder to the translated data, without any further training or fine-tuning.
Result analysis This experiment analyzes the alignment of latent spaces in different training regi- mens of the same autoencoder. The performance evaluation, as shown in Table 3 demonstrates that all methods affine, linear, 1-ortho, and ortho yield satisfactory results. Moreover, qualitative results depicted in Tigure&rreveal. minimal visual differences in the stitching outcomes across various datasets using different methods. Please refer to Figures [0 and TT for other qualitative results. In fact, these results suggest that the latent spaces of image autoencoders are not exclusively correlated
8

---

Table 3: Zero-shot stitching for generation with various methods for estimating T. Standard scaling used as normalization and the stitched decoders not have any additional normalization. We report the latent cosine similarity (Icos) and MSE (Imse) between the target encoding and the translated one, but also the reconstruction MSE (rmse) between the input and the output. 1000 anchors used on 500 dimensional spaces. Please refer to Table 4 for results on decoders scale-invariant by design (with L2 normalization on the encodings).
MNIST
Fashion MNIST
CIFAR10
CIFAR100
Icos
Imse
Icos
Imse
mmse
Icos
Imse
Icos
Imse
absolute
0.09
0.27
0.14
0.17
0.23
0.23
0.30
0.29
0.34
0.34
0.53
0.40
affine
0,94
0.08
0.02
0.94
0.06
0.03
0.96
0.03
0.05
0.96
0.04
0.05
linear
0,92
0.09
0.02
0.93
0.07
0.04
0.94
0.03
0.05
0.94
0.04
0.06
1-ortho
0.79
0.14
0.02
0.78
0.12
0.05
0.85
0.05
0.06
0.84
0.07
0,07
ortho
0.90
0.10
0.02
0.90
0.08
0.04
0.94
0.03
0.06
0.93
0.04
0,06
by orthogonal transformations. Therefore, further research warranted to explore and model the specific class of transformations that govern the correlation between neural networks during image autoencoding to constrain and enhance their approximation. For additional results pertaining to decoders with L2 normalization their input, we refer to the Table inn the Appendix.
Overall these results, combined with Cannistraci al. 2023 and Section 4. confirm that latent spaces in image autoencoders trained end-to-end are related by a class of transformations larger than orthogonal transformations.
Conclusion
At the heart the proposed latent space translation lies the synergy between the principles of relative representation and classic algebraic techniques. The efficacy of this approach surpasses that of relative representations, emphasizing the benefits of directly estimating transformation between specific latent space pairs instead of independently projecting them to common one. This distinction underscores our contribution: we repurpose well-established techniques to serve translator across multiple latent spaces, enhancing efficiency in representation learning. With an extensive analysis of its applications model reuse, we obtain a smooth compositionality of neural network modules across diverse computational frameworks, including those employing pre-trained models. Essentially, this paper showcases the adaptability and efficiency of manifold alignment methods the emerging domain of zero-shot model compositionality.
Future works and limitations As with any new approach, there are limitations that warrant further exploration of our proposed method. For example, the optimal number of anchor points required for different tasks and datasets to boost performances, investigating the factors that could be linked to latent space compatibility (e.g., their intrinsic dimension), trade-off's between the granularity of the anchor set and its condition number. These are exciting research directions that we believe hold great potential for advancing the field and improving the effectiveness and robustness of our method.
Acknowledgements This work supported by the ERC grant no.802554 (SPECGEO), PRIN 2020 project no. no.2020TA3K9N (LEGO.AI), and PNRR MUR project PE0000013-FAID Francesco Locatello did not contribute this work Amazon.
Reproducibility Statement We refer the supplementary material for implementation details that are not described here in the main manuscript. Moreover, we release modular Py ToTch codebase implementing the various translation methods and scaling techniques. All the experiments are carried out in deterministic environments to enable reproducibility, and the necessary data is versioned via DVC [Kuprieiev al. 2022].
htngss/githib.cco//eeeaaaaaaet tra.a///eeeggoeeeaeeaaaaaaaooooo

---

References
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives, 2014.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. ArXiv preprint, abs/1310.0425, 2013. URL htt.:tttttatii...ooo org. aaaaaaaaa/111 .425
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In International Conference on Learning Representations, 2023. URL https:://penrreview net/
Corum?id=SrC--ae
Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, and Francesco Locatello. ASIF: Coupled data turns unimodal models to multimodal without training. 2023.
Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights representational similar- ity in neural networks with canonical correlation. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi and Roman Garnett, editors, Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Infor- mation Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 5732-5741, 2018. URL httts::/ppoocsdddiiiii hww.ppooeeeddnnnnnr.nirprppppppp cetpuripscccppaaee httppppprr//000 2aar//208///asas a (
www.sr-bsttttcc.... html
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John E. Hoperoft. Convergent learning: Do different neural networks learn the same representations? Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:htt////ae http:///axvvvvo/ee abs/ 1511.07544
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, Califorria,, USA. volume 97 of Proceedings of Machine Learning Research. pages 3519-3351550 3529. PMLR, 2019. UtLpttpp//prooeeiingg .ml wwt.ppeessvvie v97 htt/k/onookoott///oe html
Lisa Bonheme and Marek Grzes. How do variational autoencoders learn? insights from representa- tional similarity. ArXiv preprint, abs/2205.08399, 2022. URL https:t/taatttt http::://axxvvooo www.iiowg//........oo.o..oooooot harggabb/2000 2205
08399
Anton Tsitsulin, Marina Munkhoeva, Davide Mottin, Panagiotis Karras, Alexander M. Bronstein, Ivan V. Oseledets, and Emmanuel Müller. The shape of data: Intrinsic distance for data distribu- tions. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL id-Hyebp1Hywr
Serguei Barannikov, Ilya Trofimov, Nikita Balabin, and Evgeny Burnaev. Representation topology divergence: A method for comparing neural network representations. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1607-1626. PMLR, 2022. URL / mlr press/ v162/baraniikov22..
Ivan Vulié, Sebastian Ruder, and Anders Søgaard. Are all good word vector spaces isomorphic?
In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3178-3192, Online, 2020a. Association for Computational Linguistics. doi: 10. oor.olllgooooooooooooooooo org. 2020. mmmlp-main. 257
18653/ 257.
Guillaume Lample, Alexis Conneau, Mare Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In 6th International Conference on Learning Representa- fions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https:///ppenreviww net / forum? iimummid--ii wtt.mmmii-i116iiindd
10

---

Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 991 1999. IEEE Computer Society, 2015. doi: 7298701
10.11099CVPR22015777777..11 URI. httptp/ss//aalooooo///0..0////// 7299777
Tomás Mikolov, Quoe V. Le. and Ilya Sutskever. Exploiting similaritiies among languages for machine translation. CoRR, htbs/339.4466/ 2013a. URL htt::////arxx org/ ats//30s/11000 4168
Richard Antonello, Javier S Turek, Vy Vo, and Alexander Huth. Low-dimensional struc- ture in the space of language representations is reflected in brain responses. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Ad- vances in Neural Information Processing Systems, volume 34, pages 8332 8344. Curran Associates, Inc., 2021. URL. https: httepp://procceenngs. neurips. /ttpe/ddagg..rrrrsssss/////////a 15407419922cb/5539446666661 Paper pdf
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. 2012.
Yair Movshovitz-AAtias, Alexander Toshev, Thomas K. Leung, Sergey Ioffe, and Saurabh Singh. No fuss distance metric learning using proxies. In IEEE International Conference on Computer Vision, ICCV 2017. Venice, Italy, October 22-29, 2017, pages 360-368. IEEE Computer Society, 2017.
doi: 10.1.V//C1............ URL httpt:htt////// org/ 10. 1109 ICCV. 2017 ..7
Tyler A Chang, Zhuowen Tu, and Benjamin Bergen. The geometry of multilingual language model representations. 2022.
Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Yeh-Chiang, Yehuda Dar, Richard Baraniuk, Micah Goldblum, and Tom Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. 2022.
Raghav Mehta, Vítor Albiero, Li Chen, Ivan Evtimov, Tamar Glaser, Zhiheng Li, and Tal Hassner. You only need good embeddings extractor to fix spurious correlations. 2022.
Chang Wang and Sridhar Mahadevan. Manifold alignment using procrustes analysis. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008, volume 307 of ACM International Conference Proceeding Series, pages 1 112201127. ACM, 2008. doi: 39145//1191161199999
/ttppstt/////oo.rrr/// 1145/1390156 139000111..1119977
Chang Wang and Sridhar Mahadevan. Manifold alignment without correspondence. IJCAI, volume 2, page 3, 2009.
Ivan Vulie, Sebastian Ruder, and Anders Søgaard. Are all good word vector spaces isomorphic? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3178-3192, Online, November 2020b. Association for Computational Lin- guistics. doi: 10. 10.118633v112020.coo 100.6665111/220.mmmmmmmmm..3..51 emnlp-main.257
URL https:/:::/// htttps://acllathology or20
Tomas Mikolov, Quoe V. Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation, 2013b.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. Normalized word embedding and orthogonal trans- form for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006-1011, 1, Denver, Colorado, May-June 2015. Association for Computational Linguistics. doi: 10.3115/v1/N15-111 1004 URL/attps://aclaathhoooyy htwp:/looooooo/11111111111
C. Gower. Generalized procrustes analysis. Psychometrika, 40(1):33-51, March 1975. ISSN 1860-0980. doi: 10. 10.1007/BF02291478.
Adrian Csiszarik, Peter Korosi-Szabo, Akos K. Matszangosz, Gergely Papp, and Daniel Varga. Similarity and matching of neural network representations. ArXiv preprint, abs/21 10. 14633, 2021. URL https: w/tr//axxac //arxivv.... org/ http//aa////....../ 14633

---

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017. doi: 10. 10. a a_00051. URI.
Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL wwwp::ttt///t/tt htttps://openreeiew net wtwe/foormmmddrrloeeee Aabbggg
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages
2021. URL
Michael Gygli, Jasper Uijlings, and Vittorio Ferrari. Towards reusable network components by learning compatible representations. AAAI, 3599))7600-76299 2021.
Niccolo Biondi an. Cores: Compatible representations via stationarity. ArXiv preprint. abs/2111.07632,, 2021. URL httpss//aaxxv..orr////////////
Muammer Y. Yaman, Sergei V. Kalinin, Kathryn N. Guye, David Ginger, and Maxim Ziatdinov, Learning and predicting photonic responses of plasmonic nanoparticle assemblies via dual vari- ational autoencoders. ArXiv preprint, abs/2208.03861, 2022. https: /ttpxaaaaii/.//////////////aaal/ org/abs/ abs/ 03861
Federico Bianchi an. Fantastic embeddings and how align them: Zero-shot inference i. ArXiv preprint, 21720007.44966 2020. URL /ttps:////// org httg//bbs/000...
14906
Zorah Lähner and Michael Moeller. On the direct aligmment of latent spaces. In UniReps: the First Workshop on Unifying Representations in Neural Models, 2023. URL /tRp:/ttts////ppereeee/
Alex H. Williams, Erin Kunz, Simon Kornblith, and Scott W. Linderman. Generalized shape metrics neural representations. In Advances in Neural Information Processing Systems, volume 34, 2021.
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin Yew Lin, and Deepak Ravichandran. Toward semantics based answer pinpointing. In Proceedings of the First International Conference Human Language Technology Research, 2001. URL sstps://wwxx aclweb. org/ anthology/ H01-1063
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002. URL httpp:::/s/w..ccooot aclweb. org/ anthology/
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. DBpedia: A nucleus for a web of open data. The Semantic Web, pages 722-735. Springer Berlin Heidelberg, 2007.
Zhen Wang, Xu Shan, Xiangxie Zhang, and Jie Yang. N24news: A new dataset for multimodal news classification. In Proceedings of the Language Resources and Evaluation Conference, pages 6768-6775 Marseille, France, 2022. European Language Resources Association. URL ittpp://aclaatthology htwp://aaaattollooooo///22/22220 wtt.:o//.//2........ 1.729
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text.
classification. In NIPS, 2015.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150. Portland, Oregon, USA, June 2011. Association for Computational Linguisties. URI. http: //www. acwweb. org/ http//athhooooyy1-1111
12

---

Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training traininii of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4188 Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-1 1423. URL https
Kevin Clark, Minh Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa. Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
URL httttht:t//ppreeeeeeeeee http://ppaaarreeeeeeeeeer/rrrrrr wwt.://eeeteeeeeeeeeeeeeeeee
1
Yinhan Liu, Myle Oit Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907. 1692, 2019. URL httpss://aarxvv oww.oogg/aaa..ee a/a/ /ttp//997..1111
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020. URLphttps: /ttp:://oppereieieee ntt/terumm HH1AAAEEtVS]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. ArXiv preprint, abs/2103.00020, 2021. URL httthhtt:///// :/ap::://axxxxv org/
Dongyoon Han, Sangdoo Yun, Byeongho Heo, and YoungJoon Yoo. Rethinking channel dimensions for efficient model design. ArXiv preprint, 0oss/2007.00992.. 2020. URI.https: h/apss:///axxii org/
0bbs2007.00999
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv preprint, abs/2010.1 1929, 2020. httppts://arii..rrrrrrr/
11929.
Momose Oyama, Sho Yokoi, and Hidetoshi Shimodaira. Norm of word embedding encodes in- formation gain. ArXiv, abs/2212.006663, 2022.
org/
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18123-18133, 2022.
Trene Cannistraci, Luca Moschella, Marco Fumero, Valentino Maiorca, and Emanuele Rodolà. From bricks to bridges: Product of invariances to enhance latent space communication. 2023. URL
htttps://arxiv... org/ abs/ 22100 01211
Lecun, L Bottou, Y Bengio, and P Haffner. Gradient- learning applied to document recogni- tion. Proc. IEEE, 86(1 1 1-277-23224 1998.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: novel image dataset for benchmarking machine learnin. ArXiv preprint, abs/1 hts//1008.0777,, 2017. URL http:://aaxxiv..o/abbb///sss
07747
Alex Krizhevsky, Learning multiple layers of features from tiny images. pages 32-33, 2009. URL https://www. toronto. wedu//krrz/leee seariii/eearning-feee features 2009- TR. Tdf
Ruslan Kuprieiev, skshetry, Dmitry Petrov, Pawet Redzyúski, Peter Rowlands, Casper da Costa- Luis, Alexander Schepanovski, Ivan Shcheklein, Batuhan Taskaya, Gao, Jorge Orpinel, David de la Iglesia Castro, Fábio Santos, Aman Sharma, Dave Berenbaum, Zhanibek, Dani Hodovic, daniele, Nikita Kodenko, Andrew Grigorev, Earl, Nabanita Dash, George Vyshnya, Ronan Lamy, maykulkarni, Max Hora, Vera, and Sanidhya Mangal. Dve: Data version control- git for data models, 2022. URL https: //doi. org/ 10.
7083378
13

---

F. Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel. M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and H. Duchesnay. Scikit-learn; Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
GrokAL. nn-template bootstraps pytorch projects by advocating reproducibility & best practices in deep learning, 2021. URL https://gtttt:i com/ htt//groo-oo tem/nnntemmllln Software available from http://giihub.ooomggoooooo
William Falcon and The Py Torch Lightningg team. PyTorch Lightning, 2019. URL httth:t/t/ggthiti
homm//iiaataiingAA/iiiiiitnnngg
Lukas Biewald. Experiment tracking with weights and biases, 2020, URL wwtd///www.ww.bbbbbb/../// com/
Software available from wandb.com.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, 2020. Association for Computational Linguistics. doi:
http:///0...mmmmmmmmmeoeeeee demos.6. URL https httpt:://aaaae httt:::://ccaanthll.oooo org/ 2020
emnlp-dee 6
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.1865//v/2221.. emnlp-dema 21. URL https: /tap:::://lllatholoog... org/ 2021 emnlp- demo 21
14

---

A
Additional results
present the outcomes of the multimodal experiment with an MLP employed as the classification head. The findings highlight the MLP's capability to leverage cross-modal informa- tion, leading to improved performance. However, the underlying mechanisms responsible for this enhancement remain unclear and warrant further investigation.
In Figure 9
0.8
Accuracy
0.6
0.2
No Stitching
Absolute
aftine
linear
1-orche
artho
Method
Figure 7: Cross-domain stitching on CIFARIO and grayscale CIFARIO. 84 stitched pairs (pre-trained encoder SVM classifier) for different seeds.
0.250
0.225
Reconstruction MSE
afline
linear
0.150
I-ortho
0.125
0.100
0.075
8.660
Number of anchors
Figure 8: Performance comparison (reconstruction error) of affine. linear, 1-ortho and ortho at varying anchor number on reconstruction task. Results on stitching different CIFAR100-trran AEs with 5 samplings for each anchor quantity. The naive absolute baseline is flat on 0.38 as mean.
In Tables and 6 quantitative results for stitching of MLP classifiers (differently from the main manuscript where SVMs are used) trained on top of are-trained feature extractors, with and without additional L2 normalization, respectively.
In Figures 10 and II there additional reconstruction examples with the same autoencoding setting as in the main manuscript, and with additional 1.2 normalization, respectively.
In Table more quantitative results for stitching of autoencoders, with added 1.2 normalization (at training time) to the decoders the reconstruction setting of the main manuscript.
15

---

0.46
0:3
RexNet
Encoder
Score
Scale
VIT buse 384 0.45
ViT-buse-224
0.46
90.45
ViT-small 0.40
0:33
0441
Vision
RexNet
0.38
13.46
Decoder
VEI-ResNee50 0.4 0:35
0.44 0.41 0:41 0.56
0.30
0.1
18
0.60
ViT-base-384
0.47
89.66
VIT-amall
0.40
50. 30.17
DarNNer
0.40
0:36
0.41
0.30 641 0.57 0.38 0.31 0.25
0.56
0.16
0.54
ViI-ResNer50
32.10
DarkNet
0.37
11.62
BERT-card
0011 0.27 0.32 0.28 0.11 0.30 0:40 1110 0.31
0.53
0.17
0.41
BERT mncosed
0.35 0.29
0.36 0.54 0.35 0.29 0.43 0.39 0.11
0.55
0.2
0.55
RERT-cased
0.40
15.43
BERT-uncased
0.39
14.54
0.17
1822 0.19 0.21
0.15
0.41
Electra
11.94
RoBERTs
0.51
11.06
0.30
0.25 0.31 0.28
XLM-R
0.41
18.75
XLM-R
ViEhaae-22 234
ReeNey
Vi-haee 1M
Vi-mmal
ViT-ReNNesg
DarkNar
HERT-vaed
BE3 BlarT-anchol
Eleera
RoBBrT
ALBBRT
XLM-R
BERT
Encoder
Figure 9: Performance comparison between different encoders and data modalities the N24News multimodal dataset. On the right, the accuracy of models trained end-to-end on single data modality (Score) and their average norm (Scale). On the left the stitching performance between pairs of encoders and decoder. This shows the importance of translating from good encoders, that can even improve unimodal decoder performances. Results obtained with 2000 anchors and SVD, with MLP as classification head.
7
2
O
4
3
attino
7
2
6
4
4
7
2
6
4
4
ortho
7
2
0
4 7
ortho
7
2
0
4 7
Figure 10: Reconstruction examples grouped by dataset. Each column is different image, from top to bottom: original image, absolute stitching, LSS stitching, OLSS stitching, and SVD stitching. An L2 normalization is applied to the decoder input.
Table 4: Zero-shot stitching for generation. With SVD for estimating T and standard scaling as pre-processing. An 1.2 normalization is applied to the decoder input. We report the latent cosine similarity (lcos) and MSE (Imse) between the target encoding and the translated one, but also the reconstruction MSE (rmse) between the input and the output.
MNIST
Fashion MNIST
CIFAR10
CIFAR100
Icos
Imse
rmse
Icos
Imse
rmse
Icos
Imse
rmse
lcos
Imse
imse
Abs.
0.39
0.98
0.28
0.53
0.97
0.33
0.62
1.23
0.46
0.59
1.17
0.38
affine
0.99
0.15
0.01
0.99
0.16
0.03
0.99
0.16
0.04
0.99
0.12
0.05
linear
0.98
0.17
0.01
0.98
0.18
0.03
0.99
0.16
0.04
0.99
0.13
0.05
1-ortho
0.89
0.41
0.02
0.91
0.41
0.04
0.96
0.39
0.05
0.93
0.30
0.08
ortho
0.97
0.21
0.02
0.97
0.23
0.03
0.99
0.21
0.05
0.96
0.22
0.07
16

---

5
6
7
Aba
affin
5
5
0
7
1
8
1inear
5
5
0
7
/
8
1-ortho
5
5
0
7
/
8
oztho
5
5
0
7
/
8
.
5
5
0
7
1 8
7
.4
aftino
5
5
0
7
1 /
linear
5
5
6
7
8
7
1-ortho
5
5
7
/ 8
ortho
5
5
0
7
Additional reconstruction examples grouped by dataset. Each column is different image, from top to bottom: original image, absolute stitching, LSS stitching, OLSS stitching, and SVD stitching. In the first row, additional normalization is applied on the decoder input; in the second row, an L.2 normalization is applied instead.
Figure 11:
Table 5: Cross-architecture stitching with various methods for estimating T and employing standard scaling. The stitched decoders are simple MLPs. runs for each encoder-decoder pair. (C) and (F) next to CIFAR100 indicate, respectively, coarse-graiine -nnained and fine-gra fineggrined.
Stitching
relative
1-ortho
0.22
Vison
CIFAR100-C
0.82
0.07
0.11
1:1::000 0:21
0.39 0.17
0.76 2 0.08
0.71
0.15
0.74
0.11
0.78 0.07
CLPARID0-P
0.68 1: 0.14
0.06 006:::00 0.20
0.13
0.09
0.39 0.13
0.55
0.18
0.56 2 0.17
0.62 0:12
0.87 0.02
0.22
0.02
0.83 0.05
0.80 2: 0.06
0.84 0.02
0.03
0.14
0.03
0.03
0.15
0.00
0.37
AG Newn
0.76
0.88
0.24 0.02
0.10
0.68 0.08
0.65
0.88
0.10
0.68 0.10
D8ped1a
0.64 1: 0.19
0.07 0.02
0.15 1 0.08
0.37 0.11
0.53 1: 0.19
0.44
0.21
0.56 1 0.177
IMDS
0.62 0.04
0.50 2: 0.01
0.50 0.001
0.39 0.04
0.58 0.04
0.57 2 0.04
0.60 0.04
Table 6: archtterc stitching with various methods for estimating T and applying L2 as normalization. The stitched decoders are simple MLPs. 5 runs for each encoder-decoder pair. (C) and (F) next to CIFAR100 indicate, respectively, grained and fine-grained.
Ststching
CIFAB10
0.95
0.03
0.16
0.22
0.73
4
0.93 0.04
0.89
0.89 +
0.93 0.04
CTPAR100-C
0.82 0.07
0.11
0.21
0.39
0.07
0.75
0.71
0.78 0.06
CIFARA00-P
0.88 + 0.14
0.06
0:20
0.13
+: 0.09
0.60 2 0.12
0.57
0.18
0.54 - 0.18
0.61
0.12
12115T
0.87 + 0.02
0.20 0.14
0.64
4 0.12
0.02 0.80
0.79 + 0.09
0.83 4 0.05
0.84 à: 0.02
TREC
0.07
0.15 0.04
0.06
0.27 0.10
0.06
Bous
0.08
0.24
0.08
0.10
DEpodia
0.64
0.19
0.07
0.02
0.15 1: 0.08
0.55 0.19
0.53 0.21
0.51 1 0.18
0.49 0.15
IMDS
0.62 + 0.04
0.50 0.01
0.50 + 0.01
0.60 = 0.04
U.58 0.04
0.59 4: 0.04
0.59 0.04
A.1 Scale invariance
In this section, we delve into the concept of scale invariance in neural networks and its implications for model compositionality. We start by focusing on the effect of rescaling operations on the latent input encodings and demonstrate that, by construction, certain classifiers exhibit ceale-invariance properties without the need for additional priors. Then, by examining the behavior of networks when subjected to a specific type of input manipulation, rescaling injection, we aim to demonstrate the robustness and versatility of neural networks handling different scales of input data. As illustrated in the main manuscript, this is key advantage in improving the adaptability of our method.
17

---

The softmax function, commonly used in neural classifiers, is known to be temperature-conntrllee variant of the maximum function:
c+
softmax(2); = Σ e 4
(2)
This means that the softmax temperature can be used to control the level of confidence of the classifier's ' predictions. In this study, we show that a similar effect can also be achieved by rescaling the latent encodings given as input to trained (and frozen) classifier.
In order to demonstrate this, we first note that the rescaling factor, a, can be factored out of the matrix multiplication in the Linear layers of the classifier. This can be represented mathematically as: y = a Wx + b, where x is the input latent encoding, W is the weight matrix, b is the bias vector, a is the rescaling factor, and y is the output of the linear layer. This implies that the rescaling operation can be "pushed through" the classifier without affecting its final prediction as it becomes equivalent to some temperature value applied at the softmax level.
Furthermore, we investigate the effect of rescaling when non-linear activation functions are involved and posit that as long as the function has monotonic interval, if we rescale all the dimensions by an amount similar to the mean scale of the encodings on which the classifier was trained, we end up in the monotonic interval, without losing the scale-invariance property.
In summary, our study provides empirical evidence that neural classifiers that utilize the softmax activation function can, in practice, maintain their encodings are rescaled. This property essential our method, it allows us ignore the exact scale when decoding toward an L2-normalized absolute space.
properties when the input latent
Pre-trained models and scale-invariance We observed that large pre-trained models, such as transformers and resnets, are robust internal rescaling of the encodings. Although we do not have a strong theoretical explanation for this phenomenon, we hypothesize that normalization layers and the linear separability of the information encoded in the angles instead of the norms may play a significant role. In Figure [2] we demonstrate the invariance a large transformer exhibits when the rescaling injection is applied at different layers: surprisingly, when the rescaling surpasses certain threshold, the performance difference becomes negligible. These results further emphasize the robustness of these pre-trained trainr models the rescaling injection and suggest that the scale of the embedding is not a critical factor in their performance.
Rescale Injection We define the rescaling injection as the operation of artificially altering the scale of the features produced specific layer of the network. This is achieved by normalizing the embeddings to unit norm and then rescaling them by a factor of a. By varying the value of o, we can observe how the network's 's performance is affected different scales. Through this empirical analysis, we aim to provide insight into the scale invariance properties of neural networks and their potential for use in model compositionallty.
In Figure 13 we present experimental results investigating the scale invariance properties of neural networks. We trained simple multi-layer perceptrons (MLPs) composed of two hidden layers, with no normalization layers, using encodings produced by the Clip Vision transformer (clip-vit-base- base--atcch3))
the CIFAR100 (fine) dataset. The MLPs were evaluated using dif- ferent activation functions: cosine (blue), tanh (orange), and ReLU (green). The rescaling injection technique was applied directly to the input embeddings, rescaling them by a.
We can observe that the scale of the embeddings does not significantly impact the MLPs' performance when using monotone activation functions that do not flip signs. This is non-trivial result, as the nonlinearity of the activation function, the presence of bias terms b, and the absence of normalization layers make it difficult to predict the effect of an input rescaling on the performance of the network. It is particularly interesting to see that the cosine activation function shows an oscillatory performance, comparable to the original embeddings when rescaled by the mean embeddings scale (vertical red line) or its opposite since is symmetric.
Our findings indicate that, surprisingly, even the internal layers of large deep learning models exhibit a positive scale invariance, as illustrated Figure The underlying mechanism for this behavior not straightforward, but we hypothesize that it may result from the interplay between various factors,
18

---

texco
lerdn
teior
lextg
leebo
555
9,00
Rescaling factor a
7,16
6.21
cs
Accuracy
4,79
4.50
3,84
CA
2.42
1.85
1,47
4.87
0.88
1.00
-117
1.67
1.83
Rescaled layer
Scale invariance of RoBERTa according to the performance of downstream classifier trained on the encodings of the last attention layer. At each layer (with 0 being the embedding layer and the output one), one for each run, we rescale the encodings by the specified a and measure its effect on the final accuracy. The performance without any rescaling is 0.92.
Figure 12:
Accuracy
nola
0.2
mean daea stale
0.0
le+03 letos
Ie+07 1e+08
Rescale Factor
Figure 13: Performance comparison of three Multilayer Perceptrons (MLPs) with different activation functions, namely cosine (blue), ReLU (orange), and tanh (green) at different rescaling factors a. The ReLU and tanh MIPs exhibit scale invariance, while the cosine activation function is only invariant on the mean data scale and its periodic cycles.
such the choice of activation function, the of normalization layers, the optimization objective and regularization techniques employed during the training phase. Further research needed to understand and explain this phenomenon fully.
B Implementation Details
All the experiments were conducted using a machine equipped with an Intel Core i7-9700k CPU, 64 GB of RAM, and an NVIDIA 2080TI GPU.
Decoder structure The full implementation details can be found in the attached code. The various experiments can run by their corresponding notebook, while the source code for the package they built can be found under the "sre" folder.
19

---

Autoencoding. Since the autoencoders were used only on image data, the architecture was simple sequence of convolutions (in the encoder part) and deconvolutions (in the decoder part). Each interleaved with nonlinear activations.
Classification. The main manuscript refers to "SVM" as the standard SVM implementation in scikit-learn [Pedregosa al., 201T], with default parameters. The experiments with "MLP" as a classifier refer to simple stack of linear layers, interleaved by nonlinear activations.
Software and Technologies The research of this study was facilitated by the use of various technologies and tools, which include:
NN- Template [GrokA] 202T], was used to kick-start the project while also ensuring best practices were adhered to;
DVC [Kuprieiev et al. 2022 implemented for data versioning:
PyTorch Lighining [alcon and The Py Torch Lightning team 2019], contributed to main- taining the integrity the results and promoting clean, modular code;
Weights and Biases [Biewald 2020), were employed for logging experiments, running comparisons over extensive sweeps, and sharing models;
Transformers by HuggingFace [Wolf et al. 2020), provided pre-configured transformers for processing both image and text data;
Datasets by HuggingFace (Lhoest et al. 2021] facilitated access to majority of NLP datasets and ImageNet for computer vision purposes;
Pre-trained encoders All the pre-trained encoders used come from HuggingFace and are listed in Table 7 They are various both terms of architecture and encoding size.
Table HuggingFace models used encoders (feature extractors) the various experiments, with their encoding dimensionality.
Modality
HuggingFace model name
Encoding Dim
bert-base-cased
768
bert-base-uncased
768
Language
google/deccraaass-diicinnaaa
768
roberta-bare
768
albert-base-v2
768
xlm-roberta- base
768
ppenai/llip-vit-bas--pttee
768
rexnet_100
1280
cspdarknet53
768
Vision
vit vit_sl 6acalll_atchhh____ 224
384
vit base_aaaeettee @base_patchl6 224
768
vit vit_base_patch16__
768
vit
768
openai/llip-vit-bas--ppp pattt
768
20

---

Table 8: aroostrrchie stitching for reconstruction tasks. different seeds, 2 different bottleneck sizes (250, 500) for the same architecture. Average over all combinations. 500 anchors used and standard scaling normalization. The naive absolute baseline impossible compute due the dimensionality mismatch.
MNIST
Fashion
MNIST
CIFAR10
CIFAR100
Icos
Imse
imse
Icos
Imse
rmsse
Icos
Imse
imse
lcos
Imse
rmsse
affine
0.95
0.09
0.02
0.95
0.09
0.04
0.98
0.06
0.05
0.98
0.07
0,06
linear
0.64
1.00
0.11
0.66
1.10
0.16
0.77
0.60
0.16
0.78
0.52
0.16
1-ortho
0.87
0.16
0.03
0.89
0.14
0.06
0.95
0.12
0.08
0.95
0.13
0.08
ortho
0.91
0.14
0.03
0.92
0.13
0.06
0.96
0.12
0.09
0.96
0.12
0.09
21


Reviews:
[{"review_id": "UzAAKhWVhO", "reviewer": "Reviewer_ivQN", "summary": "This paper show the latent space of different pretrained models can be translated between each other with simple transformations by using anchors. By using this method, a variety of encoders architectures are able to be cross-stiched to different classifier heads or within autoencoding models.", "questions": "None", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "This paper tackles an interesting problem for how to semantically align different trained latent spaces.\n\nThe paper is a natural extension to Moschella et. al 2023, going from just relative encoding to full latent space transformations.\n\nThe results are clear in showing zero-shot stitching is possible without retraining the decoders", "weaknesses": "It wasn't clear on my first read of the paper that this work is a direct follow up to Moschella et al. 2023. Without that context, the paper was very difficult to read the first time. After reading Moschella et al., I was better able to follow this proposed work. Multiple details are missing, such as the construction of the anchor set, which are glossed over in this work and caused confusion on the zero-shot nature of this work. Another major missing key detail is that prior work's decoders must work on the relative embedding, this crucial detail was not clear until after multiple read throughs. This improvement should be front and center.\n\nI find it surprising that tables 1 and 2 do not report Moschella's results. I realize that the decoders must be trained with the relative embeddings, but the lack of comparison feel likes an obfuscation rather than a highlight of the difference between methods.\n\nOverall, I find the clarity of the paper to be a major hinderance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's insightful feedback, highlighting important aspects warranting clarification and enhancement in our work. In light of their review, we have identified these steps to improve our work:\n\n- Emphasize the relationship with RelRep in the Preliminaries section as already stated in the **general response** to better contextualize the foundations of our work;\n- Add a direct comparison with the RelRep method in both Table 1 and 2. A preview of the new results to be added to Table 1 can be found in the rebuttal PDF.\n- The anchor set construction procedure, aligned with the standard RelRep sampling (a uniform random sampling over the training set), will be incorporated into the Preliminaries section. This will both elucidate the process and draw a parallel with the original work's findings in Moschella et al..\n- We will offer a more transparent depiction of our work's \"zero-shot\" nature. As correctly highlighted by the reviewer, our distinction from RelRep lies in removing the requirement for relative training of the decoder. Instead, we achieve (better) stitching performance by solving a closed-form problem. This methodology will be emphasized to highlight our innovation.\n\nWe hope to have clarified the reviewer's concerns, and we respectfully remark that R2, R3, and R4 didn't consider clarity a significant weakness, assigning higher scores to it. Nevertheless, we remain available to discuss any further concerns or questions."}, {"review_id": "tMQsOsVE5X", "reviewer": "Reviewer_NjdJ", "summary": "This article proposes a method of latent space translation through semantic alignment, using the similarity of latent spaces learned by different neural models on semantically similar data. The method allows for direct conversion of learned representations between different pre-trained networks, and achieve zero-shot stitching of encoders and decoders without additional training. The method is widely validated in various experimental settings, tasks (classification and generation), and modalities, demonstrating the ability to zero-shot stitch neural models under different architecture and modality changes.", "questions": "1. How much performance difference is there between the proposed zero-shot stitch method and concatenation followed by finetuning? \n2. How is the selection of anchor points determined, and how much impact does different selection have on performance? \n3. In experiments, zero-shot stitch is performed between different frameworks on the same dataset. So, can this method be effectively used for neural frameworks trained on different domain datasets?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes a method for the zero-shot stitching between encoders and decoders on various tasks, enabling seamless integration without the need for additional training.\n2. Compared to previous methods, this article expands to different latent space dimensions, architectures, and modalities. \n3. Extensive set of experiments have been done to show the possibility to zero-shot together neural models across different architectural and modalities.", "weaknesses": "1. In the experiment, only the most basic abs method is compared, but no comparison is made with other methods.\n2. In the generation task, zero-shot stitching is performed under the same model framework and training data. Whether it is more practical to migrate between different model frameworks and training data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable input, which has prompted us to further enhance the soundness of our work.  Here, we provide deeper clarification and offer additional insights regarding:\n\n**Additional Comparisons**: we will add several columns to our tables (Table 1 and Table 2): **i)** performance of relative decoders; **ii)** stitching performance of relative decoders (to be used as a direct comparison with the Moschella et al. method); **iii)** stitching performance when optimizing an affine transformation with SGD (using the same anchor set as the other methods), namely “Linear”, as a standard and robust alternative stitching method [a, b]. We remain open to additional baseline suggestions.\n\n[a] Lenc and Vedaldi “*Understanding image representations by measuring their equivariance and equivalence.*“CVPR 2015.\n\n[b] Moayeri et al. “*Text-To-Concept (and Back) via Cross-Model Alignment*”*,* ICML 2023\n\nA preview of the new Table 1 can be found in the rebuttal PDF.\n\n**Cross-Architecture stitching (generation)**: in the rebuttal PDF, we show a stitching experiment between the same autoencoder architecture used throughout the whole paper but with varying training seed (2 different ones) and bottleneck size (250 and 500) on multiple datasets. The performances are similar to the ones in Table 3 of the main manuscript, showing that cross-architecture stitching is possible even between autoencoders and has good performances.\n\n**Fine-tuning (Question 1)**: it is unclear to us the link between fine-tuning on a concatenation of the two representations and stitching. What would be the downstream training objective? Could the reviewer kindly expand on this? We’d be open to adding it as an alternative/baseline method.\n\n**Anchor selection (Question 2)**: Our approach adheres to the standard RelRep setting, where anchor points are uniformly drawn from the training samples. Notably, Moschella et al. indicate that more intricate policies do not yield discernible improvement, a finding supported by their supplementary analysis in the appendix (Moschella et al., Section A2). We will further clarify this in the Preliminaries section.\n\n**Multi-domain (Question 3)**: if we interpreted the question correctly, the reviewer is asking whether our method can be applied to translate across samples represented in different domains. We see our “Cross-modality” section (4.2), where we translate from the text domain to the image one and vice-versa, as a generalization of this idea, given that a modality change is supposedly stronger than a domain one. Nevertheless, we present in the rebuttal PDF and will incorporate in the paper a new experiment where we stitch between classifiers trained on different versions of the CIFAR10 dataset: the standard one and a grayscale version. The performances are comparable to the ones in Table 1 of the main manuscript (CIFAR10 row), showing that cross-domain stitching is possible and with good performances.\n\nWe thank the reviewer again for their feedback and remain available to discuss any further concerns or questions."}, {"review_id": "yXqZbDqPel", "reviewer": "Reviewer_rL3F", "summary": "In this paper the authors propose to translate latent space using an effectively angle-preserving affine transformation learned using pairs as anchors. Experiments are conducted on a wide range of tasks showing the okay performance by the swap-in latent space of embeddings/features.", "questions": "N/A", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Strengths**\n\nThe authors have taken an extensive suite of experiments encompassing a diverse set of tasks, including cross-training, cross-architectures, and several downstream tasks. The experimental design demonstrates a comprehensive approach in assessing the proposed method *per se*.\n\nFurthremore, this work shows that aligning two latent spaces with zero-shot stitching is poissible without any retraining. This is an important observation and the proposed method could benefit a wide range of downstream tasks.", "weaknesses": "**Weaknesses**\n\nThe following concerns regarding the design of the transformation and its intrinsic limitation warrant consideration.\n\n- **Assumption of Linearity and Lack of Analysis in Generative Model**:\n\n\nThe proposed method is effectively an angle-preserving affine transformation, which fundamentally assumes linearity of mapping between two latent spaces (of embeddings and/or features). While \"linearity of the transformation between seemingly different latent spaces has a theoretical foundation in research on identifiability in neural models,\" as the author pointed out in the rebuttal, in the context of generative model, the efficacy is still likely contingent on the extent to which two latent spaces exhibit linear mapping, because a good generative model may fill the slight mismatch with its own capacity. \n\nIn detail, it is likely that an affine transformation may yield only a coarse mapping between two latent spaces, and may not capture fine-grained relations (consider style transfer in generative models).\nThus, the good performance in classification tasks may thus be attributed to the capacities of the downstream models rather than the transformation itself. This is a critical assumption that necessitates further scrutiny, but the submission does not provide an in-depth analysis of this assumption.\n\nIn the current form of the manuscript, for this regard only an autoencoding on MNIST/Fashion MINIST/CIFAR is shown. It does not have a qualitative comparison with the reconstruction from VAE itself, nor quantitative/qualitative results with other stitching and zero-shot methods.\nThe lack of such comparison can unfortunately weaken the justification of the proposed method.\nAdding such comparison is thus suggested. Optionally, conducting such comparisons on common image datasets of higher resolution, like CelebA, ImageNet, (with more powerful VAE models) may provide more insight in this regard, since it may potentially highlight the capacity of VAE models and the quality of latent space alignment.\n\n(**After author response**: See the discussion below) \n\n- **Ambiguity in Technical Contribution**:\n\nThe technical novelty and significance of the proposed method remain unclear. The method effectively is an angle-preserving affine transformation. While this per se is not necessarily an issue, the missing studies mentioned above make it hard to assess the impact and implications of the proposed method.\n\n(**After author response**: This is not a major issue anymore, see the discussion below) \n\n- ** Position of Works within the Broad Field**:\n\nLatent space matching has been the subject of extensive research, albeit in varying contexts such as text, image processing, and generative models.\nA few (it's even not an attempt to cover the overall picture of this widely studied topic) works below in References that share technical similarity with the proposed method  are cited in the References.\nHowever, the submission does not include a substantive comparison or discussion with a wide body of works in this field, mkaing it challenging to ascertain the relative standing and uniqueness of the proposed method.\n\n(**After author response**: This is not a major issue anymore, see the discussion below) \n\n**References**:\n\n-\nSet Prediction in the Latent Space https://proceedings.neurips.cc/paper/2021/hash/d61e9e58ae1058322bc169943b39f1d8-Abstract.html\n- Formality Style Transfer with Shared Latent Space https://aclanthology.org/2020.coling-main.203.pdf\n- Homomorphic Latent Space Interpolation for Unpaired Image-To-Image Translation https://openaccess.thecvf.com/content_CVPR_2019/html/Chen_Homomorphic_Latent_Space_Interpolation_for_Unpaired_Image-To-Image_Translation_CVPR_2019_paper.html\n\n\n\n**Updates**\n\n- The previous version of this review mistakenly refer to the page limit. As the authors and the AC pointed out, this is NOT the case and has been corrected.\n\n- In Weakness, update the Assumption of Linearity to focus on the generative model.\n\n- Added reference to discussion below.\n\n- The rating has been increased.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and comments. We appreciate the opportunity to address the concerns they've raised and offer clarifications on the following points:\n\n**Page Limit**: As per the **official 2023 Call for Paper**, submissions are indeed limited to **nine** content pages and **not eight**. Therefore, we don’t see any reason to report the paper to be desk-rejected.\n\n**Assumption of Linearity and Lack of Analysis**:\n\n- The assumption of linearity of the transformation between seemingly different latent spaces has a theoretical foundation in research on identifiability in neural models [a, b, c,e,f]. In particular, in [a], it is proved that representations learned by a large family of supervised and generative models are identifiable up to a linear transformation. We will incorporate these additional references in the related works section. Our work is intrinsically centered around exploring the emergence of these properties in practical settings within deep neural networks (DNNs).\n- While we acknowledge the broader context of latent space matching, the references the Reviewer kindly suggested involve the introduction of constraints, penalties, or adjustments *during training*, which lie outside the scope of our investigation.\n- Our focus, instead, aligns closely with the works highlighted in the \"Stitching and Zero-Shot\" section of our manuscript (Section 2). These works exploit the linearity assumption and focus on finding a transformation that aligns the latent spaces ex-post. For example, in a very recent work, Moayeri et al. [d] further prove that these emerging properties exist, can be exploited, and deserve more attention from the community.\n\n[a] Roeder, Geoffrey, Luke Metz, and Durk Kingma. \"On linear identifiability of learned representations.\" *ICML*, PMLR, 2021.\n\n[b] Khemakhem, Ilyes, et al. \"Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica.\", NeurIPS, 2020\n\n[c] Willetts, Matthew, and Brooks Paige. \"I Don't Need u: Identifiable Non-Linear ICA Without Side Information.\" ArXiv\n\n[d] Moayeri et al. “*Text-To-Concept (and Back) via Cross-Model Alignment*”*,* ICML 2023\n\n[e] Hyvarinen et al \"Nonlinear ICA using auxiliary variables and generalized contrastive learning.\"  PMLR, 2019.\n\n[f] Sorrenson,et al. “Disentanglement by nonlinear ica with general incompressible-flow networks (gin)” 2020\n\n**Coarse-only mapping:** We would like to emphasize that we worked with both classification and generation as decoder tasks, observing good performance even in the latter scenario. This contrasts with the following comment: “*an affine transformation may yield only a coarse mapping between two latent spaces, and may not capture fine-grained relations*”. We will be happy to include other datasets/tasks to broaden our study, following any suggestions from the Reviewer."}, {"review_id": "OLjuh9N834", "reviewer": "Reviewer_zvRG", "summary": "This work proposed a method to translate learned representations between two pre-trained networks, using surprisingly simple transformations. The method is demonstrated on models with different architectures, trained on different modalities and across different tasks.\n\n", "questions": "See questions mentioned in the Weaknesses section.", "limitations": "Yes.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well written and easy to follow. \n2. To my understanding, the discussed setting where models are ad-hoc zero-shot stitched, as opposed to being trained to have compatible representations is original. This distinction is important as it makes the approach applicable for the myriad of pre-trained models that already exist.\n3. The method is surprisingly simple and yet produces very convincing results as demonstrated through sufficient evaluation.\n", "weaknesses": "1. The exact setting leading to Figure 2 is not clear. Caption mentions results being produced on “CIFAR-100”. However, I was not able to find the entire setting. What two models were used? Are both trained on CIFAR-100? What are the architectures used? Are reported results averaged across multiple settings? \n2. Also regarding Figure 2 - the two trends not being monotone is surprising to me. Especially the very clear outliers. I suppose this behavior might occur if each point represents a single experiment and some randomness might affect performance. In that case, it would probably be better to average these results across several experiments as done in other tables in the paper. Can the authors explain this behavior?\n\n\nMinor comments:\n\n1. Typo line 49 - mphasizing -> emphasizing.\n2. Notation line 139 - $\\mu$ is used in equation 3 but $\\bar{x}$ in text.\n3. Clarity line 167 - I suggest the “naive absolute baseline” should be concisely explained.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our work, providing valuable feedback, and firmly pushing for its acceptance.\n\n- **Figure 2 (Additional Details)**: In response to the concern about missing details, we will provide comprehensive information in the appendix. For the classification aspect (left), the results are derived from averaging across all potential stitching combinations (pairs) among the utilized pre-trained encoders and their classification heads (SVMs) trained on CIFAR100. With a total of 7 distinct encoders (refer to the appendix, Table 4), this results in a pool of 42 pairs for each \"number of anchors\" configuration. Regarding the generation aspect (right), we train vanilla convolutional autoencoder pairs on the CIFAR100 dataset for each \"number of anchors\" configuration. No normalization is applied to the dataset images, and each autoencoder has ~2 million trainable parameters and a size 500 bottleneck.\n- **Figure 2 (Trends)**: The presence of outliers and the non-monotonic pattern observed in the lstsq curve can be attributed to a confluence of factors, including numerical instability and the PyTorch **driver selection** policy. This policy adapts based on the condition number of the matrix. In particular, we observed that outliers tend to emerge as the amount of anchors approaches the number of dimensions, leading to a square matrix.\n- **Naive Absolute Baseline**: this baseline entails the direct stitching between models considering their vanilla representations without any transformation, to verify any pre-existing compatibility and establish it as a lower-bound. We agree it deserves further explanation, we will integrate this description in the preliminaries, since it is also used in Moschella et al..\n\n**Figure 2 Updates**\n\nIn response to these observations, we systematically repeated all classification experiments using five distinct random seeds, influencing initialization and anchor selection. Additionally, we have transformed the plot into a violin chart format to present the variance clearly. We will do the same for the generation experiments. The updated visualization we will include in the paper can be found in the rebuttal PDF."}]

Author rebuttal:
We sincerely appreciate the constructive feedback provided by the reviewers. We would like to address the relationship between our work and the concepts presented in "relative representations" (Moschella et al.). While we draw upon the foundational principles of Moschella et al., it is essential to clarify that our work takes a different path. Therefore, we do not consider it a direct extension.

- Moschella et al. showed that when different latent spaces share semantics: 1) a simple angle-preserving transformation connects them; 2) a reduced set of key points (anchors) can be used to reconcile them into a **new, distinct (relative) representation**.
- Instead, we directly estimate the **transformation** from one space to another using provided anchor points without relying on an auxiliary representation. A key advantage of this approach is that it circumvents the necessity of training the decoder on relative representations, simplifying the integration of diverse models arbitrarily trained.

Considering the reviewers' valuable input on clarifying this aspect to strengthen its message, we commit to adding a new section to specify the connection to Moschella et al. better. We'll present this relationship accurately and concisely, expanding the “Preliminaries” section, utilizing available space within the nine-page limit.

Meta-review: All reviewers agree that the paper extends existing work nicely, enabling post hoc stitching of trained models. Initial concerns were mainly around comparisons to existing works, presentation clarity and linearity assumptions on the feature space. Authors have addressed the first 2 concerns in the rebuttal. The 3rd concern seems to be not specific to this work, but more broadly to these line of works, which needs to be addressed in future works. Overall reviewers are happy with author's response and I am happy to suggest acceptance. I encourage authors to improve the presentation of the final version as per reviewers suggestions.
Decision: Accept (poster)