Title: Latent Space Translation via Semantic Alignment
Venue: NeurIPS 2023
Accepted: yes
Keywords: latent space translation

relative representation

Procrustes analysis

zero-shot

stitching

latent communication

representation learning

manifold alignment

multimodal
Abstract: While different neural models often exhibit latent spaces that are alike when exposed to semantically related data, this intrinsic similarity is not always immediately discernible. Towards a better understanding of this phenomenon, our work shows how representations learned from these neural modules can be translated between different pre-trained networks via simpler transformations than previously thought. An advantage of this approach is the ability to estimate these transformations using standard, well-understood algebraic procedures that have closed-form solutions. Our method directly estimates a transformation between two given latent spaces, thereby enabling effective stitching of encoders and decoders without additional training. We extensively validate the adaptability of this translation procedure in different experimental settings: across various trainings, domains, architectures (e.g., ResNet, CNN, ViT), and in multiple downstream tasks (classification, reconstruction). Notably, we show how it is possible to zero-shot stitch text encoders and vision decoders, or vice-versa, yielding surprisingly good classification performance in this multimodal setting.

Latent Space Translation via Semantic Alignment

Abstract

While different neural models often exhibit latent spaces that are alike when exposed to semantically related data, this intrinsic similarity is not always immediately discernible. Towards a better understanding of this phenomenon, our work shows how representations learned from these neural modules can be translated between different pre-trained networks via simpler transformations than previously thought. An advantage of this approach is the ability to estimate these transformations using standard, well-understood algebraic procedures that have closed-form solutions. Our method directly estimates a transformation between two given latent spaces, thereby enabling effective stitching of encoders and decoders without additional training. We extensively validate the adaptability of this translation procedure in different experimental settings: across various trainings, domains, architectures (e.g., ResNet, CNN, ViT), and in multiple downstream tasks (classification, reconstruction). Notably, we show how it is possible to zero-shot stitch text encoders and vision decoders, or vice-versa, yielding surprisingly good classification performance in this multimodal setting.

1 Introduction

Representation learning [Bengio et al., 2014] is a fundamental paradigm in the field of artificial intelligence, aimed at uncovering the underlying structure of complex data. One of the main goals of representation learning is to discover a robust representation of the data that is insensitive to certain transformations of the input. The Manifold Hypothesis [Fefferman et al., 2013] posits that real-world data lies on a low-dimensional non-linear manifold embedded in a high-dimensional space. Yet, a complication arises in modeling these non-linear manifolds: the learning process is usually influenced by stochasticities in the training dynamics and extrinsic factors that do not pertain to the data's core attributes, resulting in different representations for samples expected to be similar (e.g., different views of the same object, multiple translations of the same sentence, or even the exact same input sample). This is critical as it hinders knowledge transfer between these networks. Recently, the concept of relative representations [Moschella et al., 2023] has been proposed as a method for zero-shot communication between latent spaces that is invariant to these extrinsic factors. The idea is that latent spaces of neural networks trained on comparable data can be projected into the same relative space, derived from the distances between the data points. One of the main contributions of relative encoding is that it shows how the signal encoded in the angles with respect to a reduced set of data points (called anchors ) is enough to capture the intrinsic shape of the latent space, reaching results on various benchmarks comparable to those using the original (absolute) encodings. As a consequence, they empirically demonstrate that different latent spaces that share the same data semantics (i.e., different representations of the same high-level concepts, such as images and their captions), mostly differ only by an angle-preserving transformation.

Building on this intuition of the existence of a relatively simple transformation, we show the effectiveness and applications of directly translating between different latent spaces - provided that a partial (and possibly sparse) correspondence between data points is given. Remarkably, the process of seamlessly combining different neural networks, pre-trained on diverse datasets, modalities, architectures, domains and downstream tasks, proves unexpectedly straightforward. For instance, we show how it enables the ability to effectively integrate any pre-trained text encoder with any image classification head, and vice versa, without requiring any additional re-training or assumptions (indeed, Moschella et al. [2023] assumes the decoders are trained on relative representations). The method difference is emphasized in Figure 1. While zero-shot stitching with relative representations assumes the use of a single decoder specifically trained on a relative space, our method permits the reuse of the decoders originally trained on the absolute spaces.

Our main contributions can be summarized as follows:

We explore the direct translation between latent spaces of distinct neural networks to enable latent communication . In particular, leveraging a semantic correspondence in the data, we directly translate for the first time across different trainings, architectures, and modalities. Notably, we obtain excellent stitching performances even in cross-modal settings, where we apply arbitrary text classifiers on top of pre-trained image encodings (and vice-versa).

We show that different downstream tasks, namely classification and generation, require modeling different transformations to obtain the most out of the translation between their latent spaces;

2 Related Works

Representations similarity Recent years have witnessed a growing consensus among researchers in the deep learning community that effective neural networks tend to learn similar representations for semantically similar data, regardless of the architecture, task, or domain in which they are applied. This idea is supported by a plethora of empirical studies [Moschella et al., 2023, Norelli et al., 2023, Morcos et al., 2018, Li et al., 2016, Kornblith et al., 2019, Bonheme and Grzes, 2022, Tsitsulin et al., 2020, Barannikov et al., 2022, Vuli´ c et al., 2020a, Lample et al., 2018, Lenc and Vedaldi, 2015, Mikolov et al., 2013a, Antonello et al., 2021, Bengio et al., 2012, Movshovitz-Attias et al., 2017, Chang et al., 2022] and the phenomenon is particularly pronounced for large and wide models [Somepalli et al., 2022, Mehta et al., 2022]. Nevertheless, despite this intrinsic similarity, latent spaces can still exhibit extrinsic variations. Our work analyzes the possibility of translating these spaces from one to another, linking these extrinsic variations to different classes of transformation.

Manifold alignment Procrustes analysis has been instrumental in the alignment of latent spaces in deep neural networks [Wang and Mahadevan, 2008, 2009], particularly in Natural Language Processing (NLP) where it is well-known that latent spaces of different languages are isomorphic [Vuli´ c et al., 2020b] and can be effectively aligned [Mikolov et al., 2013b, Xing et al., 2015]. Rooted

Source

X

Target

Y

︸

︷︷

︸

Normalization

︸

︷︷

︸

De-normalization

Centering

Scaling

T

De-Scaling

De-Centering

≈

2

2

2

2

2

2

2

1

1

1

1

1

1

1

0

0

0

0

0

0

0

-1

1

-1

-1

1

1

-2

-2

0

N

-2

0

2

-2

0

2

-2

2

0

2

-2

0

2

-2

0

Z

2

0

2

in shape analysis, this method efficiently uncovers correspondences between latent spaces of different models through the estimation of an optimal orthogonal transformation [Gower, 1975]. Previous works largely exploit Procrustes analysis to align latent spaces originating from models of the same architecture [Csiszarik et al., 2021], such as multi-lingual FastText embeddings [Bojanowski et al., 2017, Smith et al., 2017]. Instead, in this work, we extend the application of Procrustes analysis to network stitching in new domains, architectures, and even modalities for multiple downstream tasks.

Stitching and zero-shot Model stitching, which involves the combination of different neural networks to create a new model, has been a topic of active research in the field of representation learning. A key concept in this area is that of relative representations [Moschella et al., 2023, Norelli et al., 2023], which enables zero-shot stitching between different neural networks trained on semantically similar data. While this approach assumes the use of decoders trained on relative representations , our work removes this constraint by introducing a zero-shot mechanism for translating one absolute space to another without relying on a shared (relative) representation, enabling the stitching of arbitrarily trained models, further generalizable by assuming only the positive scale invariance of their decoder part. Previously, trainable stitching layers [Lenc and Vedaldi, 2015, Bansal et al., 2021, Csiszarik et al., 2021] have been introduced to allow for the combination of parts of different networks or to verify statements regarding latent space similarity. Other works [Gygli et al., 2021, an, 2021, Yaman et al., 2022, an, 2020] have proposed alternative methods for producing directly compatible and reusable network components without specific stitching layers. Here, we sidestep the need to create a new compatible representation and instead focus on obtaining a direct transformation to map from a source space to a target one to enable seamless network stitching. Concurrently to this work, a similar approach by [Lähner and Moeller, 2023] also targeted the direct alignment of representational spaces, focusing on the compatibility of models trained end-to-end.

3 Method

3.1 Preliminaries

Relative representation is a framework introduced in Moschella et al. [2023], which enables latent spaces of arbitrary neural models to communicate with each other. This is obtained by projecting the latent spaces into a common one, transitioning from an absolute coordinate frame to a relative space : each sample is represented as a function of a set of fixed samples denoted as anchor set . Specifically, the new representation is computed by independently projecting each sample point x in the latent space X ∈ R n × d , into the anchor set A X ⊂ X . Formally, this is represented as

where X rel ∈ R n × k , X abs ∈ R n × d and A X ∈ R k × d Samples in X and in A are rescaled to unit norm, i.e. x = x ∥ x ∥ 2 ∀ x ∈ X and a = a ∥ a ∥ 2 ∀ a ∈ A X .

We assume to have access to subsets A X ⊂ X and A Y ⊂ Y , with X and Y being the data distributions, and that there exists a correspondence Γ : A X ↦→A Y between these two sets of parallel anchors . Parallel anchors act as a "Rosetta stone" [Norelli et al., 2023], meaning they establish a semantic correspondence between their respective spaces: an anchor sample in the first set represents the same high-level concept as its counterpart in the second set. This allows stitching together components of different models: i.e., merging independently trained encoder and decoder modules from different networks. The relative projection will map latent spaces into the same one as long as the core assumption that they differ by an angle-preserving transformation is satisfied. However, in

order to perform the stitching procedure in Moschella et al. [2023], decoders must be trained from scratch at least once to process samples in this shared relative space.

In this work, we overcome this need by substituting the costly retraining procedure with an efficient strategy to directly estimate the transformation necessary to map between spaces. Moreover, we relax the "angle-preserving" constraint by allowing for a broader class of transformations obtained via robust, closed-form algorithms.

3.2 Latent Space Translation

Consider two latent spaces, X ∈ R n × d 1 and Y ∈ R n × d 2 . Our objective is to estimate the transformation T that translates X into Y : Y = T ( X ) , exploiting the semantic alignment between the two spaces. Throughout this work, we identify two main steps in the translation process: pre-processing the spaces and estimating the transformation T , as outlined in Figure 2.

Pre-processing Generally, the two spaces may have different dimensionalities - in those cases, we zero-pad the smaller one to match the dimension of the other without changing its underlying structure [Williams et al., 2021]. Moreover, we standardize each feature in the encoding to have zero mean and unit variance (standard scaling) if not otherwise specified, whose statistics are computed only on the anchor sets for both source and target space, to perform the necessary de-normalization.

Estimating T In Moschella et al. [2023], it is empirically shown that the spaces mostly differ by an angle-preserving transformation. Nevertheless, we broaden our investigation by considering different ways of obtaining T to evaluate the robustness of that assumption and the versatility of our approach. Throughout our experiments, we primarily operate under the assumption that T can be constrained to encode, at most, an affine transformation: T ( x ) = RX + b

This general formulation, without additional constraints, corresponds to our affine method in the experiments, and it is optimized via gradient descent. The other transformations are trivially obtained by progressively adding constraints on this one:

linear . To model a linear transformation, we can just set the bias term to zero b = ⃗ 0 and optimize via Least Square. Here we are both simplifying the class of transformations and switching from a gradient descent optimization to a closed-form procedure.

l-ortho . Additionally, we could require R to be orthogonal to encode an isometry. In this case, we obtain this by applying Singular Value Decomposition (SVD) on the corresponding R obtained by the linear solution. Through this, we aim to understand the implications of enforcing orthogonality on a transformation that was originally not constrained to be so, in a setting similar to Xing et al. [2015].

ortho . To obtain the optimal orthogonal R , we apply Procrustes analysis [Gower, 1975].

This methodology facilitates efficient and precise zero-shot translation between disparate latent spaces. The transformation T , derived solely from the subset of corresponding points, provides a robust and versatile foundation for model reuse and interoperability in diverse machine learning contexts.

4 Latent Communication via Translation

In this section, we evaluate the capabilities and effectiveness of our translation method through various scenarios, highlighting its applicability in diverse contexts. We present empirical results in three different novel settings: i) cross-architecture; ii) cross-modality; iii) autoencoding. In each case, the translation performance of each method for obtaining the transformation T is evaluated against two baselines, the naive absolute one and the relative one.

Stitching Procedure In line with the zero-shot stitching concept outlined in Moschella et al. [2023], we combine independent encoders and decoders (e.g., classifiers, generators) without subsequent training or fine-tuning. This study does not necessitate a decoder trained on relative representations; instead, we directly employ the original decoders trained on absolute spaces. Each benchmark we perform follows the same procedure unless otherwise specified: we measure the mean performance over all the possible combinations of (encoder, decoder) for each test set in different settings:

Number of anchors

no-stitch . The end-to-end performance of the decoder applied to the original space it was trained on. This is useful to establish un upper-bound in performances;

absolute . The result of using the encodings without any transformation, we consider this as a probe for any pre-existing compatibility among encodings and, therefore, a lower-bound;

translation . These are the results of the application of our latent translation method, with the estimation of T via affine , linear , l-ortho and ortho .

In each instance, we use the same parallel anchors, that are uniformly chosen, in a quantity comparable with the dimensionality of the absolute representation.

4.1 Cross-Architecture

Firstly, we test our method in a cross-architecture setting, zero-shot stitching together encodings coming from a variety of pre-trained networks and their associated absolute decoders (classifiers). This scenario provides an extensive testing ground for our method and demonstrates its robustness across different architectures. Please refer to Table 8 in the Appendix for further results on crossarchitecture stitching in generation tasks.

Experimental setting We consider a variety of Computer Vision ( MNIST , Fashion MNIST , N24News , CIFAR10 , CIFAR100 ) and Natural Language Processing ( TREC [Hovy et al., 2001, Li and Roth, 2002], DBpedia [Auer et al., 2007] , N24News [Wang et al., 2022], AG News [Zhang et al., 2015], IMDB [Maas et al., 2011] ) datasets. For the text domain we consider 7 different language models as encoders (uncased and cased BERT [Devlin et al., 2019], Electra [Clark et al., 2020], RoBERTa base [Liu et al., 2019], ALBERT [Lan et al., 2020], and the text encoder of [Radford et al., 2021]), and for the image domain 6 encoders (RexNet100 [Han et al., 2020], 4 variations of ViT [Dosovitskiy et al., 2020], and the image encoder of [Radford et al., 2021]), all pre-trained and frozen.

The full encoder list can be found in Table 7 in the Appendix. For each dataset and for each encoder, we train an SVM classification head (decoder) on top of their specific encodings. We then proceed with the standard stitching procedure outlined in Section 4 and collect the results. Please see Table 8 in the Appendix for cross-architecture stitching in generation tasks, where we extend this analysis by verifying that our method works even across autoencoders of different bottleneck sizes.

Result analysis The stitching results are in Table 1. As expected, the absolute encodings obtain a score comparable to random guessing while also considering fewer encoder combinations out of the possible ones due to the dimensionality mismatch between some of them. Notably, these results show that the transformation relating to these pre-trained encoders is indeed mostly orthogonal: i) ortho and affine , the narrowest and the broadest transformation classes considered, are the better-performing translation methods. But while the former is obtained via a simple and efficient closed-form algorithm, the latter is SGD-optimized (Section 3.2). ii) the l-ortho version improves or has small drops in performances over the linear transformation it is obtained from, confirming that the least squares procedure converges to an R which is almost orthogonal. Note that these results demonstrate the feasibility of combining pre-trained models without the need for retraining or fine-tuning, with negligible drops in performances across the board and without any additional assumption on the decoders. Please refer to Tables 5 and 6 in the Appendix for results with different decoders. In the Appendix (Figure 7), we extend the cross-architecture transfer to decoders trained on different domains (styles) of the same CIFAR10 dataset: the original one and a grayscale one.

Sensibility to Anchor Quantity The number of anchors is an essential parameter in our approach. In Figure 3, we evaluate how the quantity of these anchors impacts the residual error and the overall performance of our method for this experimental setting. This analysis offers insights into the optimal number of anchors necessary for efficient latent space translation.

Role of Scaling Our approach is designed to accommodate generic (re)scaling methods as preprocessing steps. We advocate for the use of standard scaling, as it shows reliable performance in our experiments, indicating that the scale of the data points is useful in estimating the latent transformation T . However, for completeness, we also consider L2 normalization, which is the standard normalization in relative representations. This normalization method generalizes the class of transformations handled by our method and introduces an element of complete scale invariance. It's important to note that when this level of generalization is introduced, a scale-invariant decoder is required since the norm information is effectively removed. In the relative representation work, this is implicitly accomplished by training a decoder on relative representations. In our setting, since we do not train the decoder, in this setting we just assume its scale invariance (more details in Appendix A.1). This investigation exemplifies the flexibility of our approach, capable of adapting to different normalization and pre-processing strategies based on the specific requirements of the task at hand. The results presented in Table 2, when compared with Table 1, indicate a stronger reliance of the text modalities on the information encoded in the norm. This is aligned with existing literature in the NLP domain [Oyama et al., 2022], which suggests that the scale of the encodings contains information (e.g., it is correlated with the token frequency).

These results in diverse scenarios showcase the flexibility and adaptability of our method, especially its robustness in translating between latent spaces of different dimensionality and domains.

Number of samples (

log

)

Scale

Vision

Scale

Language

Decoder

Encoder

Score

Scale

Vision

ViT-base-224

0.40

90.45

RexNet

0.33

13.46

ViT-base-384

0.41

89.66

ViT-small

0.39

50.17

ViT-ResNet50

0.41

32.10

DarkNet

0.34

11.62

Text

BERT-cased

0.61

15.43

BERT-uncased

0.42

14.54

Electra

0.50

11.94

RoBERTa

0.75

11.06

ALBERT

0.27

32.27

XLM-R

0.73

18.75

24

0.40

0.33

0.41

0.38

0.40

0.33

0.40

0.29

0.29

0.58

0.17

0.56

Het

0.36

0.33

0.37

0.370.380.330.37

0.26

0.27

0.54

0.160.51

84

0.40

0.32

0.41

0.39

0.40

0.33

0.42

0.29

0.31

0.60

0.18

0.57

all

0.39

0.33

0.40

0.39

0.39

0.33

0.40

0.28

0.28

0.56

0.15

0.55

50

0.38

0.32

0.39

0.38

0.41

0.33

0.40

0.29

0.30

0.58

0.16

0.56

Het

0.36

0.33

0.37

0.37

0.38

0.34

0.37

0.26

0.26

0.56

0.17

0.53

ed

0.37

0.31

0.39

0.37

0.40

0.32

0.61

0.40

0.41

0.72

0.22

0.70

ed

F

0.34

0.29

0.35

0.33

0.37

0.28

0.48

0.42

0.36

0.65

0.24

0.64

tra

0.34

0.29

0.35

0.32

0.37

0.28

0.49

0.34

0.50

0.66

0.20

0.64

Ta

0.38

0.31

0.39

0.37

0.39

0.32

0.56

0.39

0.40

0.75

0.21

0.72

RT

0.30

0.25

0.31

0.28

0.32

0.23

0.40

0.33

0.32

0.56

0.27

0.55

-R

0.37

0.30

0.38

0.37

0.39

0.31

0.54

0.38

0.38

0.73

0.22

0.73

Encoder

4.2 Cross-Modality

This scenario illustrates the applicability of our method in cross-modality settings, where we aim to translate between latent spaces of different modalities: text and image.

Experimental setting We adopt N24News [Wang et al., 2022], a multimodal news classification dataset that contains both text and associated pictures. We apply the standard encoding procedure to these two features separately, using different pre-trained uni-modal encoders. Then, we train a classification head (an SVM, please refer to Appendix Figure 9 for further results employing an MLP as classification head) on top of each one. Lastly, we zero-shot stitch each encoder with a classification head different from its corresponding one, measuring its classification accuracy, without further training or fine-tuning.

S

Abs.

affine

linear

l-ortho

ortho

7

2

4

4

Vee

全

7

2

h

2

R

7

2

h

R

2

7

2

Scale distributions In Figure 4, we present the scale distribution of the embeddings produced by several encoders on the N24News dataset. This empirical analysis shows a consistent pattern among encoders in that the scale distribution of their embeddings follows a Gaussian one with a single mode and a well-defined mean, which are usually compatible with standard scaling. This consistent behavior across encoders is likely attributed to their architectural choices, such as the normalization techniques, regularizations and the optimization problems they are designed to solve.

Result analysis The discrepancy in the mean accuracy represented by the marginal bar plots in Figure 5 is a signal that can be used to identify spaces more suited to be decoded into and the ones that are stronger in encoding from . In fact, the language models as source space for the translation exhibit stronger performance than the vision encoders. We relate this behavior to the higher generality of the text domain data used during pre-training with respect to the image domain one [Zhai et al., 2022]. A remarkable finding in this setting is the improvement in classification performance when a modality-specific classifier trained on images is fed zero-shot with corresponding text encodings translated to the image domain via our method. This result underlines the significance of a good encoder and demonstrates the broad applicability of our technique. In practice, this means we can seamlessly apply image classifiers on textual data, and vice-versa.

These results show that our method: i) obtains effective zero-shot translation over different modalities; ii) improves unimodal decoders when translating from a better encoder than the one it was trained on.

4.3 Autoencoding

In this setting, our method is applied to align latent spaces of different trainings of the same autoencoder. The novelty of this scenario lies in the generation setting itself, as most prior works (Section 2) primarily focus on classification tasks. One key observation of [Cannistraci et al., 2023] is that the task at hand (e.g., classification, generation) defines a certain class of transformations (e.g. rotations) which act among the latent spaces. Restricting the search for the transformation to the right class, is fundamental in order to guarantee optimal performance and efficiency.

Experimental setting We utilize four datasets for these experiments: MNIST [Lecun et al., 1998], Fashion MNIST [Xiao et al., 2017], and CIFAR10 and CIFAR100 Krizhevsky [2009]. For each dataset, we train two standard CNN-based autoencoder, with convolutions in the encoder and deconvolutions in the decoder, please refer to the supplementary material for further implementation details. The two autoencoders are identical in structure, differing only in the random seed used for weight initialization and data shuffling. To perform zero-shot stitching, we first translate each data point from the latent space of the first encoder to the latent space of the second using 1000 anchors. We then apply the second decoder to the translated data, without any further training or fine-tuning.

Result analysis This experiment analyzes the alignment of latent spaces in different training regimens of the same autoencoder. The performance evaluation, as shown in Table 3, demonstrates that all methods affine , linear , l-ortho , and ortho yield satisfactory results. Moreover, qualitative results depicted in Figure 6 reveals minimal visual differences in the stitching outcomes across various datasets using different methods. Please refer to Figures 10 and 11 for other qualitative results. In fact, these results suggest that the latent spaces of image autoencoders are not exclusively correlated

by orthogonal transformations. Therefore, further research is warranted to explore and model the specific class of transformations that govern the correlation between neural networks during image autoencoding to constrain and enhance their approximation. For additional results pertaining to decoders with L2 normalization on their input, we refer to the Table 4 in the Appendix.

Overall these results, combined with Cannistraci et al. [2023] and Section 4.1, confirm that latent spaces in image autoencoders trained end-to-end are related by a class of transformations larger than orthogonal transformations.

5 Conclusion

At the heart of the proposed latent space translation lies the synergy between the principles of relative representation and classic algebraic techniques. The efficacy of this approach surpasses that of relative representations, emphasizing the benefits of directly estimating a transformation between specific latent space pairs instead of independently projecting them to a common one. This distinction underscores our contribution: we repurpose well-established techniques to serve as a translator across multiple latent spaces, enhancing efficiency in representation learning. With an extensive analysis of its applications in model reuse, we obtain a smooth compositionality of neural network modules across diverse computational frameworks, including those employing pre-trained models. Essentially, this paper showcases the adaptability and efficiency of manifold alignment methods in the emerging domain of zero-shot model compositionality.

Future works and limitations As with any new approach, there are limitations that warrant further exploration of our proposed method. For example, the optimal number of anchor points required for different tasks and datasets to boost performances, investigating the factors that could be linked to latent space compatibility (e.g., their intrinsic dimension), trade-offs between the granularity of the anchor set and its condition number. These are exciting research directions that we believe hold great potential for advancing the field and improving the effectiveness and robustness of our method.

Acknowledgements This work is supported by the ERC grant no.802554 (SPECGEO), PRIN 2020 project no.2020TA3K9N (LEGO.AI), and PNRR MUR project PE0000013-FAIR. Francesco Locatello did not contribute to this work at Amazon.

Reproducibility Statement We refer to the supplementary material for implementation details that are not described here in the main manuscript. Moreover, we release a modular PyTorch codebase 2 implementing the various translation methods and scaling techniques. All the experiments are carried out in deterministic environments to enable reproducibility, and the necessary data is versioned via DVC [Kuprieiev et al., 2022].

References

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives, 2014.

Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. ArXiv preprint , abs/1310.0425, 2013. URL https://arxiv.org/abs/1310.0425 .

Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In International Conference on Learning Representations , 2023. URL https://openreview.net/ forum?id=SrC-nwieGJ .

Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, and Francesco Locatello. ASIF: Coupled data turns unimodal models to multimodal without training. 2023. URL https://neurips.cc/virtual/2023/poster/71339 .

Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 5732-5741, 2018. URL https://proceedings.neurips.cc/paper/2018/hash/ a7a3d70c6d17a73140918996d03c014f-Abstract.html .

Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John E. Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 2016. URL http://arxiv.org/abs/ 1511.07543 .

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 3519-3529. PMLR, 2019. URL http://proceedings.mlr.press/v97/kornblith19a. html .

Lisa Bonheme and Marek Grzes. How do variational autoencoders learn? insights from representational similarity. ArXiv preprint , abs/2205.08399, 2022. URL https://arxiv.org/abs/2205. 08399 .

Anton Tsitsulin, Marina Munkhoeva, Davide Mottin, Panagiotis Karras, Alexander M. Bronstein, Ivan V. Oseledets, and Emmanuel Müller. The shape of data: Intrinsic distance for data distributions. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum? id=HyebplHYwB .

Serguei Barannikov, Ilya Trofimov, Nikita Balabin, and Evgeny Burnaev. Representation topology divergence: A method for comparing neural network representations. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 1607-1626. PMLR, 2022. URL https://proceedings.mlr.press/ v162/barannikov22a.html .

Ivan Vuli´ c, Sebastian Ruder, and Anders Søgaard. Are all good word vector spaces isomorphic? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 3178-3192, Online, 2020a. Association for Computational Linguistics. doi: 10. 18653/v1/2020.emnlp-main.257. URL https://aclanthology.org/2020.emnlp-main.257 .

Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/forum?id=H196sainb .

Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015 , pages 991-999. IEEE Computer Society, 2015. doi: 10.1109/CVPR.2015.7298701. URL https://doi.org/10.1109/CVPR.2015.7298701 .

Tomás Mikolov, Quoc V. Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. CoRR , abs/1309.4168, 2013a. URL http://arxiv.org/abs/1309.4168 .

Richard Antonello, Javier S Turek, Vy Vo, and Alexander Huth. Low-dimensional structure in the space of language representations is reflected in brain responses. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 8332-8344. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/file/ 464074179972cbbd75a39abc6954cd12-Paper.pdf .

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. 2012.

Yair Movshovitz-Attias, Alexander Toshev, Thomas K. Leung, Sergey Ioffe, and Saurabh Singh. No fuss distance metric learning using proxies. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017 , pages 360-368. IEEE Computer Society, 2017. doi: 10.1109/ICCV.2017.47. URL https://doi.org/10.1109/ICCV.2017.47 .

Tyler A Chang, Zhuowen Tu, and Benjamin K Bergen. The geometry of multilingual language model representations. 2022.

Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Yeh-Chiang, Yehuda Dar, Richard Baraniuk, Micah Goldblum, and Tom Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. 2022.

Raghav Mehta, Vítor Albiero, Li Chen, Ivan Evtimov, Tamar Glaser, Zhiheng Li, and Tal Hassner. You only need a good embeddings extractor to fix spurious correlations. 2022.

Chang Wang and Sridhar Mahadevan. Manifold alignment using procrustes analysis. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008 , volume 307 of ACM International Conference Proceeding Series , pages 1120-1127. ACM, 2008. doi: 10.1145/1390156.1390297. URL https://doi.org/10.1145/1390156.1390297 .

Chang Wang and Sridhar Mahadevan. Manifold alignment without correspondence. In IJCAI , volume 2, page 3, 2009.

Ivan Vuli´ c, Sebastian Ruder, and Anders Søgaard. Are all good word vector spaces isomorphic? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 3178-3192, Online, November 2020b. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.257. URL https://aclanthology.org/2020. emnlp-main.257 .

Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation, 2013b.

Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1006-1011, Denver, Colorado, May-June 2015. Association for Computational Linguistics. doi: 10.3115/v1/N15-1104. URL https://aclanthology.org/N15-1104 .

C. Gower. Generalized procrustes analysis. Psychometrika , 40(1):33-51, March 1975. ISSN 1860-0980. doi: 10.1007/BF02291478.

Adrian Csiszarik, Peter Korosi-Szabo, Akos K. Matszangosz, Gergely Papp, and Daniel Varga. Similarity and matching of neural network representations. ArXiv preprint , abs/2110.14633, 2021. URL https://arxiv.org/abs/2110.14633 .

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics , 5:135-146, 2017. doi: 10.1162/tacl_a_00051. URL https://aclanthology.org/Q17-1010 .

Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=r1Aab85gg .

Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 225-236, 2021. URL https://proceedings.neurips.cc/paper/ 2021/hash/01ded4259d101feb739b06c399e9cd9c-Abstract.html .

Michael Gygli, Jasper Uijlings, and Vittorio Ferrari. Towards reusable network components by learning compatible representations. AAAI , 35(9):7620-7629, 2021.

Niccolo Biondi an. Cores: Compatible representations via stationarity. ArXiv preprint , abs/2111.07632, 2021. URL https://arxiv.org/abs/2111.07632 .

Muammer Y. Yaman, Sergei V. Kalinin, Kathryn N. Guye, David Ginger, and Maxim Ziatdinov. Learning and predicting photonic responses of plasmonic nanoparticle assemblies via dual variational autoencoders. ArXiv preprint , abs/2208.03861, 2022. URL https://arxiv.org/abs/ 2208.03861 .

Federico Bianchi an. Fantastic embeddings and how to align them: Zero-shot inference i. ArXiv preprint , abs/2007.14906, 2020. URL https://arxiv.org/abs/2007.14906 .

Zorah Lähner and Michael Moeller. On the direct alignment of latent spaces. In UniReps: the First Workshop on Unifying Representations in Neural Models , 2023. URL https://openreview. net/forum?id=nro8tEfIfw .

Alex H. Williams, Erin Kunz, Simon Kornblith, and Scott W. Linderman. Generalized shape metrics on neural representations. In Advances in Neural Information Processing Systems , volume 34, 2021.

Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. Toward semantics-based answer pinpointing. In Proceedings of the First International Conference on Human Language Technology Research , 2001. URL https://www.aclweb.org/anthology/ H01-1069 .

Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics , 2002. URL https://www.aclweb.org/anthology/ C02-1150 .

Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. DBpedia: A nucleus for a web of open data. In The Semantic Web , pages 722-735. Springer Berlin Heidelberg, 2007.

Zhen Wang, Xu Shan, Xiangxie Zhang, and Jie Yang. N24news: A new dataset for multimodal news classification. In Proceedings of the Language Resources and Evaluation Conference , pages 6768-6775, Marseille, France, 2022. European Language Resources Association. URL https://aclanthology.org/2022.lrec-1.729 .

Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS , 2015.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: //www.aclweb.org/anthology/P11-1015 .

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https: //aclanthology.org/N19-1423 .

Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=r1xMH1BtvB .

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint , abs/1907.11692, 2019. URL https://arxiv.org/abs/1907.11692 .

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id= H1eA7AEtvS .

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. ArXiv preprint , abs/2103.00020, 2021. URL https://arxiv.org/abs/2103.00020 .

Dongyoon Han, Sangdoo Yun, Byeongho Heo, and YoungJoon Yoo. Rethinking channel dimensions for efficient model design. ArXiv preprint , abs/2007.00992, 2020. URL https://arxiv.org/ abs/2007.00992 .

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv preprint , abs/2010.11929, 2020. URL https://arxiv.org/abs/2010.11929 .

Momose Oyama, Sho Yokoi, and Hidetoshi Shimodaira. Norm of word embedding encodes information gain. ArXiv , abs/2212.09663, 2022. URL https://api.semanticscholar.org/ CorpusID:254853643 .

Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 18123-18133, 2022.

Irene Cannistraci, Luca Moschella, Marco Fumero, Valentino Maiorca, and Emanuele Rodolà. From bricks to bridges: Product of invariances to enhance latent space communication. 2023. URL https://arxiv.org/abs/2310.01211 .

Y Lecun, L Bottou, Y Bengio, and P Haffner. Gradient-based learning applied to document recognition. Proc. IEEE , 86(11):2278-2324, 1998.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learnin. ArXiv preprint , abs/1708.07747, 2017. URL https://arxiv.org/abs/1708. 07747 .

Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009. URL https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf .

Ruslan Kuprieiev, skshetry, Dmitry Petrov, Paweł Redzy´ nski, Peter Rowlands, Casper da CostaLuis, Alexander Schepanovski, Ivan Shcheklein, Batuhan Taskaya, Gao, Jorge Orpinel, David de la Iglesia Castro, Fábio Santos, Aman Sharma, Dave Berenbaum, Zhanibek, Dani Hodovic, daniele, Nikita Kodenko, Andrew Grigorev, Earl, Nabanita Dash, George Vyshnya, Ronan Lamy, maykulkarni, Max Hora, Vera, and Sanidhya Mangal. Dvc: Data version control - git for data & models, 2022. URL https://doi.org/10.5281/zenodo.7083378 .

Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

GrokAI. nn-template bootstraps pytorch projects by advocating reproducibility & best practices in deep learning, 2021. URL https://github.com/grok-ai/nn-template . Software available from https://github.com/grok-ai/.

William Falcon and The PyTorch Lightning team. PyTorch Lightning, 2019. URL https://github. com/Lightning-AI/lightning .

Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www.wandb. com/ . Software available from wandb.com.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020. emnlp-demos.6 .

Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 175-184, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. emnlp-demo.21. URL https://aclanthology.org/2021.emnlp-demo.21 .

A Additional results

In Figure 9, we present the outcomes of the multimodal experiment with an MLP employed as the classification head. The findings highlight the MLP's capability to leverage cross-modal information, leading to improved performance. However, the underlying mechanisms responsible for this enhancement remain unclear and warrant further investigation.

Method

Reconstruction MSE

Number of anchors

In Tables 5 and 6 quantitative results for stitching of MLP classifiers (differently from the main manuscript where SVMs are used) trained on top of pre-trained feature extractors, with and without additional L2 normalization, respectively.

In Figures 10 and 11, there are additional reconstruction examples with the same autoencoding setting as in the main manuscript, and with additional L2 normalization, respectively.

In Table 4 more quantitative results for stitching of autoencoders, with added L2 normalization (at training time) to the decoders of the reconstruction setting of the main manuscript.

Decoder

Encoder

Score

Scale

Vision

ViT-base-224

0.46

90.45

RexNet

0.38

13.46

ViT-base-384

0.47

89.66

ViT-small

0.40

50.17

ViT-ResNet50

0.44

32.10

DarkNet

0.37

11.62

Text

BERT-cased

0.40

15.43

BERT-uncased

0.39

14.54

Electra

0.27

11.94

RoBERTa

0.51

11.06

ALBERT

0.30

32.27

XLM-R

0.41

18.75

24

0.46

0.350.460.420.430.360.45

0.33

0.34

0.64

0.19

0.61

Het

0.410.380.420.400.420.360.400.300.290.59

0.16

0.57

84

0.45

0.35

0.470.42 0.420.360.46

0.34

0.34

0.63

0.19

0.61

all

0.40

0.330.410.400.400.340.37

0.31

0.29

0.54

0.18

0.51

50

0.43

0.350.440.410.440.360.430.33

0.32

0.63

0.18

0.60

Jet

0.40

0.36

0.41

0.39

0.41

0.37

0.38

0.31

0.28

0.56

0.16

0.54

ed

一

0.31

0.27

0.32

0.28

0.33

0.26

0.40

0.30

0.31

0.53

0.17

0.51

0.35

0.29

0.36

0.34

0.35

0.29

0.43

0.39

0.31

0.58

0.22

0.56

tra

0.21

0.17

0.22

0.19

0.23

0.18

0.30

0.22

0.27

0.43

0.15

0.41

Ta

0.30

0.26

0.31

0.28

0.31

0.25

0.36

0.29

0.26

0.51

0.17

0.48

RT

0.32

0.26

0.33

0.30

0.33

0.26

0.40

0.33

0.30

0.55

0.30

0.53

-R

0.23

0.22

0.24

0.21

0.26

0.21

0.31

0.21

0.21

0.43

0.14

0.41

Encoder

S

Abs.

affine

linear

l-ortho

ortho

7

2

口

4

Vee

7

2

4

h

7

2

h

4

7

Z

2

S

Abs.

affine

linear

l-ortho

ortho

S

Abs.

affine

linear

l-ortho

ortho

0

工

8

5

7

8

5

6

7

8

5

7

8

5

6

7

8

5

7

8

5

6

7

I

8888

5

6

5

7

7

I

A.1 Scale invariance

In this section, we delve into the concept of scale invariance in neural networks and its implications for model compositionality. We start by focusing on the effect of rescaling operations on the latent input encodings and demonstrate that, by construction, certain classifiers exhibit scale-invariance properties without the need for additional priors. Then, by examining the behavior of networks when subjected to a specific type of input manipulation, rescaling injection , we aim to demonstrate the robustness and versatility of neural networks in handling different scales of input data. As illustrated in the main manuscript, this is a key advantage in improving the adaptability of our method.

The softmax function, commonly used in neural classifiers, is known to be a temperature-controlled variant of the maximum function:

This means that the softmax temperature can be used to control the level of confidence of the classifier's predictions. In this study, we show that a similar effect can also be achieved by rescaling the latent encodings given as input to a trained (and frozen) classifier.

In order to demonstrate this, we first note that the rescaling factor, α , can be factored out of the matrix multiplication in the Linear layers of the classifier. This can be represented mathematically as: y = α Wx + b , where x is the input latent encoding, W is the weight matrix, b is the bias vector, α is the rescaling factor, and y is the output of the linear layer. This implies that the rescaling operation can be 'pushed through' the classifier without affecting its final prediction as it becomes equivalent to some temperature value applied at the softmax level.

Furthermore, we investigate the effect of rescaling when non-linear activation functions are involved and posit that as long as the function has a monotonic interval, if we rescale all the dimensions by an amount similar to the mean scale of the encodings on which the classifier was trained, we end up in the monotonic interval, without losing the scale-invariance property.

In summary, our study provides empirical evidence that neural classifiers that utilize the softmax activation function can, in practice, maintain their scale-invariance properties when the input latent encodings are rescaled. This property is essential to our method, as it allows us to ignore the exact scale when decoding toward an L2-normalized absolute space.

Pre-trained models and scale-invariance We observed that large pre-trained models, such as transformers and resnets, are robust to internal rescaling of the encodings. Although we do not have a strong theoretical explanation for this phenomenon, we hypothesize that normalization layers and the linear separability of the information encoded in the angles instead of the norms may play a significant role. In Figure 12, we demonstrate the invariance a large transformer exhibits when the rescaling injection is applied at different layers: surprisingly, when the rescaling surpasses a certain threshold, the performance difference becomes negligible. These results further emphasize the robustness of these pre-trained models to the rescaling injection and suggest that the scale of the embedding is not a critical factor in their performance.

Rescale Injection We define the rescaling injection as the operation of artificially altering the scale of the features produced at a specific layer of the network. This is achieved by normalizing the embeddings to unit norm and then rescaling them by a factor of α . By varying the value of α , we can observe how the network's performance is affected at different scales. Through this empirical analysis, we aim to provide insight into the scale invariance properties of neural networks and their potential for use in model compositionality.

In Figure 13, we present experimental results investigating the scale invariance properties of neural networks. We trained simple multi-layer perceptrons (MLPs) composed of two hidden layers, with no normalization layers, using encodings produced by the Clip Vision transformer ( clip-vit-base-patch32 ) on the CIFAR100 (fine) dataset. The MLPs were evaluated using different activation functions: cosine (blue), tanh (orange), and ReLU (green). The rescaling injection technique was applied directly to the input embeddings, rescaling them by α .

We can observe that the scale of the embeddings does not significantly impact the MLPs' performance when using monotone activation functions that do not flip signs. This is a non-trivial result, as the nonlinearity of the activation function, the presence of bias terms b , and the absence of normalization layers make it difficult to predict the effect of an input rescaling on the performance of the network. It is particularly interesting to see that the cosine activation function shows an oscillatory performance, comparable to the original embeddings when rescaled by the mean embeddings scale (vertical red line) or its opposite since it is symmetric.

Our findings indicate that, surprisingly, even the internal layers of large deep learning models exhibit a positive scale invariance , as illustrated in Figure 12. The underlying mechanism for this behavior is not straightforward, but we hypothesize that it may result from the interplay between various factors,

0

1

2

3

4

5

6

7

8

9

10

11

12

-2.00

-1.83

-1.67

-1.50

-1.33

-1.17

-1.00

-0.83

-0.67

-0.50

1.00

1.47

1.95

2.42

2.90

3.37

3.84

4.32

4.79

5.26

5.74

6.21

6.68

7.16

7.63

8.11

8.58

9.05

9.53

10.00

11.44

1e+02

1e+03

1e+04

1e+05

1e+06

1e+07

1e+08

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Rescaled layer

Rescaling factor

α

Rescale Factor

such as the choice of activation function, the use of normalization layers, the optimization objective and regularization techniques employed during the training phase. Further research is needed to understand and explain this phenomenon fully.

B Implementation Details

All the experiments were conducted using a machine equipped with an Intel Core i7-9700k CPU, 64 GB of RAM, and an NVIDIA 2080TI GPU.

Decoder structure The full implementation details can be found in the attached code. The various experiments can be run by their corresponding notebook, while the source code for the package they are built on can be found under the "src" folder.

Autoencoding . Since the autoencoders were used only on image data, the architecture was a simple sequence of convolutions (in the encoder part) and deconvolutions (in the decoder part). Each interleaved with nonlinear activations.

Classification . The main manuscript refers to "SVM" as the standard SVM implementation in scikit-learn [Pedregosa et al., 2011], with default parameters. The experiments with "MLP" as a classifier refer to a simple stack of 3 linear layers, interleaved by nonlinear activations.

Software and Technologies The research of this study was facilitated by the use of various technologies and tools, which include:

NN-Template [GrokAI, 2021], was used to kick-start the project while also ensuring best practices were adhered to;

DVC [Kuprieiev et al., 2022], was implemented for data versioning;

PyTorch Lightning [Falcon and The PyTorch Lightning team, 2019], contributed to maintaining the integrity of the results and promoting clean, modular code;

Weights and Biases [Biewald, 2020], were employed for logging experiments, running comparisons over extensive sweeps, and sharing models;

Transformers by HuggingFace [Wolf et al., 2020], provided pre-configured transformers for processing both image and text data;

Datasets by HuggingFace [Lhoest et al., 2021], facilitated access to a majority of NLP datasets and ImageNet for computer vision purposes;

Pre-trained encoders All the pre-trained encoders used come from HuggingFace and are listed in Table 7. They are various both in terms of architecture and encoding size.

Table 7: HuggingFace models used as encoders (feature extractors) in the various experiments, with their encoding dimensionality.

Peer reviews:
comments: SUMMARY:
This work proposed a method to translate learned representations between two pre-trained networks, using surprisingly simple transformations. The method is demonstrated on models with different architectures, trained on different modalities and across different tasks.

STRENGTHS:
1. The paper is well written and easy to follow. 
2. To my understanding, the discussed setting where models are ad-hoc zero-shot stitched, as opposed to being trained to have compatible representations is original. This distinction is important as it makes the approach applicable for the myriad of pre-trained models that already exist.
3. The method is surprisingly simple and yet produces very convincing results as demonstrated through sufficient evaluation.

WEAKNESSES:
1. The exact setting leading to Figure 2 is not clear. Caption mentions results being produced on “CIFAR-100”. However, I was not able to find the entire setting. What two models were used? Are both trained on CIFAR-100? What are the architectures used? Are reported results averaged across multiple settings? 
2. Also regarding Figure 2 - the two trends not being monotone is surprising to me. Especially the very clear outliers. I suppose this behavior might occur if each point represents a single experiment and some randomness might affect performance. In that case, it would probably be better to average these results across several experiments as done in other tables in the paper. Can the authors explain this behavior?


Minor comments:

1. Typo line 49 - mphasizing -> emphasizing.
2. Notation line 139 - $\mu$ is used in equation 3 but $\bar{x}$ in text.
3. Clarity line 167 - I suggest the “naive absolute baseline” should be concisely explained.

QUESTIONS:
See questions mentioned in the Weaknesses section.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors propose to translate latent space using an effectively angle-preserving affine transformation learned using pairs as anchors. Experiments are conducted on a wide range of tasks showing the okay performance by the swap-in latent space of embeddings/features.\n\nSTRENGTHS:\n**Strengths**\n\nThe authors have taken an extensive suite of experiments encompassing a diverse set of tasks, including cross-training, cross-architectures, and several downstream tasks. The experimental design demonstrates a comprehensive approach in assessing the proposed method *per se*.\n\nFurthremore, this work shows that aligning two latent spaces with zero-shot stitching is poissible without any retraining. This is an important observation and the proposed method could benefit a wide range of downstream tasks.\n\nWEAKNESSES:\n**Weaknesses**\n\nThe following concerns regarding the design of the transformation and its intrinsic limitation warrant consideration.\n\n- **Assumption of Linearity and Lack of Analysis in Generative Model**:\n\n\nThe proposed method is effectively an angle-preserving affine transformation, which fundamentally assumes linearity of mapping between two latent spaces (of embeddings and/or features). While \"linearity of the transformation between seemingly different latent spaces has a theoretical foundation in research on identifiability in neural models,\" as the author pointed out in the rebuttal, in the context of generative model, the efficacy is still likely contingent on the extent to which two latent spaces exhibit linear mapping, because a good generative model may fill the slight mismatch with its own capacity. \n\nIn detail, it is likely that an affine transformation may yield only a coarse mapping between two latent spaces, and may not capture fine-grained relations (consider style transfer in generative models).\nThus, the good performance in classification tasks may thus be attributed to the capacities of the downstream models rather than the transformation itself. This is a critical assumption that necessitates further scrutiny, but the submission does not provide an in-depth analysis of this assumption.\n\nIn the current form of the manuscript, for this regard only an autoencoding on MNIST/Fashion MINIST/CIFAR is shown. It does not have a qualitative comparison with the reconstruction from VAE itself, nor quantitative/qualitative results with other stitching and zero-shot methods.\nThe lack of such comparison can unfortunately weaken the justification of the proposed method.\nAdding such comparison is thus suggested. Optionally, conducting such comparisons on common image datasets of higher resolution, like CelebA, ImageNet, (with more powerful VAE models) may provide more insight in this regard, since it may potentially highlight the capacity of VAE models and the quality of latent space alignment.\n\n(**After author response**: See the discussion below) \n\n- **Ambiguity in Technical Contribution**:\n\nThe technical novelty and significance of the proposed method remain unclear. The method effectively is an angle-preserving affine transformation. While this per se is not necessarily an issue, the missing studies mentioned above make it hard to assess the impact and implications of the proposed method.\n\n(**After author response**: This is not a major issue anymore, see the discussion below) \n\n- ** Position of Works within the Broad Field**:\n\nLatent space matching has been the subject of extensive research, albeit in varying contexts such as text, image processing, and generative models.\nA few (it's even not an attempt to cover the overall picture of this widely studied topic) works below in References that share technical similarity with the proposed method  are cited in the References.\nHowever, the submission does not include a substantive comparison or discussion with a wide body of works in this field, mkaing it challenging to ascertain the relative standing and uniqueness of the proposed method.\n\n(**After author response**: This is not a major issue anymore, see the discussion below) \n\n**References**:\n\n-\nSet Prediction in the Latent Space https://proceedings.neurips.cc/paper/2021/hash/d61e9e58ae1058322bc169943b39f1d8-Abstract.html\n- Formality Style Transfer with Shared Latent Space https://aclanthology.org/2020.coling-main.203.pdf\n- Homomorphic Latent Space Interpolation for Unpaired Image-To-Image Translation https://openaccess.thecvf.com/content_CVPR_2019/html/Chen_Homomorphic_Latent_Space_Interpolation_for_Unpaired_Image-To-Image_Translation_CVPR_2019_paper.html\n\n\n\n**Updates**\n\n- The previous version of this review mistakenly refer to the page limit. As the authors and the AC pointed out, this is NOT the case and has been corrected.\n\n- In Weakness, update the Assumption of Linearity to focus on the generative model.\n\n- Added reference to discussion below.\n\n- The rating has been increased.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article proposes a method of latent space translation through semantic alignment, using the similarity of latent spaces learned by different neural models on semantically similar data. The method allows for direct conversion of learned representations between different pre-trained networks, and achieve zero-shot stitching of encoders and decoders without additional training. The method is widely validated in various experimental settings, tasks (classification and generation), and modalities, demonstrating the ability to zero-shot stitch neural models under different architecture and modality changes.\n\nSTRENGTHS:\n1. The paper proposes a method for the zero-shot stitching between encoders and decoders on various tasks, enabling seamless integration without the need for additional training.\n2. Compared to previous methods, this article expands to different latent space dimensions, architectures, and modalities. \n3. Extensive set of experiments have been done to show the possibility to zero-shot together neural models across different architectural and modalities.\n\nWEAKNESSES:\n1. In the experiment, only the most basic abs method is compared, but no comparison is made with other methods.\n2. In the generation task, zero-shot stitching is performed under the same model framework and training data. Whether it is more practical to migrate between different model frameworks and training data?\n\nQUESTIONS:\n1. How much performance difference is there between the proposed zero-shot stitch method and concatenation followed by finetuning? \n2. How is the selection of anchor points determined, and how much impact does different selection have on performance? \n3. In experiments, zero-shot stitch is performed between different frameworks on the same dataset. So, can this method be effectively used for neural frameworks trained on different domain datasets?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper show the latent space of different pretrained models can be translated between each other with simple transformations by using anchors. By using this method, a variety of encoders architectures are able to be cross-stiched to different classifier heads or within autoencoding models.

STRENGTHS:
This paper tackles an interesting problem for how to semantically align different trained latent spaces.

The paper is a natural extension to Moschella et. al 2023, going from just relative encoding to full latent space transformations.

The results are clear in showing zero-shot stitching is possible without retraining the decoders

WEAKNESSES:
It wasn't clear on my first read of the paper that this work is a direct follow up to Moschella et al. 2023. Without that context, the paper was very difficult to read the first time. After reading Moschella et al., I was better able to follow this proposed work. Multiple details are missing, such as the construction of the anchor set, which are glossed over in this work and caused confusion on the zero-shot nature of this work. Another major missing key detail is that prior work's decoders must work on the relative embedding, this crucial detail was not clear until after multiple read throughs. This improvement should be front and center.

I find it surprising that tables 1 and 2 do not report Moschella's results. I realize that the decoders must be trained with the relative embeddings, but the lack of comparison feel likes an obfuscation rather than a highlight of the difference between methods.

Overall, I find the clarity of the paper to be a major hinderance.

QUESTIONS:
None

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06