SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Role of Scaling Our approach is designed to accommodate generic (re)scaling methods as preprocessing steps. We advocate for the use of standard scaling, as it shows reliable performance in our experiments, indicating that the scale of the data points is useful in estimating the latent transformation  $\mathcal{T}$ . However, for completeness, we also consider L2 normalization, which is the standard normalization in relative representations. This normalization method generalizes the class of transformations handled by our method and introduces an element of complete scale invariance. It's important to note that when this level of generalization is introduced, a scale-invariant decoder is required since the norm information is effectively removed. In the relative representation work, this is implicitly accomplished by training a decoder on relative representations. In our setting, since we do not train the decoder, in this setting we just assume its scale invariance (more details in Appendix A.1). This investigation exemplifies the flexibility of our approach, capable of adapting to different normalization and pre-processing strategies based on the specific requirements of the task at hand. The results presented in Table 2, when compared with Table 1, indicate a stronger reliance of the text modalities on the information encoded in the norm. This is aligned with existing literature in the NLP domain [Oyama et al., 2022], which suggests that the scale of the encodings contains information (e.g., it is correlated with the token frequency).
These results in diverse scenarios showcase the flexibility and adaptability of our method, especially its robustness in translating between latent spaces of different dimensionality and domains.
Figure 4: Scale distribution in encodings of different pre-trained encoders on the N24News dataset.
Figure 5: Performance comparison between different encoders and data modalities on the N24News multimodal dataset. On the right the accuracy of models trained end-to-end on a single data modality (Score) and their average norm (Scale). On the left the stitching performance between pairs of encoders and decoder. This shows the importance of translating from good encoders, that can even improve unimodal decoder performances. Results obtained with 2000 anchors and ortho, with an SVM as classification head. In the Appendix Figure 9, additional results using MLPs as decoders.
Encoder	Score	Scale
Vision	ViT-base-224	0.40	90.45
RexNet	0.33	13.46
ViT-base-384	0.41	89.66
ViT-small	0.39	50.17
ViT-ResNet50	0.41	32.10
DarkNet	0.34	11.62
Text	BERT-cased	0.61	15.43
BERT-uncased	0.42	14.54
Electra	0.50	11.94
RoBERTa	0.75	11.06
ALBERT	0.27	32.27
XLM-R	0.73	18.75
4.2 Cross-Modality
This scenario illustrates the applicability of our method in cross-modality settings, where we aim to translate between latent spaces of different modalities: text and image.
Experimental setting We adopt N24News [Wang et al., 2022], a multimodal news classification dataset that contains both text and associated pictures. We apply the standard encoding procedure to these two features separately, using different pre-trained uni-modal encoders. Then, we train a classification head (an SVM, please refer to Appendix Figure 9 for further results employing an MLP as classification head) on top of each one. Lastly, we zero-shot stitch each encoder with a classification head different from its corresponding one, measuring its classification accuracy, without further training or fine-tuning.
Figure 6: Reconstruction examples grouped by dataset. Each column is a different image, from top to bottom: original image, absolute stitching, affine stitching linear stitching, l-ortho stitching, and ortho stitching. No additional normalization applied on the decoder part. Please refer to Figures 10 and 11 in the Appendix for decoders trained with L2 normalization.
Scale distributions In Figure 4, we present the scale distribution of the embeddings produced by several encoders on the N24News dataset. This empirical analysis shows a consistent pattern among encoders in that the scale distribution of their embeddings follows a Gaussian one with a single mode and a well-defined mean, which are usually compatible with standard scaling. This consistent behavior across encoders is likely attributed to their architectural choices, such as the normalization techniques, regularizations and the optimization problems they are designed to solve.
Result analysis The discrepancy in the mean accuracy represented by the marginal bar plots in Figure 5 is a signal that can be used to identify spaces more suited to be decoded into and the ones that are stronger in encoding from. In fact, the language models as source space for the translation exhibit stronger performance than the vision encoders. We relate this behavior to the higher generality of the text domain data used during pre-training with respect to the image domain one [Zhai et al., 2022]. A remarkable finding in this setting is the improvement in classification performance when a modality-specific classifier trained on images is fed zero-shot with corresponding text encodings translated to the image domain via our method. This result underlines the significance of a good encoder and demonstrates the broad applicability of our technique. In practice, this means we can seamlessly apply image classifiers on textual data, and vice-versa.
These results show that our method: i) obtains effective zero-shot translation over different modalities; ii) improves unimodal decoders when translating from a better encoder than the one it was trained on.
4.3 Autoencoding
In this setting, our method is applied to align latent spaces of different trainings of the same autoencoder. The novelty of this scenario lies in the generation setting itself, as most prior works (Section 2) primarily focus on classification tasks. One key observation of [Cannistraci et al., 2023] is that the task at hand (e.g., classification, generation) defines a certain class of transformations (e.g. rotations) which act among the latent spaces. Restricting the search for the transformation to the right class, is fundamental in order to guarantee optimal performance and efficiency.
Experimental setting We utilize four datasets for these experiments: MNIST [Lecun et al., 1998], Fashion MNIST [Xiao et al., 2017], and CIFAR10 and CIFAR100 Krizhevsky [2009]. For each dataset, we train two standard CNN-based autoencoders, with convolutions in the encoder and deconvolutions in the decoder, please refer to the supplementary material for further implementation details. The two autoencoders are identical in structure, differing only in the random seed used for weight initialization and data shuffling. To perform zero-shot stitching, we first translate each data point from the latent space of the first encoder to the latent space of the second using 1000 anchors. We then apply the second decoder to the translated data, without any further training or fine-tuning.
Result analysis This experiment analyzes the alignment of latent spaces in different training regimens of the same autoencoder. The performance evaluation, as shown in Table 3, demonstrates that all methods affine, linear, 1-ortho, and ortho yield satisfactory results. Moreover, qualitative results depicted in Figure 6 reveals minimal visual differences in the stitching outcomes across various datasets using different methods. Please refer to Figures 10 and 11 for other qualitative results. In fact, these results suggest that the latent spaces of image autoencoders are not exclusively correlated
Table 3: Zero-shot stitching for generation with various methods for estimating  $\mathcal{T}$ . Standard scaling used as normalization and the stitched decoders do not have any additional normalization. We report the latent cosine similarity (lcos) and MSE (lmse) between the target encoding and the translated one, but also the reconstruction MSE (rmse) between the input and the output. 1000 anchors used on 500 dimensional spaces. Please refer to Table 4 for results on decoders scale-invariant by design (with L2 normalization on the encodings).
MNIST	Fashion MNIST	CIFAR10	CIFAR100
lcos	lmse	rmse	lcos	lmse	rmse	lcos	lmse	rmse	lcos	lmse	rmse
absolute	0.09	0.27	0.14	0.17	0.23	0.23	0.30	0.29	0.34	0.34	0.53	0.40
affine	0.94	0.08	0.02	0.94	0.06	0.03	0.96	0.03	0.05	0.96	0.04	0.05
linear	0.92	0.09	0.02	0.93	0.07	0.04	0.94	0.03	0.05	0.94	0.04	0.06
l-ortho	0.79	0.14	0.02	0.78	0.12	0.05	0.85	0.05	0.06	0.84	0.07	0.07
ortho	0.90	0.10	0.02	0.90	0.08	0.04	0.94	0.03	0.06	0.93	0.04	0.06
by orthogonal transformations. Therefore, further research is warranted to explore and model the specific class of transformations that govern the correlation between neural networks during image autoencoding to constrain and enhance their approximation. For additional results pertaining to decoders with L2 normalization on their input, we refer to the Table 4 in the Appendix.
Overall these results, combined with Cannistraci et al. [2023] and Section 4.1, confirm that latent spaces in image autoencoders trained end-to-end are related by a class of transformations larger than orthogonal transformations.
At the heart of the proposed latent space translation lies the synergy between the principles of relative representation and classic algebraic techniques. The efficacy of this approach surpasses that of relative representations, emphasizing the benefits of directly estimating a transformation between specific latent space pairs instead of independently projecting them to a common one. This distinction underscores our contribution: we repurpose well-established techniques to serve as a translator across multiple latent spaces, enhancing efficiency in representation learning. With an extensive analysis of its applications in model reuse, we obtain a smooth compositionality of neural network modules across diverse computational frameworks, including those employing pre-trained models. Essentially, this paper showcases the adaptability and efficiency of manifold alignment methods in the emerging domain of zero-shot model compositionality.
Future works and limitations As with any new approach, there are limitations that warrant further exploration of our proposed method. For example, the optimal number of anchor points required for different tasks and datasets to boost performances, investigating the factors that could be linked to latent space compatibility (e.g., their intrinsic dimension), trade-offs between the granularity of the anchor set and its condition number. These are exciting research directions that we believe hold great potential for advancing the field and improving the effectiveness and robustness of our method.
Acknowledgements This work is supported by the ERC grant no.802554 (SPECGEO), PRIN 2020 project no.2020TA3K9N (LEGO.AI), and PNRR MUR project PE0000013-FAIR. Francesco Locatello did not contribute to this work at Amazon.
Reproducibility Statement We refer to the supplementary material for implementation details that are not described here in the main manuscript. Moreover, we release a modular PyTorch codebase<sup>2</sup> implementing the various translation methods and scaling techniques. All the experiments are carried out in deterministic environments to enable reproducibility, and the necessary data is versioned via DVC [Kuprieiev et al., 2022].
$^{2}$ https://github.com/Flegyas/latent-translation
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
GrokAI. nn-template bootstraps pytorch projects by advocating reproducibility & best practices in deep learning, 2021. URL https://github.com/grok-ai/nn-template. Software available from https://github.com/grok-ai/.
William Falcon and The PyTorch Lightning team. PyTorch Lightning, 2019. URL https://github.com/Lighting-AI/lightning.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www.wandb.com/. Software available from wandb.com.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumont, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-demo.21. URL https://aclanthology.org/2021.emnlp-demo.21.
In Figure 9, we present the outcomes of the multimodal experiment with an MLP employed as the classification head. The findings highlight the MLP's capability to leverage cross-modal information, leading to improved performance. However, the underlying mechanisms responsible for this enhancement remain unclear and warrant further investigation.
Figure 7: Cross-domain stitching on CIFAR10 and grayscale CIFAR10. 84 stitched pairs (pre-trained encoder - SVM classifier) for 5 different seeds.
Figure 8: Performance comparison (reconstruction error) of affine, linear, l-ortho and ortho at varying anchor number on reconstruction task. Results on stitching 2 different CIFAR100-trained AEs with 5 samplings for each anchor quantity. The naive absolute baseline is flat on 0.38 as mean.
In Tables 5 and 6 quantitative results for stitching of MLP classifiers (differently from the main manuscript where SVMs are used) trained on top of pre-trained feature extractors, with and without additional L2 normalization, respectively.
In Figures 10 and 11, there are additional reconstruction examples with the same autoencoding setting as in the main manuscript, and with additional L2 normalization, respectively.
In Table 4 more quantitative results for stitching of autoencoders, with added L2 normalization (at training time) to the decoders of the reconstruction setting of the main manuscript.
Figure 9: Performance comparison between different encoders and data modalities on the N24News multimodal dataset. On the right, the accuracy of models trained end-to-end on a single data modality (Score) and their average norm (Scale). On the left the stitching performance between pairs of encoders and decoder. This shows the importance of translating from good encoders, that can even improve unimodal decoder performances. Results obtained with 2000 anchors and SVD, with a MLP as classification head.
Encoder	Score	Scale
Vision	ViT-base-224	0.46	90.45
RexNet	0.38	13.46
ViT-base-384	0.47	89.66
ViT-small	0.40	50.17
ViT-ResNet50	0.44	32.10
DarkNet	0.37	11.62
Text	BERT-cased	0.40	15.43
BERT-uncased	0.39	14.54
Electra	0.27	11.94
RoBERTa	0.51	11.06
ALBERT	0.30	32.27
XLM-R	0.41	18.75
Figure 10: Reconstruction examples grouped by dataset. Each column is a different image, from top to bottom: original image, absolute stitching, LSS stitching, OLSS stitching, and SVD stitching. An L2 normalization is applied to the decoder input.
Table 4: Zero-shot stitching for generation. With SVD for estimating  $\mathcal{T}$  and standard scaling as pre-processing. An L2 normalization is applied to the decoder input. We report the latent cosine similarity (lcos) and MSE (lmse) between the target encoding and the translated one, but also the reconstruction MSE (rmse) between the input and the output.
MNIST	Fashion MNIST	CIFAR10	CIFAR100
lcos	lmse	rmse	lcos	lmse	rmse	lcos	lmse	rmse	lcos	lmse	rmse
Abs.	0.39	0.98	0.28	0.53	0.97	0.33	0.62	1.23	0.46	0.59	1.17	0.38
affine	0.99	0.15	0.01	0.99	0.16	0.03	0.99	0.16	0.04	0.99	0.12	0.05
linear	0.98	0.17	0.01	0.98	0.18	0.03	0.99	0.16	0.04	0.99	0.13	0.05
l-ortho	0.89	0.41	0.02	0.91	0.41	0.04	0.96	0.39	0.05	0.93	0.30	0.08
ortho	0.97	0.21	0.02	0.97	0.23	0.03	0.99	0.21	0.05	0.96	0.22	0.07
Figure 11: Additional reconstruction examples grouped by dataset. Each column is a different image, from top to bottom: original image, absolute stitching, LSS stitching, OLSS stitching, and SVD stitching. In the first row, no additional normalization is applied on the decoder input; in the second row, an L2 normalization is applied instead.
Table 5: Cross-architecture stitching with various methods for estimating  $\mathcal{T}$  and employing standard scaling. The stitched decoders are simple MLPs. 5 runs for each encoder-decoder pair. (C) and (F) next to CIFAR100 indicate, respectively, coarse-grained and fine-grained.
Dataset	No Stitching	absolute	relative	affine	linear	l-ortho	ortho
Vision	CIFAR10	0.95 ± 0.03	0.16 ± 0.22	0.73 ± 0.21	0.93 ± 0.05	0.89 ± 0.11	0.90 ± 0.09	0.93 ± 0.04
CIFAR100-C	0.82 ± 0.07	0.11 ± 0.21	0.39 ± 0.17	0.76 ± 0.08	0.71 ± 0.15	0.74 ± 0.11	0.78 ± 0.07
CIFAR100-F	0.68 ± 0.14	0.06 ± 0.20	0.13 ± 0.09	0.59 ± 0.13	0.55 ± 0.18	0.56 ± 0.17	0.62 ± 0.12
F-MNIST	0.87 ± 0.02	0.14 ± 0.20	0.64 ± 0.12	0.85 ± 0.02	0.83 ± 0.05	0.80 ± 0.06	0.84 ± 0.02
MNIST	0.92 ± 0.03	0.15 ± 0.20	0.36 ± 0.14	0.92 ± 0.03	0.87 ± 0.08	0.74 ± 0.12	0.88 ± 0.03
Text	TREC	0.41 ± 0.07	0.15 ± 0.04	0.27 ± 0.09	0.40 ± 0.08	0.37 ± 0.11	0.23 ± 0.08	0.41 ± 0.09
AG News	0.76 ± 0.08	0.24 ± 0.02	0.36 ± 0.10	0.68 ± 0.08	0.65 ± 0.08	0.64 ± 0.10	0.68 ± 0.10
DBpedia	0.64 ± 0.19	0.07 ± 0.02	0.15 ± 0.08	0.57 ± 0.19	0.53 ± 0.19	0.44 ± 0.21	0.56 ± 0.17
IMDB	0.62 ± 0.04	0.50 ± 0.01	0.50 ± 0.01	0.59 ± 0.04	0.58 ± 0.04	0.57 ± 0.04	0.60 ± 0.04
Table 6: Cross-architecture stitching with various methods for estimating  $\mathcal{T}$  and applying L2 as normalization. The stitched decoders are simple MLPs. 5 runs for each encoder-decoder pair. (C) and (F) next to CIFAR100 indicate, respectively, coarse-grained and fine-grained.
Dataset	No Stitching	absolute	relative	affine	linear	1-ortho	ortho
Vision	CIFAR10	0.95 ± 0.03	0.16 ± 0.22	0.73 ± 0.21	0.93 ± 0.04	0.89 ± 0.11	0.89 ± 0.11	0.93 ± 0.04
CIFAR100-C	0.82 ± 0.07	0.11 ± 0.21	0.39 ± 0.17	0.77 ± 0.07	0.75 ± 0.13	0.71 ± 0.15	0.78 ± 0.06
CIFAR100-F	0.68 ± 0.14	0.06 ± 0.20	0.13 ± 0.09	0.60 ± 0.12	0.57 ± 0.18	0.54 ± 0.18	0.61 ± 0.12
F-MNIST	0.87 ± 0.02	0.14 ± 0.20	0.64 ± 0.12	0.86 ± 0.02	0.79 ± 0.09	0.83 ± 0.05	0.84 ± 0.02
MNIST	0.92 ± 0.03	0.15 ± 0.20	0.36 ± 0.14	0.91 ± 0.03	0.80 ± 0.17	0.86 ± 0.08	0.86 ± 0.04
Text	TREC	0.41 ± 0.07	0.15 ± 0.04	0.27 ± 0.09	0.51 ± 0.06	0.27 ± 0.10	0.47 ± 0.13	0.49 ± 0.06
AG Nevs	0.76 ± 0.08	0.24 ± 0.02	0.36 ± 0.10	0.68 ± 0.08	0.64 ± 0.10	0.65 ± 0.08	0.66 ± 0.10
DBpedia	0.64 ± 0.19	0.07 ± 0.02	0.15 ± 0.08	0.55 ± 0.19	0.53 ± 0.21	0.51 ± 0.18	0.49 ± 0.15
IMDB	0.62 ± 0.04	0.50 ± 0.01	0.50 ± 0.01	0.60 ± 0.04	0.58 ± 0.04	0.59 ± 0.04	0.59 ± 0.04
A.1 Scale invariance
In this section, we delve into the concept of scale invariance in neural networks and its implications for model compositionality. We start by focusing on the effect of rescaling operations on the latent input encodings and demonstrate that, by construction, certain classifiers exhibit scale-invariance properties without the need for additional priors. Then, by examining the behavior of networks when subjected to a specific type of input manipulation, rescaling injection, we aim to demonstrate the robustness and versatility of neural networks in handling different scales of input data. As illustrated in the main manuscript, this is a key advantage in improving the adaptability of our method.
The softmax function, commonly used in neural classifiers, is known to be a temperature-controlled variant of the maximum function:
$$
\operatorname {s o f t m a x} (x) _ {i} = \frac {e ^ {\frac {y _ {i}}{T}}}{\sum_ {j} ^ {N} e ^ {\frac {y _ {j}}{T}}}. \tag {2}
$$
This means that the softmax temperature can be used to control the level of confidence of the classifier's predictions. In this study, we show that a similar effect can also be achieved by rescaling the latent encodings given as input to a trained (and frozen) classifier.
In order to demonstrate this, we first note that the rescaling factor,  $\alpha$ , can be factored out of the matrix multiplication in the Linear layers of the classifier. This can be represented mathematically as:  $\mathbf{y} = \alpha \mathbf{W}\mathbf{x} + b$ , where  $\mathbf{x}$  is the input latent encoding,  $\mathbf{W}$  is the weight matrix,  $b$  is the bias vector,  $\alpha$  is the rescaling factor, and  $\mathbf{y}$  is the output of the linear layer. This implies that the rescaling operation can be "pushed through" the classifier without affecting its final prediction as it becomes equivalent to some temperature value applied at the softmax level.
Furthermore, we investigate the effect of rescaling when non-linear activation functions are involved and posit that as long as the function has a monotonic interval, if we rescale all the dimensions by an amount similar to the mean scale of the encodings on which the classifier was trained, we end up in the monotonic interval, without losing the scale-invariance property.
In summary, our study provides empirical evidence that neural classifiers that utilize the softmax activation function can, in practice, maintain their scale-invariance properties when the input latent encodings are rescaled. This property is essential to our method, as it allows us to ignore the exact scale when decoding toward an L2-normalized absolute space.
Pre-trained models and scale-invariance We observed that large pre-trained models, such as transformers and resnets, are robust to internal rescaling of the encodings. Although we do not have a strong theoretical explanation for this phenomenon, we hypothesize that normalization layers and the linear separability of the information encoded in the angles instead of the norms may play a significant role. In Figure 12, we demonstrate the invariance a large transformer exhibits when the rescaling injection is applied at different layers: surprisingly, when the rescaling surpasses a certain threshold, the performance difference becomes negligible. These results further emphasize the robustness of these pre-trained models to the rescaling injection and suggest that the scale of the embedding is not a critical factor in their performance.
Rescale Injection We define the rescaling injection as the operation of artificially altering the scale of the features produced at a specific layer of the network. This is achieved by normalizing the embeddings to unit norm and then rescaling them by a factor of  $\alpha$ . By varying the value of  $\alpha$ , we can observe how the network's performance is affected at different scales. Through this empirical analysis, we aim to provide insight into the scale invariance properties of neural networks and their potential for use in model compositionality.
In Figure 13, we present experimental results investigating the scale invariance properties of neural networks. We trained simple multi-layer perceptrons (MLPs) composed of two hidden layers, with no normalization layers, using encodings produced by the Clip Vision transformer (clip-vit-base-patch32) on the CIFAR100 (fine) dataset. The MLPs were evaluated using different activation functions: cosine (blue), tanh (orange), and ReLU (green). The rescaling injection technique was applied directly to the input embeddings, rescaling them by  $\alpha$ .