SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Valentino Majorca $^{1,*}$  Luca Moschella $^{1,*}$
Antonio Norelli<sup>1</sup> Marco Fumero<sup>1</sup> Francesco Locatello<sup>2</sup> Emanuele Rodolà<sup>1</sup>
$^{1}$ Sapienza University of Rome  $^{2}$ Institute of Science and Technology Austria (ISTA)
While different neural models often exhibit latent spaces that are alike when exposed to semantically related data, this intrinsic similarity is not always immediately discernible. Towards a better understanding of this phenomenon, our work shows how representations learned from these neural modules can be translated between different pre-trained networks via simpler transformations than previously thought. An advantage of this approach is the ability to estimate these transformations using standard, well-understood algebraic procedures that have closed-form solutions. Our method directly estimates a transformation between two given latent spaces, thereby enabling effective stitching of encoders and decoders without additional training. We extensively validate the adaptability of this translation procedure in different experimental settings: across various trainings, domains, architectures (e.g., ResNet, CNN, ViT), and in multiple downstream tasks (classification, reconstruction). Notably, we show how it is possible to zero-shot stitch text encoders and vision decoders, or vice-versa, yielding surprisingly good classification performance in this multimodal setting.
Representation learning [Bengio et al., 2014] is a fundamental paradigm in the field of artificial intelligence, aimed at uncovering the underlying structure of complex data. One of the main goals of representation learning is to discover a robust representation of the data that is insensitive to certain transformations of the input. The Manifold Hypothesis [Fefferman et al., 2013] posits that real-world data lies on a low-dimensional non-linear manifold embedded in a high-dimensional space. Yet, a complication arises in modeling these non-linear manifolds: the learning process is usually influenced by stochasticities in the training dynamics and extrinsic factors that do not pertain to the data's core attributes, resulting in different representations for samples expected to be similar (e.g., different views of the same object, multiple translations of the same sentence, or even the exact same input sample). This is critical as it hinders knowledge transfer between these networks. Recently, the concept of relative representations [Moschella et al., 2023] has been proposed as a method for zero-shot communication between latent spaces that is invariant to these extrinsic factors. The idea is that latent spaces of neural networks trained on comparable data can be projected into the same relative space, derived from the distances between the data points. One of the main contributions of relative encoding is that it shows how the signal encoded in the angles with respect to a reduced set of data points (called anchors) is enough to capture the intrinsic shape of the latent space, reaching results on various benchmarks comparable to those using the original (absolute) encodings. As a consequence, they empirically demonstrate that different latent spaces that share the same data semantics (i.e., different representations of the same high-level concepts, such as images and their captions), mostly differ only by an angle-preserving transformation.
Figure 1: Zero-shot stitching of  $\mathbf{X}$  and  $\mathbf{Y}$  absolute spaces utilizing relative representations and our method (the estimation of  $\mathcal{T}$ ). Our approach does not require a decoder specifically trained on relative representations ( $dec_{\mathbb{Z}}$ ). Instead, we directly translate latent spaces, enabling the use of arbitrarily pre-trained decoders originally trained on absolute spaces.
Building on this intuition of the existence of a relatively simple transformation, we show the effectiveness and applications of directly translating between different latent spaces – provided that a partial (and possibly sparse) correspondence between data points is given. Remarkably, the process of seamlessly combining different neural networks, pre-trained on diverse datasets, modalities, architectures, domains and downstream tasks, proves unexpectedly straightforward. For instance, we show how it enables the ability to effectively integrate any pre-trained text encoder with any image classification head, and vice versa, without requiring any additional re-training or assumptions (indeed, Moschella et al. [2023] assumes the decoders are trained on relative representations). The method difference is emphasized in Figure 1. While zero-shot stitching with relative representations assumes the use of a single decoder specifically trained on a relative space, our method permits the reuse of the decoders originally trained on the absolute spaces.
- We explore the direct translation between latent spaces of distinct neural networks to enable latent communication. In particular, leveraging a semantic correspondence in the data, we directly translate for the first time across different trainings, architectures, and modalities. Notably, we obtain excellent stitching performances even in cross-modal settings, where we apply arbitrary text classifiers on top of pre-trained image encodings (and vice-versa).
- We show that different downstream tasks, namely classification and generation, require modeling different transformations to obtain the most out of the translation between their latent spaces;
Representations similarity Recent years have witnessed a growing consensus among researchers in the deep learning community that effective neural networks tend to learn similar representations for semantically similar data, regardless of the architecture, task, or domain in which they are applied. This idea is supported by a plethora of empirical studies [Moschella et al., 2023, Norelli et al., 2023, Morcos et al., 2018, Li et al., 2016, Kornblith et al., 2019, Bonheme and Grzes, 2022, Tsitsulin et al., 2020, Barannikov et al., 2022, Vulic et al., 2020a, Lample et al., 2018, Lenc and Vedaldi, 2015, Mikolov et al., 2013a, Antonello et al., 2021, Bengio et al., 2012, Movshovitz-Attias et al., 2017, Chang et al., 2022] and the phenomenon is particularly pronounced for large and wide models [Somepalli et al., 2022, Mehta et al., 2022]. Nevertheless, despite this intrinsic similarity, latent spaces can still exhibit extrinsic variations. Our work analyzes the possibility of translating these spaces from one to another, linking these extrinsic variations to different classes of transformation.
Manifold alignment Procrustes analysis has been instrumental in the alignment of latent spaces in deep neural networks [Wang and Mahadevan, 2008, 2009], particularly in Natural Language Processing (NLP) where it is well-known that latent spaces of different languages are isomorphic [Vulić et al., 2020b] and can be effectively aligned [Mikolov et al., 2013b, Xing et al., 2015]. Rooted
Figure 2: Method illustration on a synthetic example. Given a source space  $\mathbf{X}$ , the steps to translate it to a target  $\mathbf{Y}$  are sequentially applied as described in Section 3.2. Note that the translation is not perfect due to an arbitrary distortion of the data.
in shape analysis, this method efficiently uncovers correspondences between latent spaces of different models through the estimation of an optimal orthogonal transformation [Gower, 1975]. Previous works largely exploit Procrustes analysis to align latent spaces originating from models of the same architecture [Csiszarik et al., 2021], such as multi-lingual FastText embeddings [Bojanowski et al., 2017, Smith et al., 2017]. Instead, in this work, we extend the application of Procrustes analysis to network stitching in new domains, architectures, and even modalities for multiple downstream tasks.
Stitching and zero-shot Model stitching, which involves the combination of different neural networks to create a new model, has been a topic of active research in the field of representation learning. A key concept in this area is that of relative representations [Moschella et al., 2023, Norelli et al., 2023], which enables zero-shot stitching between different neural networks trained on semantically similar data. While this approach assumes the use of decoders trained on relative representations, our work removes this constraint by introducing a zero-shot mechanism for translating one absolute space to another without relying on a shared (relative) representation, enabling the stitching of arbitrarily trained models, further generalizable by assuming only the positive scale invariance of their decoder part. Previously, trainable stitching layers [Lenc and Vedaldi, 2015, Bansal et al., 2021, Csiszarik et al., 2021] have been introduced to allow for the combination of parts of different networks or to verify statements regarding latent space similarity. Other works [Gygli et al., 2021, an, 2021, Yaman et al., 2022, an, 2020] have proposed alternative methods for producing directly compatible and reusable network components without specific stitching layers. Here, we sidestep the need to create a new compatible representation and instead focus on obtaining a direct transformation to map from a source space to a target one to enable seamless network stitching. Concurrently to this work, a similar approach by [Lahner and Moeller, 2023] also targeted the direct alignment of representational spaces, focusing on the compatibility of models trained end-to-end.
Relative representation is a framework introduced in Moschella et al. [2023], which enables latent spaces of arbitrary neural models to communicate with each other. This is obtained by projecting the latent spaces into a common one, transitioning from an absolute coordinate frame to a relative space: each sample is represented as a function of a set of fixed samples denoted as anchor set. Specifically, the new representation is computed by independently projecting each sample point  $\mathbf{x}$  in the latent space  $\mathbf{X} \in \mathbb{R}^{n \times d}$ , into the anchor set  $\mathbf{A}_{\mathbf{X}} \subset \mathbf{X}$ . Formally, this is represented as
$$
\mathbf {X} _ {r e l} = \mathbf {X} _ {a b s} \cdot \mathbf {A} _ {\mathbf {X}} ^ {T}, \tag {1}
$$
where  $\mathbf{X}_{rel} \in \mathbb{R}^{n \times k}$ ,  $\mathbf{X}_{abs} \in \mathbb{R}^{n \times d}$  and  $\mathbf{A}_{\mathbf{X}} \in \mathbb{R}^{k \times d}$ . Samples in  $\mathbf{X}$  and in  $\mathbf{A}$  are rescaled to unit norm, i.e.  $\mathbf{x} = \frac{\mathbf{x}}{\|\mathbf{x}\|_2} \forall \mathbf{x} \in \mathbf{X}$  and  $\mathbf{a} = \frac{\mathbf{a}}{\|\mathbf{a}\|_2} \forall \mathbf{a} \in \mathbf{A}_{\mathbf{X}}$ .
We assume to have access to subsets  $\mathcal{A}_{\mathcal{X}} \subset \mathcal{X}$  and  $\mathcal{A}_{\mathcal{Y}} \subset \mathcal{Y}$ , with  $\mathcal{X}$  and  $\mathcal{Y}$  being the data distributions, and that there exists a correspondence  $\Gamma: \mathcal{A}_{\mathcal{X}} \mapsto \mathcal{A}_{\mathcal{Y}}$  between these two sets of parallel anchors. Parallel anchors act as a "Rosetta stone" [Norelli et al., 2023], meaning they establish a semantic correspondence between their respective spaces: an anchor sample in the first set represents the same high-level concept as its counterpart in the second set. This allows stitching together components of different models: i.e., merging independently trained encoder and decoder modules from different networks. The relative projection will map latent spaces into the same one as long as the core assumption that they differ by an angle-preserving transformation is satisfied. However, in
order to perform the stitching procedure in Moschella et al. [2023], decoders must be trained from scratch at least once to process samples in this shared relative space.
In this work, we overcome this need by substituting the costly retraining procedure with an efficient strategy to directly estimate the transformation necessary to map between spaces. Moreover, we relax the "angle-preserving" constraint by allowing for a broader class of transformations obtained via robust, closed-form algorithms.
3.2 Latent Space Translation
Consider two latent spaces,  $\mathbf{X} \in \mathbb{R}^{n \times d_1}$  and  $\mathbf{Y} \in \mathbb{R}^{n \times d_2}$ . Our objective is to estimate the transformation  $\mathcal{T}$  that translates  $\mathbf{X}$  into  $\mathbf{Y}$ :  $\mathbf{Y} = \mathcal{T}(\mathbf{X})$ , exploiting the semantic alignment between the two spaces. Throughout this work, we identify two main steps in the translation process: pre-processing the spaces and estimating the transformation  $\mathcal{T}$ , as outlined in Figure 2.
Pre-processing Generally, the two spaces may have different dimensionalities – in those cases, we zero-pad the smaller one to match the dimension of the other without changing its underlying structure [Williams et al., 2021]. Moreover, we standardize each feature in the encoding to have zero mean and unit variance (standard scaling) if not otherwise specified, whose statistics are computed only on the anchor sets for both source and target space, to perform the necessary de-normalization.
Estimating  $\mathcal{T}$  In Moschella et al. [2023], it is empirically shown that the spaces mostly differ by an angle-preserving transformation. Nevertheless, we broaden our investigation by considering different ways of obtaining  $\mathcal{T}$  to evaluate the robustness of that assumption and the versatility of our approach. Throughout our experiments, we primarily operate under the assumption that  $\mathcal{T}$  can be constrained to encode, at most, an affine transformation:  $\mathcal{T}(\mathbf{x}) = \mathbf{R}\mathbf{X} + \mathbf{b}$
This general formulation, without additional constraints, corresponds to our affine method in the experiments, and it is optimized via gradient descent. The other transformations are trivially obtained by progressively adding constraints on this one:
- linear. To model a linear transformation, we can just set the bias term to zero  $\mathbf{b} = \vec{0}$  and optimize via Least Square. Here we are both simplifying the class of transformations and switching from a gradient descent optimization to a closed-form procedure.
- 1-ortho. Additionally, we could require  $R$  to be orthogonal to encode an isometry. In this case, we obtain this by applying Singular Value Decomposition (SVD) on the corresponding  $R$  obtained by the linear solution. Through this, we aim to understand the implications of enforcing orthogonality on a transformation that was originally not constrained to be so, in a setting similar to Xing et al. [2015].
- ortho. To obtain the optimal orthogonal  $R$ , we apply Procrustes analysis [Gower, 1975].
This methodology facilitates efficient and precise zero-shot translation between disparate latent spaces. The transformation  $\mathcal{T}$ , derived solely from the subset of corresponding points, provides a robust and versatile foundation for model reuse and interoperability in diverse machine learning contexts.
4 Latent Communication via Translation
In this section, we evaluate the capabilities and effectiveness of our translation method through various scenarios, highlighting its applicability in diverse contexts. We present empirical results in three different novel settings: i) cross-architecture; ii) cross-modality; iii) autoencoding. In each case, the translation performance of each method for obtaining the transformation  $\mathcal{T}$  is evaluated against two baselines, the naive absolute one and the relative one.
**Stitching Procedure** In line with the zero-shot stitching concept outlined in Moschella et al. [2023], we combine independent encoders and decoders (e.g., classifiers, generators) without subsequent training or fine-tuning. This study does not necessitate a decoder trained on relative representations; instead, we directly employ the original decoders trained on absolute spaces. Each benchmark we perform follows the same procedure unless otherwise specified: we measure the mean performance over all the possible combinations of (encoder, decoder) for each test set in different settings:
Figure 3: Performance comparison of affine, linear, l-ortho, and ortho at varying number of anchors on classification accuracy. Results on CIFAR100 fine-grained. The same analysis for the generation case is in Figure 8 in the Appendix.
Table 1: Cross-architecture stitching with various methods for estimating  $\mathcal{T}$  and applying standard scaling. The stitched decoders are SVMs with a linear kernel. 5 runs for each encoder-decoder pair. (C) and (F) next to CIFAR100 indicate, respectively, coarse-grained and fine-grained. Please refer to the Appendix in Table 5 for additional results with MLPs as classification heads.
Dataset	No Stitching	absolute	relative	affine	linear	l-ortho	ortho
Vision	CIFAR10	0.95 ± 0.03	0.16 ± 0.22	0.80 ± 0.22	0.92 ± 0.05	0.88 ± 0.11	0.90 ± 0.09	0.93 ± 0.04
CIFAR100-C	0.85 ± 0.07	0.11 ± 0.21	0.54 ± 0.25	0.78 ± 0.09	0.73 ± 0.16	0.77 ± 0.11	0.81 ± 0.07
CIFAR100-F	0.76 ± 0.09	0.07 ± 0.21	0.30 ± 0.24	0.68 ± 0.11	0.62 ± 0.19	0.64 ± 0.16	0.71 ± 0.09
F-MNIST	0.88 ± 0.01	0.15 ± 0.20	0.63 ± 0.23	0.86 ± 0.01	0.83 ± 0.06	0.82 ± 0.05	0.85 ± 0.02
MNIST	0.96 ± 0.01	0.15 ± 0.21	0.50 ± 0.22	0.94 ± 0.01	0.89 ± 0.08	0.81 ± 0.11	0.91 ± 0.02
Text	TREC	0.87 ± 0.12	0.20 ± 0.06	0.36 ± 0.13	0.82 ± 0.12	0.74 ± 0.25	0.57 ± 0.25	0.79 ± 0.11
AG News	0.73 ± 0.09	0.25 ± 0.02	0.39 ± 0.13	0.65 ± 0.08	0.62 ± 0.08	0.61 ± 0.10	0.66 ± 0.10
DBpedia	0.78 ± 0.23	0.07 ± 0.01	0.16 ± 0.10	0.66 ± 0.24	0.62 ± 0.23	0.57 ± 0.23	0.66 ± 0.22
IMDB	0.61 ± 0.04	0.50 ± 0.01	0.51 ± 0.02	0.59 ± 0.04	0.57 ± 0.04	0.56 ± 0.03	0.59 ± 0.04
- no-stitch. The end-to-end performance of the decoder applied to the original space it was trained on. This is useful to establish un upper-bound in performances;
- **absolute.** The result of using the encodings without any transformation, we consider this as a probe for any pre-existing compatibility among encodings and, therefore, a lower-bound;
- translation. These are the results of the application of our latent translation method, with the estimation of  $\mathcal{T}$  via affine, linear, l-ortho and ortho.
In each instance, we use the same parallel anchors, that are uniformly chosen, in a quantity comparable with the dimensionality of the absolute representation.
4.1 Cross-Architecture
Firstly, we test our method in a cross-architecture setting, zero-shot stitching together encodings coming from a variety of pre-trained networks and their associated absolute decoders (classifiers). This scenario provides an extensive testing ground for our method and demonstrates its robustness across different architectures. Please refer to Table 8 in the Appendix for further results on cross-architecture stitching in generation tasks.
Experimental setting We consider a variety of Computer Vision (MNIST, Fashion MNIST, N24News, CIFAR10, CIFAR100) and Natural Language Processing (TREC [Hovy et al., 2001, Li and Roth, 2002], DBpedia [Auer et al., 2007], N24News [Wang et al., 2022], AG News [Zhang et al., 2015], IMDB [Maas et al., 2011]) datasets. For the text domain we consider 7 different language models as encoders (uncased and cased BERT [Devlin et al., 2019], Electra [Clark et al., 2020], RoBERTa base [Liu et al., 2019], ALBERT [Lan et al., 2020], and the text encoder of [Radford et al., 2021]), and for the image domain 6 encoders (RexNet100 [Han et al., 2020], 4 variations of ViT [Dosovitskiy et al., 2020], and the image encoder of [Radford et al., 2021]), all pre-trained and frozen.
Table 2: Cross-architecture stitching with various methods for estimating  $\mathcal{T}$  and applying L2 normalization. The stitched decoders are SVMs with linear kernel. 5 runs for each encoder-decoder pair. (C) and (F) next to CIFAR100 indicate, respectively, coarse-grained and fine-grained. Please refer to Table 6 in the Appendix for additional results with MLPs as classification heads.
Dataset	No Stitching	absolute	relative	affine	linear	l-ortho	ortho
Vision	CIFAR10	0.95 ± 0.03	0.16 ± 0.22	0.80 ± 0.22	0.93 ± 0.04	0.78 ± 0.27	0.88 ± 0.12	0.91 ± 0.09
CIFAR100-C	0.85 ± 0.07	0.11 ± 0.21	0.54 ± 0.25	0.79 ± 0.07	0.65 ± 0.25	0.73 ± 0.17	0.79 ± 0.10
CIFAR100-F	0.76 ± 0.09	0.07 ± 0.21	0.30 ± 0.24	0.69 ± 0.10	0.52 ± 0.25	0.62 ± 0.19	0.68 ± 0.13
F-MNIST	0.88 ± 0.01	0.15 ± 0.20	0.63 ± 0.23	0.86 ± 0.01	0.65 ± 0.23	0.83 ± 0.06	0.84 ± 0.05
MNIST	0.96 ± 0.01	0.15 ± 0.21	0.50 ± 0.22	0.94 ± 0.01	0.61 ± 0.23	0.90 ± 0.08	0.90 ± 0.04
Text	TREC	0.87 ± 0.12	0.20 ± 0.06	0.36 ± 0.13	0.82 ± 0.12	0.44 ± 0.20	0.74 ± 0.23	0.77 ± 0.12
AG News	0.73 ± 0.09	0.25 ± 0.02	0.39 ± 0.13	0.66 ± 0.08	0.56 ± 0.10	0.62 ± 0.08	0.64 ± 0.10
DBpedia	0.78 ± 0.23	0.07 ± 0.01	0.16 ± 0.10	0.66 ± 0.24	0.44 ± 0.20	0.62 ± 0.23	0.60 ± 0.22
IMDB	0.61 ± 0.04	0.50 ± 0.01	0.51 ± 0.02	0.59 ± 0.04	0.55 ± 0.03	0.58 ± 0.04	0.59 ± 0.04
The full encoder list can be found in Table 7 in the Appendix. For each dataset and for each encoder, we train an SVM classification head (decoder) on top of their specific encodings. We then proceed with the standard stitching procedure outlined in Section 4 and collect the results. Please see Table 8 in the Appendix for cross-architecture stitching in generation tasks, where we extend this analysis by verifying that our method works even across autoencoders of different bottleneck sizes.
Result analysis The stitching results are in Table 1. As expected, the absolute encodings obtain a score comparable to random guessing while also considering fewer encoder combinations out of the possible ones due to the dimensionality mismatch between some of them. Notably, these results show that the transformation relating to these pre-trained encoders is indeed mostly orthogonal: i) ortho and affine, the narrowest and the broadest transformation classes considered, are the better-performing translation methods. But while the former is obtained via a simple and efficient closed-form algorithm, the latter is SGD-optimized (Section 3.2). ii) the 1-ortho version improves or has small drops in performances over the linear transformation it is obtained from, confirming that the least squares procedure converges to an R which is almost orthogonal. Note that these results demonstrate the feasibility of combining pre-trained models without the need for retraining or fine-tuning, with negligible drops in performances across the board and without any additional assumption on the decoders. Please refer to Tables 5 and 6 in the Appendix for results with different decoders. In the Appendix (Figure 7), we extend the cross-architecture transfer to decoders trained on different domains (styles) of the same CIFAR10 dataset: the original one and a grayscale one.
Sensibility to Anchor Quantity The number of anchors is an essential parameter in our approach. In Figure 3, we evaluate how the quantity of these anchors impacts the residual error and the overall performance of our method for this experimental setting. This analysis offers insights into the optimal number of anchors necessary for efficient latent space translation.