SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Antonio Norelli *
Marco Fumero* Valentino Maiorca* Luca Moschella*
Emanuele Rodolà* Francesco Locatello†
* Sapienza Università di Roma, dipartimento di Informatica
†Institute of Science and Technology Austria (ISTA)
CLIP [1] proved that aligning visual and language spaces is key to solving many vision tasks without explicit training, but required to train image and text encoders from scratch on a huge dataset. LiT [2] improved this by only training the text encoder and using a pre-trained vision network. In this paper, we show that a common space can be created without any training at all, using single-domain encoders (trained with or without supervision) and a much smaller amount of image-text pairs. Furthermore, our model has unique properties. Most notably, deploying a new version with updated training samples can be done in a matter of seconds. Additionally, the representations in the common space are easily interpretable as every dimension corresponds to the similarity of the input to a unique image-text pair in the multimodal dataset. Experiments on standard zero-shot visual benchmarks demonstrate the typical transfer ability of image-text models. Overall, our method represents a simple yet surprisingly strong baseline for foundation multimodal models, raising important questions on their data efficiency and on the role of retrieval in machine learning.
Large multimodal models such as CLIP [1] are rapidly becoming the standard for foundation models [3] in computer vision. This is largely due to their zero-shot and open-world capabilities that enable diverse suites of downstream tasks, from classification to detection and visual search.
Figure 1: ASIF is a simple recipe to align the representations of two frozen pre-trained encoders.
Overall, Radford et al. [1] demonstrated that treating image recognition as language interpretation allows generalizing to a multitude of tasks without training explicitly for them. By building an interpreter, CLIP changed the way computer vision is approached [4]: rather than extracting the "dog" label from an image like a CNN [5], CLIP tests the hypothesis of a dog being in the image against all the other hypotheses. The success of this image-text association is a testament to the power of deep learning: the CLIP model was the first of its kind, and building it required a joint training of two large neural encoders on a vast collection of image-text pairs.
Correspondence to Antonio Norelli <norelli@di.uniroma1.it>. A demo sufficient to reproduce the main results in the paper within minutes, even from smartphone, can be found here: https://github.com/noranta4/ASIF
Still, training neural networks at such scale presents several challenges beside the obvious infrastructure and training costs. Notably, it requires collecting massive training sets, making it difficult to interpret the predictions of the model in light of their training data. Additionally, the training assets are often not owned by the institution training the model [6].
Figure 2: The ASIF construction. An ASIF model is defined by two unimodal pretrained encoders and a collection of coupled embeddings. This is sufficient to compare elements from different modes through representations made of similarities with ground-truth pairs:  $y_{j}$  is more similar to  $x^{*}$  than  $y_{i}$ .
This introduces several additional challenges, from reproducibility to the difficulty of ensuring that an asset owner can remove their data from the model [7-11]. Overall, these considerations make large multimodal models relatively inaccessible to researchers and practitioners until checkpoints are released or access to demo is granted. And even then, the ability to tweak the models by adding or removing training data or to interpret their results is limited.
In this paper, we present ASIC, a simple non-parametric procedure that turns pretrained uni-modal image and text encoders into a multimodal model using a relatively small collection of image-text pairs and no additional training, as depicted in Figure 1. The resulting model is functionally equivalent to CLIP, effectively producing aligned representations of images and their captions.
Intuition. The key insight is that captions of similar images are themselves similar (Fig. 3), and therefore a representation crafted using just similarities to ground-truth multimodal pairs is quasi
mode-invariant (Fig. 2).
The ASIF procedure is not only efficient but also has several intriguing features built-in. One of the key advantages is the ability to easily edit the model - adding new image-text pairs or removing outdated ones is as simple as encoding or canceling the corresponding embeddings. Furthermore, the multimodal representations are highly interpretable, as each dimension corresponds to the similarity of the input to a specific entry in the multimodal dataset.
Contribution. Our results are surprising and raise several questions. Despite (1) the simplicity of the approach, (2) a multimodal dataset that is up to 250 times smaller than in prior work and (3) the lack of actually training the model on multimodal data, ASIF achieves zero-shot classification accuracy on downstream datasets that is comparable to CLIP [1, 2]. This raises important questions on the data efficiency in foundation models, making ASIF a very powerful and cheap baseline for future work, and opening new doors for data centric AI [12].
In summary, we:
- Introduce the ASIF procedure, which turns two pretrained unimodal black-box encoders into an interpretable multimodal model without tuning a neuron.
- Demonstrate the effectiveness of ASIF models on zero-shot image classification tasks, where they achieve performance in the same ballpark of CLIP with significantly fewer image-text pairs.
- Discuss the unique properties of ASIF, its implications on the role of memory and retrieval in machine learning, and the new opportunities it opens.
Figure 3: Captions of similar images are themselves similar. Distribution of similarities of  $100\mathrm{k}$  embedded pairs in the training set versus the above image and caption embeddings. We highlighted in orange the 1000 pairs with the highest image similarity.
<sup>1</sup>CLIP [1] experiments used from 400M to 15M captioned images as training samples, LiT [2] from 901M to 10M. Our experiments use 1.6M.
Figure 4: Zero shot classification with ASIF. In this example we determine the best caption for the image  $x^{*}$  from the two candidates,  $y_{i}$  and  $y_{j}$ . a. Compute and store the embeddings of the multimodal dataset and the test samples. b. Compute the relative representation of the test image and the candidate captions. c. We consider the relative representation of  $x^{*}$  with respect to the image collection  $x_{1},\ldots ,x_{n}$  as if it was the relative representation of  $y^{*}$  - the ideal caption for  $x^{*}$  - with respect to the corresponding caption collection  $y_{1},\ldots ,y_{n}$ . d. We choose the candidate caption most similar to the ideal one.
2 Aligning Pre-Trained Models with ASIF
In the following we present how a collection of captioned pictures implicitly defines a common space for images and texts through relative representations [13], allowing to build a multimodal model without training. Here we focus exclusively on vision and language as modalities due to the wider availability of relevant models and paired data. However, we anticipate that our procedure could be more generally applicable. Indeed, subsequent research by other teams has already utilized ASIF as a baseline in other modalities, such as audio [14]. Before delving into the specifics of our method, we will briefly review existing techniques for establishing this common space.
Contrastive training to build a common space. With multimodal models, we refer to architectures that embed inputs of diverse modes into the same space. The better is a multimodal model, the closer are representations of different modes of the same object. So far, this common space has been obtained as the result of a contrastive training of one [2] or both the neural mode encoders [1, 15]. Using a collection of image-text pairs as training set, a contrastive loss promotes the closeness of image and text embeddings from the same pair, while spacing out the ones from distinct pairs. Zhai et al. [2] train just the text encoder to match the image embeddings of a pretrained visual encoder. Once the encoders are trained, a multimodal model can be adapted to perform any visual classification task just by crafting a caption for each label and selecting the one with the embedding closer to the embedding of the input image (zero-shot image classification).
Relative representations. Our idea to build a common latent space is to use a collection of coupled data as a "rosetta stone" between modalities, and represent each new data point as its similarities to the points of the same modality in the collection. In other words, we compute a relative representation for each new data point:
Definition 2.1. Given an encoder  $E: X \to \mathbb{R}^d$  and a subset of samples  $\{x_1, \ldots, x_n\}$  denoted as anchors, we define the relative representation of  $x'$  as the  $n$ -dimensional vector:
$$
x ^ {\prime} = \left[ \operatorname {s i m} \left(x ^ {\prime}, x _ {1}\right), \dots , \operatorname {s i m} \left(x ^ {\prime}, x _ {n}\right) \right]
$$
for some similarity function sim, e.g. the cosine similarity.  $x_{i} \in X$  are input samples, e.g. images or texts;  $x_{i} \in \mathbb{R}^{d}$  are the embeddings of  $x_{i}$  obtained with the encoder  $E$ , i.e. the absolute representations of  $x_{i}$ ; while  $x_{i} \in \mathbb{R}^{n}$  are the relative representations of  $x_{i}$ .
We observe that when each anchor is available in two or more modalities, we can compute relative representations of samples from those modalities using the same subset of anchors. Most notably, these relative representations will all live in the same space, even when they are representing samples from different modalities. This foundational insight is illustrated in Figure 2.
Relation with Kernel methods. Definition 2.1 may not look surprising to readers familiar with the literature on kernel methods [16]. Instead of presenting  $x'$  as a kernel, we say it is a relative representation to stress that (1) we want to explicitly represent the coordinates in our ASIC procedure as opposed to operating in an implicit feature space and (2) we do not aim at learning regression parameters, although we note this may help with the inference speed. Instead, we rely on a simpler procedure that may be interpreted as a hard version of the Watson-Nadaraya [17, 18] regression with a distance threshold. Nevertheless, it is worth noting that while the ASIC procedure hinges on the ability to compute similarities between samples, such computation can be achieved using a kernel function, thus sidestepping the need for explicit representations generated by unimodal encoders. Although integrating kernel methods could offer benefits, as alluded to earlier, delving into these prospects is beyond the scope of this paper. Our central focus remains on illustrating how single-domain pre-trained networks can be aligned without additional training.
ASIF: relative representations inducing a meaningful common space. Consider the embedding spaces of any two good text and visual encoders, we expect captions of images that are close in the visual space to be themselves close in the language space. This fact makes a representation defined in terms of similarities against a set of ground-truth multimodal pairs almost mode-invariant, i.e. an image and its caption share almost the same representation.
That is why we can assign the best caption to a new image  $x^{*}$  just by performing nearest neighbors in this new space: we can consider the relative representation of  $x^{*}$  respect to the image collection  $(x_{1},\ldots ,x_{n})$  as if it was the relative representation of its ideal caption  $y^{*}$  with respect to the counterpart collection  $(y_{1},\dots ,y_{n})$ , see Figure 4. The whole procedure to set up an ASIC model and use it to find the best caption for a new image follows.
ASIF recipe. Ingredients:
- Two good encoders, each mapping a single data modality to a vector space. Let  $X$  and  $Y$  be the mode domains, for instance a pixel space and a text space, we need  $E_1: X \to \mathbb{R}^{d_1}$  and  $E_2: Y \to \mathbb{R}^{d_2}$ .
- A collection of ground truth multimodal pairs:  $D = \{(x_1, y_1), \ldots, (x_n, y_n)\}$ , for instance captioned images.
Procedure to find the best caption among a set of original ones  $\hat{Y} = \{\hat{y}_1,\dots ,\hat{y}_c\}$  for a new image  $x^{*}$ :
1. Compute and store the embeddings of the multimodal dataset  $D$  with the encoders  $E_1, E_2$  and discard  $D$ . Now in memory there should be just  $D_E = \{(x_1, y_1), \ldots, (x_n, y_n)\}$ ;
2. Compute the  $n$ -dimensional relative representation for each candidate caption  $\hat{y}_i = [\mathrm{sim}(\hat{y}_i, y_1), \dots, \mathrm{sim}(\hat{y}_i, y_n)]$ , where sim is a similarity function, e.g. cosine similarity. Then for each  $\hat{y}_i$  set to zero all dimensions except for the highest  $k$ , and raise them to  $p \geq 1$ . Finally normalize and store the processed  $c$  vectors  $\hat{y}_i$ . Choose  $k$  and  $p$  to taste, in our experiments  $k = 800$  and  $p = 8$ ;
3. Compute the relative representation of  $x^{*}$  using the other half of the embedded multimodal dataset  $D_{E}$  and repeat the same processing with the chosen  $k$  and  $p$ ;
4. We consider the relative representation of the new image  $x^{*}$  as if it was the relative representation of its ideal caption  $y^{*}$ , i.e. we define  $y^{*} \coloneqq x^{*}$ . So we choose the candidate caption  $\hat{y}_i$  most similar to the ideal one, with  $i = \operatorname{argmax}_i(\operatorname{sim}(y^*, \hat{y}_i))$ .
To assign one of the captions to a different image  $x^{**}$  repeat from step 3.
Properties of ASIF models. The above construction yields several intriguing properties for free:
No training and "data centric". As we have seen, an ASIC model is built on top of two independently pretrained encoders and the embeddings of a multimodal dataset, and so without training or finetuning any neuron. Being deployable or updatable in seconds, an ASIC model is radically "data centric" [12]. For example, it is trivial to adjust the model by adding or forgetting specific samples. The latter use-case is particularly important, as the right to use specific assets may change over time and removing the effect of specific samples from a trained network requires sophisticated forgetting techniques, e.g. [7-11]. In our procedure, the encoders should be pre-trained with established data sets that do not change over time, while removing the effect of a multimodal pair is as simple as deleting its embeddings.
Data efficiency: Being able to exploit two pretrained encoders, ASIF models require far less ground-truth multimodal pairs to become effective. As confirmed by our experiments, ASIF reaches competitive zero-shot performance on diverse classification benchmarks by using a fraction of the multimodal data of its predecessors, reaching a respectable accuracy even with thousands of pairs (we refer to Section 3 for more details). This is in line with classical work in computer vision, where prototypical networks [19] are a strong baseline in the extremely few-shot regime.
Interpretability: Sparse relative representations make the ASIF models interpretable classifiers. In fact, we can trace back every prediction to a small set of data points in the multimodal dataset -corresponding to the dimensions that are nonzero in both the image and the label relative representations- accountable for the outcome (at most  $k$ ), see Figure 6. This enables visualizations of the relevant samples contributing to a correct or incorrect prediction at no extra cost, in stark contrast with other approaches that are often costly [20] and fragile [21, 22].
Relation to k-Nearest Neighbors. Like k-NN, ASIF is a non-parametric supervised learning algorithm that requires an explicit representation of every entry in the training dataset at test time. Differently from k-NN, ASIF can perform open-ended classification, as shown e.g. in Figure 4 with two competing brand new captions. Indeed, ASIF is functionally equivalent to CLIP, and can function as a drop-in replacement in applications using CLIP-like models.
Implementation that scales. Clearly, our method pays the price of avoiding training with a larger memory footprint and increased computation at inference time, since we need to compute not only the embeddings but also the cosine similarities against the multimodal dataset. As such, our approach should not be considered a general one-stop replacement for CLIP, although in our experiments we managed to scale ASIF to 1.6M pairs while maintaining a reasonable inference speed. Our non-optimized implementation of ASIF is less than  $2\mathrm{x}$  slower than CLIP. On a positive note, there are two well-established techniques that could radically enhance the efficiency of ASIF and potentially enable scalability to billions of entries. The memory footprint required to store all the embeddings of the multimodal dataset can be dramatically reduced using product quantization [23], while inverse indexing [24] can be used to circumvent the need for computing the cosine similarities against the entire dataset at test time. These techniques are both implemented e.g. in the FAISS library [25]. Finally, we find that the distribution of pairs chosen during inference is rather short-tailed, presenting opportunities to massively prune the model even at deployment time, deleting from memory the data that is never used. It should be noted, however, that the assessment of the performance of large-scale optimized ASIF models is beyond the scope of this work. While it is an interesting direction for future research, the focus of our current study is on establishing the potential the ASIF method.
2.1 Design choices and implementation of ASIF models.
Curating the multimodal dataset. While neural models like CLIP or LiT are defined just by the weights of the two encoders, to univocally identify an ASIF model we also need the embeddings of the multimodal dataset. Even if two image-text ASIF models rely on the very same encoders, they comply to different visual understandings of the world if they are based on different collections of image-text pairs, therefore achieving different performances on the same tasks. Unlike conventional neural vision models, ASIF enables effective curation of the training dataset through swift iterations, given the absence of training and the smaller datasets. Furthermore, ASIF provides the means to assess the impact of each training pair, as demonstrated in Figure 6.
Salient hyperparameters. While using the raw relative representations already produces an ASIF multimodal model with non-trivial capabilities, we found that two simple treatments greatly improve performance and efficiency, and also foster interpretability.
Method	Dataset size	ImageNet	CIFAR100	Pets	ImageNet v2
CLIP [1]	400M (private)	68.6	68.7	88.9	-
CLIP [1]	15M (public)	31.3	-	-	-
LiT [2]	10M (public)	66.9	-	-	-
CLIP( Zhai et al. 2, uu)	901M (private)	50.6	47.9	70.3	43.3
LiT [2]	901M (private)	70.1	70.9	88.1	61.7
ASIF (sup vis. encoder)	1.6M (public)	60.9*	50.2	81.5	52.2
ASIF (unsup vis. encoder)	1.6M (public)	53.0*	46.5	74.7	45.9
* We used a subset of the ImageNet validation set to tune the two hyperparameters of ASIF which were then used on the other data sets. The number reported in the table is a test set. When tuning on different datasets, accuracies stay consistent, see the appendix.
Table 1: Zero shot classification accuracy of different multimodal designs. CLIP and LiT implementations vary by dataset and the visual transformer used as image encoder. The first CLIP and LiT entries use a VITb16 as ASIF, the last CLIP and LiT entries use a VITb32 (larger patch size). The public dataset of CLIP is a curated subset of YFCC100m [38], while LiT and ASIF use CC12M.
i) Sparsification. We set to 0 all the entries of the  $n$ -dimensional relative representation except for the top  $k$ . In our experiments  $n$  and  $k$  are respectively in the order of millions and hundreds. In this way we cut off the small noisy signals from the dissimilar entries, that accumulated during comparisons would destroy the signal from the few similar entries. Furthermore we get highly interpretable representations that can be efficiently compared, since we have just  $k$  nonzero features, each one linked to a single entry in the multimodal dataset.
ii) Exponentiation. We raise all the nonzeroed similarities  $\sin (x', x_i)$  to  $p$ , with  $p \geq 1$ . This nonlinearity weighs more the contribution of the most similar entries in the relative representation.
Besides the pivotal choice of the ground-truth multimodal pairs, the number of non-zero elements  $k$  and the exponent  $p$  are the salient hyperparameters to consider when deploying an ASIF model. In general, we found that picking a  $p \neq 1$  may help, while choosing a  $k \ll n$  is always crucial. For more details see Section 3.
2.2 Closely Related Works
Classics. In [26], Stanley Ulam affirms that a mathematical formalization of the word "as" on a par with the connectives "and", "or", "implies" and "not"—would be a key milestone to artificial intelligence. This idea of analogies being at the core of cognition is shared by Hofstadter [27], who states that a concept is a collection of analogies, in line with what the ASIF procedure prescribes.
Retrieval augmented foundation models. Recent works in NLP enhance unimodal language models with retrieval to reduce the size of the architectures and the training dataset while also making results more transparent [28, 29]. Our work is in line with this view, that the ASIF procedure extends to the multimodal case. Importantly, ASIF offers a new perspective on data centric AI [12], where data and the external memory implement the alignment between modalities. Networks with discrete Key-Value bottlenecks [30] are closely related to our approach, with the critical differences that our memory is not learned and that their decoder is trained for classification. Retrieval and memory-augmented models have also been successful in Reinforcement Learning [31], physical reasoning [32], and code generation [33]. Finally, we notice that making predictions on new samples by exploiting the similarity with a dictionary of previous data points is a common approach in computer vision [19] for few-shot learning. Our procedure is also related to compressed sensing algorithms where a signal (in our case an image) is sensed as a sparse combination of fixed atoms [34, 35] with an iterative projection procedure [36, 37] and only transmitting the coefficients to the receiver (text modality).
Learning multimodal models. Following the intuition outlined by early works on aligning text and image embeddings [39, 40], today large multimodal models are conquering the computer vision scene thanks to their wide applicability and easy transfer to new downstream tasks [1, 2, 15, 41].
We identify two key leaps respect to traditional models like ResNet [42]: (i) Free text allows to learn visual concepts beyond a finite set of predefined categories and to exploit the structure of language to compose them, as masterfully seen in Dall-E [43]. (ii) The recognition tag transitioned from being an output pulled out of the image by the neural stack (the label) to become an input that should be interpreted, and therefore processed by its own encoder (the free text caption). This corresponds to an epistemological perspective shift, as discussed by Norelli et al. [4]. Data and learning efficiency are clear challenges for large multimodal models, that often require hundreds of millions of examples. Efforts such as [2, 29] attempt to reduce this. ASIF presents a different take on this problem, showing how much can be achieved by simply remembering the training data efficiently.
3 Empirical Evidence