SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the following we compare ASIF to traditional multimodal models based on contrastive training, CLIP and LiT. We then take a closer look to the classification of a single image, unpacking the relative representations and following the classification algorithm step by step. As a prelude to the above, we start by discussing the pretrained encoders and dataset forming the ASIF models we tested.
Pretrained encoders and multimodal dataset used. For our primary experiment, we utilized vision transformers as image encoders, pretrained either in a supervised manner (DEIT base, [44]) or in an unsupervised manner (DINO VITs8, [45]), on Imagenet 1k [46] and 21k [47] respectively. The embedding size was 768 for DEIT and 384 for DINO. This configuration aligns with that used in LiT [2], with the sole distinction being that, unlike LiT, we used a pre-trained, frozen text encoder. Regarding the text encoder, we employed the SentenceT transformer [48], trained on a dataset comprising more than 1 billion sentences obtained from the internet. We employed the first 1.6M entries of the Conceptual Caption dataset (CC12M, [49]) as our multimodal dataset. This dataset amasses images and filtered alt-texts collected from the internet. To optimize performance on a single Tesla T4 GPU, we limited our analysis to the initial 1.6M pairs. In our "scaling-laws" experiments, we also utilized DEIT tiny and small vision transformers [44] and two smaller SentenceT encoders.
Zero-shot performance. We assessed the quality of our ASIF multimodal model by comparing its zero-shot classification performance against CLIP and LiT on four datasets: CIFAR100, Imagenet, Imagenetv2, and PETS [46, 50-52]; see Table 1. We crafted label prompts as in Zhai et al. [2, Table 11].
Remarkably, we achieve competitive performance with CLIP and LiT using two frozen pretrained encoders and a fraction of the image-text pairs.
The two ASIF models reported in Table 1 differ for the vision encoder, that is respectively supervisedly (DEIT) and unsupervisedly pretrained (DINO). We tuned  $k$  and  $p$  on the ImageNet validation set, in both cases we used  $k = 800$  and  $p = 8$ . The comparison between the two versions is not completely fair since the visual transformer architecture of DINO is smaller (e.g. the codings are 384-dimensional rather than 768) but corroborates the effectiveness of ASIF with encoders pretrained using no supervision. In the appendix we report the results of a wider collection of ASIF models based on different visual and textual backbones.
Summary: Overall, we observe that our ASIF procedure can achieve competitive zero-shot results with a fraction of the image-text pairs used in prior work (assuming access to other pre-trained, even unsupervisedly, unimodal models).
ASIF scaling laws. In Figure 5 we show the full zero-shot accuracy trend on Imagenet as the size of the multimodal dataset grows for different choices of  $k$  and  $p$ . ASIF models become effective very quickly: we reach a non-trivial  $18\%$  classification accuracy using just the first 10,000 pairs in the multimodal dataset. We recall that building an ASIF model from scratch still requires a lot of unimodal data to pretrain the encoders, but now this data may come untied and even without any label.
We tested ASIF further with smaller image and text encoders on multimodal datasets of different sizes (10 $^2$  to 10 $^6$  image-text pairs from CC12M) to provide early evidence about ASIF scaling laws. We used DEIT tiny, small, and base vision trans
Figure 5: ASIF is a learning algorithm: Imagenet accuracy improves smoothly as we increase the size of the multimodal dataset. Colors show the impact of  $k$  and  $p$  (non-zeros, val exp).
formers [44], along with two smaller SentenceT encoders. As we see in Figure 5 and in the appendix, Imagenet classification accuracy grows smoothly with the size of the multimodal dataset, and performance does not saturate earlier with smaller encoders. These results are promising but still preliminary, further experiments with larger multimodal datasets are left for future work.
Summary: Our experiments show a steady improvement in ASIF's performance as the size of the multimodal dataset increases. Performance deteriorates with smaller encoders, but even then there is no sign of saturation or plateau.
Figure 6: ASIF representations are interpretable and amendable. Thorough analysis of the relative representations –the four vectors in green– produced by ASIF to assign the best caption to a given image  $x^{*}$ . Every dimension corresponds to the similarity of the input image (text) to a unique image (text) entry in the multimodal dataset. We can visualize the training samples leading to correct or incorrect predictions, which is useful to curate better data. For example, the second training sample is broken, we can remove it and produce an updated ASIF model in seconds.
Adjusting an ASIF model in seconds. Consider classifying EuroSAT [53] satellite images using ASIF. Initially, the zero-shot performance outperforms random chance but is not impressive (29.4% unsup. configuration, 10 classes). CLIP, while better, also falls short with a 54.1% performance rate.
Now, imagine we acquire 100 new image-text pairs from EuroSAT. Our goal is to develop a new multimodal model based on an updated training set, anticipating the need to process more satellite images in the future. With CLIP, this would require us to fine-tune or retrain the model from scratch using the updated dataset. In contrast, ASIF requires only to obtain and store the new pairs' embeddings. The model retains its usability for all the previous images, but it now also makes accurate predictions for satellite images, achieving an  $82.2\% \pm 2.0$  accuracy on EuroSAT.[2]
Similarly, if we need to remove samples from the multimodal training dataset—either because they are faulty (as shown in Figure 6) or because we have lost the license to use them—the process is as simple as deleting the corresponding embeddings to get a new model.
Summary: ASIF enables quick model adjustments and data handling without the need for retraining, unlike traditional models such as CLIP. This demonstrates its practicality in real-world scenarios, such as the emergence of a new setting or the loss of rights for assets used during training.
Deep dive into a classification. To better understand why and how the ASIF procedure works, we are going to follow step by step the single classification depicted in Figure 6, showing the entries of the multimodal dataset used to build the relative representations. For simplicity we assume  $k = 23$ .
We want to find the Imagenet label of the upper left image in Figure 6. The first step is to compute its relative representation with respect to the multimodal dataset, this procedure selects the 23 most similar images in our collection. The most similar are indeed triumphal archs that—as similarity decreases—turn to generic structures with archs or ancient monuments. No other image in the multimodal dataset is involved in the classification, we could replace all the entries in our collection except for these 23, and as long as no new image is more similar to the input image, every computation in the classification process would remain the same. This consistency of the ASIF procedure is in contrast to traditional approaches like CLIP and LiT, where the parameters defining the encoder computations are shaped by the whole multimodal dataset through contrastive learning.
Now, we approach the pivotal step of the ASIF procedure: jumping into the text space by treating the relative representation of the input image as if it were the relative representation of its ideal caption. The vector remains the same, but its meaning changed: now it signifies how much the ideal caption should be similar to 23 captions.
<sup>2</sup>Why does CLIP perform better zero-shot? Likely, its 400M pairs have more samples close to satellite images than ASIF's 1.6M samples from CC12M, evidenced by ASIF's drastic improvement with just 100 new samples.
The efficacy of this jump hinges entirely on the quality of the multimodal dataset. Looking at the 23 captions in Figure 6, we are confident that the most fitting candidate caption corresponds to one of the prompts associated with the triumphal_arch label in ImageNet, such as "a photo of a triumphal arch". Conversely, a dataset featuring 23 non-informative captions, such as file names "IMG_20180823.jpg" or camera settings "D90 18.0-70.0 mm f/3.5-4.5", would not allow the model to recognize the correct class even with the best image and text encoders. This limitation is actually a defining feature of the ASIF procedure, as the meaning attributed to the input is ultimately determined by the multimodal dataset and not by the encoders: by acting just on the image-text pairs, we have full control over the model's output.
Summary: Our simple (non-cherry picked) example showcases how the ASIF predictions can be easily attributed to specific examples in the multimodal training data by construction. This feedback can be used to explain predictions and grow high quality datasets in data centric AI, for example by inspecting which examples contribute to incorrect classifications.
The effectiveness of the ASIF procedure raises questions on the role of memory and retrieval in machine learning, while at the same time opens new opportunities for products based on multimodal models, opening many avenues for future works. In the following we will discuss these aspects.
Perception and interpretation disentangled. In ASIF there is no trace of the multimodal data in the weights of the encoders, which are pretrained on different unimodal datasets. Nonetheless, the relative representations and the outcome of any classification task fundamentally depend on the multimodal dataset. This state of affairs reflects the factorization of perception and interpretation in the two stages constituting an ASIF model; the encoding and the construction of the relative representations. Such factorization is desirable because it relieves the black-box neural encoders from the responsibility of attributing meaning to their inputs, as envisaged by Eco [54, Par. 3.3.1.3]. Therefore, we can consider the neural encoders as mere sensors and focus only on the second stage to analyze, explain, and act on the interpretations of an ASIF model, as shown in Figure 6.
Learning or retrieval? As we have seen, the ASIF procedure requires no training: it does not distill the multimodal data into any learnable parameter. Rather, it prescribes a rigid memorization of the embeddings of the multimodal dataset, where each entry has its fixed-size spot, similarly to a retrieval process. On the other hand it seems impossible to not describe ASIF as a learning algorithm; for instance it satisfies the fundamental characterization given by Mitchell [55]: the more the multimodal data the more ASIF improves, as we can clearly see in Figure 5. Ultimately, an ASIF model is functionally comparable to CLIP. ASIF blurs the border between learning and retrieval by questioning the effectiveness of storing information only in the weights, and rather advocates to combine learning representations with external memories. We encourage more work on memory augmented neural networks and towards understanding the implications of memory for generalization.
Generalization to new distributions. The empirical performance of ASIF calls for a discussion on zero-shot and out-of-distribution generalization in foundation models trained with huge data sets. Clearly, the performance of ASIF will depend strongly on the multimodal data used for alignment. As an example, the good performance on Imagenet may not be particularly surprising in light of the qualitative evaluation seen in Figure 6. There, our query image might as well had been part of our multimodal data set, as the semantic gap with its neighbours appears to be small. Despite this, our choice of downstream evaluation and pre-training data set is identical compared to prior work [2]. As such, while it appears clear that ASIF should fail when the semantic gap between downstream task and "training data" is large, it is unclear why it should be different for more standard models like CLIP [1] and LiT [2]: if a gap does exist, future work should work to address it. In the meanwhile, we recommend that future foundation models are benchmarked on significantly broader sets of downstream tasks, ideally with some analysis of the semantic distance between test and training data (if any). Alternatively, our results may be interpreted in light of the strong performance of unimodal models. There may be a limited benefit of training from scratch on less curated multimodal data sets compared to standard and well established unimodal data sets, although we posit that at even larger scales the difference may be more significant.
Limitations. The simple ASIF procedure presented here offers a strong baseline for multimodal models, but its performance still falls apart to CLIP and LiT when the multimodal dataset is abundant
and the cost of training is not a concern. Additionally, the large dimensionality of the relative representations, even if sparse, poses challenges for directly applying ASIF to tasks like text-to-image generation. We recognize that the experiments reported in this paper do not provide a comprehensive examination of the myriad of downstream tasks multimodal models are known to adeptly handle, it is important to note that such broad coverage was explicitly out of scope for this work. Our primary objective here was to introduce and justify the ASIF procedure, illustrating its effectiveness on the representative task of zero-shot classification. In making this choice we followed [2], that used the very same datasets to showcase the benefits of a locked image encoder, that is their main claim. We anticipate and welcome a more extensive evaluation of ASIF in the context of a wider range of tasks in future research endeavors.
Conclusions. We presented a simple procedure called ASIC to assemble a fully functional multimodal model like CLIP from two unimodal pretrained encoders and a collection of image-text pairs without tuning a single neuron. While achieving competitive zero-shot classification results with its predecessors using a fraction of the data, the proposed ASIC procedure enriches multimodal models with editability-a new model based on different pairs can be deployed in seconds-and interpretable representations. The effectiveness of ASIC models also clashes with the dominant view of a learning algorithm as a way to distill data into the parameters of a model, and raises questions on the role of memory and retrieval in machine learning.
AN, MF, and FL partially worked on ASIF when they were at Amazon Web Services in Tübingen, Germany.
This paper is financially supported by the PRIN 2020 project no.2020TA3K9N (LEGO.AI), PNRR MUR project PE0000013-FAIR, and ERC Grant no.802554 (SPECGEO).
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[2] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123-18133, 2022.
[3] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
[4] Antonio Norelli, Giorgio Mariani, Luca Moschella, Andrea Santilli, Giambattista Parascandolo, Simone Melzi, and Emanuele Rodola. Explanatory learning: Beyond empiricism in neural networks. arXiv preprint arXiv:2201.10222, 2022.
[5] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
[6] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843-852, 2017.
[7] Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 792–801, 2021.
In this appendix, we further showcase the interpretability of ASIF models when used for classification in Figure 7. Then we provide additional details for the scaling laws and EuroSAT experiments presented in the main paper, and report additional results about the impact of the size of the encoders (Table 2), and of the image training dataset. We also report further evidence that the ASIF construction is not overly sensitive to its hyperparameters. Lastly, we discuss more in detail the idea that captions of similar images are alike in Figure 10.
Figure 7: Interpretability of EuroSAT classifications through ASIF. Analysis of the classification outcome of two EuroSAT query images using ASIF. The scatter plot shows the samples in the training set closer to the query image and the candidate caption of the corresponding color. Image and text similarity are computed through cosine similarity in the visual space of DINO and the text space of SentenceT. The size of the marks is proportional to the product of the image and text similarity. The class chosen is the one with the largest total area. Below are shown the corresponding pairs from the training dataset CC12M. We can notice the distance between the EuroSAT dataset and the 1.6M samples of CC12M we used, many of the closest images are not from satellite and even then may have misleading descriptions, as image  $A$  in the second example. An interactive version of this plot for any ASIF classification can be obtained using our code demo attached in the supplementary material.
A Additional details on the scaling laws experiment
Models used in the scaling laws experiments. As discussed in the main paper, we tested ASIF with smaller image and text encoders to provide early evidence about ASIF scaling laws. We used three different instances of DEIT [44] vision transformers, the tiny (5.6M parameters, 192-dimensional embeddings), small (22M, 384), and base (87M, 768), and the original VITb16 vision transformer [56] (86M, 768). The DEIT models were pre-trained on a smaller dataset, the standardImagenet1k training set [46], while VITb16 was pretrained onImagenet21k [47]. As text encoders, we used smaller versions of SentenceT [48], with 23M and 33M parameters (both 384-dimensional embeddings), in contrast to the 110M parameters of the main model (768).
Figure 8 shows that, with smaller encoders producing smaller embeddings, we do not observe a performance saturation within 1.6M image-text couples. Further experiments with larger datasets are left for future work.
Impact of image pre-training data. In Table 2 we report the complete results of ASIF models using DEIT encoders [44]. We observe the expected positive correlation between the size of the encoders and the classification accuracy. Interestingly, ASIF with the largest instance of DEIT beats the one based on the standard VIT pretrained on Imagenet21k on three out of four of test datasets, while losing more than 10 points on CIFAR. These results may be interpreted in light of the similarity of the datasets we are using, with features useful to classify CIFAR images less overlapping with Imagenet1k features with respect to the other datasets.
Figure 8: ASIF performance does not saturate earlier with smaller encoders. Classification accuracy keeps growing without saturating but is lower for smaller models. Furthermore, we observe that the quality of the vision encoder is more relevant than the quality of the text encoder with respect to zero-shot Imagenet classification.
B Additional details on the EuroSAT experiment.
EuroSAT, a renowned benchmark for satellite image classification, serves as a testing ground for out-of-distribution generalization in zero-shot and few-shot scenarios [53]. The dataset contains 27,000 images labeled under ten categories. Our ASIF model with a DINO visual backbone (denoted as 'ASIF unsup' in table 1) achieved a zero-shot classification score of  $29.4\%$ . While significantly better than random chance, this modest performance is not surprising considering the scarce presence of satellite images in the CC12M dataset.
As a further experiment, we randomly selected 100 images from the EuroSAT dataset and incorporated them into our ASIF training set, raising the total to 1,500,100 image-text pairs and leaving 26,900 images for testing. We created captions for the EuroSAT images using the template "a satellite image of [CLASS NAME]". This way the ASIF model improves dramatically, reaching a classification accuracy of  $82.5 \pm 2.8\%$  on EuroSAT (average  $\pm$  standard deviation of 5 trials).
ASIF backbones (Params, pre-training data)	ImNet	CIFAR	PETS	ImNet-v2
DEITtiny (5.6M, Im1k) - STminiL6 (23M, see Sec. 3)	46.5	37.3	75.6	38.3
DEITsmall (22M, Im1k) - STminiL12 (33M, see Sec. 3)	59.3	46.0	80.4	50.3
DEITbase (87M, Im1k) - STbase (110M, see Sec. 3)	60.9	50.2	81.5	52.2
VITb16 (86M, Im21k) - STbase (110M, see Sec. 3)	55.4	63.3	71.5	45.6
Contrarily, CLIP [1], while demonstrating better zero-shot accuracy at  $54.1\%$ , is trained on a private dataset comprising 400 million images. This dataset may contain a larger number of satellite images than our 1.6 million subset of CC12M. Given the substantial improvement observed when we added just 100 EuroSAT images, it's reasonable to speculate that CLIP's enhanced performance might stem from its larger database of satellite images. However, confirming this theory is impossible due to the private nature of CLIP's training set.
We can, nevertheless, examine the presence of satellite images in the CC12M dataset. Using ASIF models' unique interpretability property, we can trace the training samples behind each classification. Figure 7 displays two EuroSAT samples, one classified correctly and the other not, along with the corresponding CC12M pairs responsible for the classifications. We note that our subset of CC12M is lacking in satellite images, and the few available often have misleading captions, such as a map of a drainage network tagged as "a satellite image of a canal, a river, a waterway, or a stream" instead of an urban area.
The images shown are an adaptation of the interactive plot to analyze any ASIF image classification we provided in the code demo attached in the supplementary material.
C ASIF sensibility to its hyperparameters
Finally, we present evidence about the sensitivity of the ASIF model to the hyperparameters  $p$  and  $k$ . Specifically, we show the hyperparameter search for PETS and CIFAR100 in Figure 9. Table C with results on the parameters fine-tuned on the two datasets reveals marginal improvements over the standard choice of  $k = 800$  and  $p = 8$ . This suggests that the ASIF model is relatively insensitive to the choice of these hyperparameters.
Table 2: Zero shot classification accuracy of ASIF models with different backbones. We observe that the ASIF procedure remains effective even with smaller encoders pre-trained on reduced visual datasets such as Imagenet1k.
Tuned on	Parameters p,k	CIFAR	PETS
PETS	(200,8)	60.9	72.3
CIFAR	(1600,6)	64.9	63
ImageNet1K	(800,8)	63.3	71.5
Table 3: Hyperparams search: tuning on each dataset per row.
Figure 9: Hyperparameters search over Left Pets, Right CIFAR100. Highlighted in the red square the performance achieved tuning onImagenet1K.