Title: ASIF: Coupled Data Turns Unimodal Models to Multimodal without Training
Venue: NeurIPS 2023
Accepted: yes
Keywords: Representation learning

Multimodal models

Analogy

Sparsity

Nonparametric

Relative representations

Language

Semiotics
Abstract: CLIP proved that aligning visual and language spaces is key to solving many vision tasks without explicit training, but required to train image and text encoders from scratch on a huge dataset. LiT improved this by only training the text encoder and using a pre-trained vision network. In this paper, we show that a common space can be created without any training at all, using single-domain encoders (trained with or without supervision) and a much smaller amount of image-text pairs. Furthermore, our model has unique properties. Most notably, deploying a new version with updated training samples can be done in a matter of seconds. Additionally, the representations in the common space are easily interpretable as every dimension corresponds to the similarity of the input to a unique entry in the multimodal dataset. Experiments on standard zero-shot visual benchmarks demonstrate the typical transfer ability of image-text models. Overall, our method represents a simple yet surprisingly strong baseline for foundation multi-modal models, raising important questions on their data efficiency and on the role of retrieval in machine learning.

ASIF: Coupled Data Turns Unimodal Models to Multimodal without Training

Abstract

CLIP [1] proved that aligning visual and language spaces is key to solving many vision tasks without explicit training, but required to train image and text encoders from scratch on a huge dataset. LiT [2] improved this by only training the text encoder and using a pre-trained vision network. In this paper, we show that a common space can be created without any training at all, using single-domain encoders (trained with or without supervision) and a much smaller amount of imagetext pairs. Furthermore, our model has unique properties. Most notably, deploying a new version with updated training samples can be done in a matter of seconds. Additionally, the representations in the common space are easily interpretable as every dimension corresponds to the similarity of the input to a unique imagetext pair in the multimodal dataset. Experiments on standard zero-shot visual benchmarks demonstrate the typical transfer ability of image-text models. Overall, our method represents a simple yet surprisingly strong baseline for foundation multimodal models, raising important questions on their data efficiency and on the role of retrieval in machine learning.

1 Introduction

Large multimodal models such as CLIP [1] are rapidly becoming the standard for foundation models [3] in computer vision. This is largely due to their zero-shot and open-world capabilities that enable diverse suites of downstream tasks, from classification to detection and visual search.

CLIP

LiT

ASIF

Trained

from

scratch

Image

Text

Image

Text

Image

Text

Frozen

pretrained

Overall, Radford et al. [1] demonstrated that treating image recognition as language interpretation allows generalizing to a multitude of tasks without training explicitly for them. By building an interpreter, CLIP changed the way computer vision is approached [4]: rather than extracting the 'dog' label from an image like a CNN [5], CLIP tests the hypothesis of a dog being in the image against all the other hypotheses. The success of this image-text association is a testament to the power of deep learning: the CLIP model was the first of its kind, and building it required a joint training of two large neural encoders on a vast collection of image-text pairs.

Correspondence to Antonio Norelli <norelli@di.uniroma1.it>. A demo sufficient to reproduce the main results in the paper within minutes, even from smartphone, can be found here: https://github.com/noranta4/ASIF

Still, training neural networks at such scale presents several challenges beside the obvious infrastructure and training costs. Notably, it requires collecting massive training sets, making it difficult to interpret the predictions of the model in light of their training data. Additionally, the training assets are often not owned by the institution training the model [6].

32

E1

C1

·C

水

Yi

M3

yj

二

C

*

yj

y2

E2

yi

yj

y3

3

Yi

y1

This introduces several additional challenges, from reproducibility to the difficulty of ensuring that an asset owner can remove their data from the model [7-11]. Overall, these considerations make large multimodal models relatively inaccessible to researchers and practitioners until checkpoints are released or access to demo is granted. And even then, the ability to tweak the models by adding or removing training data or to interpret their results is limited.

In this paper, we present ASIF, a simple non-parametric procedure that turns pretrained uni-modal image and text encoders into a multimodal model using a relatively small 1 collection of image-text pairs and no additional training, as depicted in Figure 1. The resulting model is functionally equivalent to CLIP, effectively producing aligned representations of images and their captions.

Intuition. The key insight is that captions of similar images are themselves similar (Fig. 3), and therefore a representation crafted using just similarities to ground-truth multimodal pairs is quasi mode-invariant (Fig. 2). Freshly picked

The ASIF procedure is not only efficient but also has several intriguing features built-in. One of the key advantages is the ability to easily edit the model - adding new image-text pairs or removing outdated ones is as simple as encoding or canceling the corresponding embeddings. Furthermore, the multimodal representations are highly interpretable, as each dimension corresponds to the similarity of the input to a specific entry in the multimodal dataset.

Contribution. Our results are surprising and raise several questions. Despite (1) the simplicity of the approach, (2) a multimodal dataset that is up to 250 times smaller than in prior work and (3) the lack of actually training the model on multimodal data, ASIF achieves zero-shot classification accuracy on downstream datasets that is comparable to CLIP [1, 2]. This raises important questions on the data efficiency in foundation models, making ASIF a very powerful and cheap baseline for future work, and opening new doors for data centric AI [12].

In summary, we:

Introduce the ASIF procedure, which turns two pretrained unimodal black-box encoders into an interpretable multimodal model without tuning a neuron.

flowersarepoured

into a copper

still.

Image similarities

104

103

10.44

102

101

100

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Textsimilarities

104

1

103

!0.10

!0.43

102

101

100

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Demonstrate the effectiveness of ASIF models on zero-shot image classification tasks, where they achieve performance in the same ballpark of CLIP with significantly fewer image-text pairs.

Discuss the unique properties of ASIF, its implications on the role of memory and retrieval in machine learning, and the new opportunities it opens.

Multimodal dataset

Test

samples

y2

32

2*

.09

-.14

.54

2*

Absoluterepresentations

= Image Encoder

Relativerepresentations

yi

triumphal arch

tourists visit a

9i

Text Encoder

triumphal arch

tourists visit a

y,y

.74

.11

-.08

.57

yi

91

yj

a sunny day in

the forest

yj

a sunny day in

the forest

y1.y

.49

-.07

.06

.87

yj

y1

y2

yn

a

b

C

.11

.71

.09

-.14

.54

*C

* used as if it were y*

Weselectyisince(yi,y*)>(yi，y*)

IDEALCAPTION

UNKNOWN

.11

.71

.09

-.14

··

.54

IDEALCAPTION

UNKNOWN

y*

yi

tourists visit a

triumphal arch

.08

.74

.11

-.08

···

.57

yi

yi

touristsvisita

triumphalarch

yi

9j

a sunny day in

the forest

.81

.49

-.07

.06

···

.87

yj

yj

a sunny day in

the forest

9j

C

2 Aligning Pre-Trained Models with ASIF

In the following we present how a collection of captioned pictures implicitly defines a common space for images and texts through relative representations [13], allowing to build a multimodal model without training. Here we focus exclusively on vision and language as modalities due to the wider availability of relevant models and paired data. However, we anticipate that our procedure could be more generally applicable. Indeed, subsequent research by other teams has already utilized ASIF as a baseline in other modalities, such as audio [14]. Before delving into the specifics of our method, we will briefly review existing techniques for establishing this common space.

Contrastive training to build a common space. With multimodal models, we refer to architectures that embed inputs of diverse modes into the same space. The better is a multimodal model, the closer are representations of different modes of the same object. So far, this common space has been obtained as the result of a contrastive training of one [2] or both the neural mode encoders [1, 15]. Using a collection of image-text pairs as training set, a contrastive loss promotes the closeness of image and text embeddings from the same pair, while spacing out the ones from distinct pairs. Zhai et al. [2] train just the text encoder to match the image embeddings of a pretrained visual encoder. Once the encoders are trained, a multimodal model can be adapted to perform any visual classification task just by crafting a caption for each label and selecting the one with the embedding closer to the embedding of the input image (zero-shot image classification).

Relative representations. Our idea to build a common latent space is to use a collection of coupled data as a 'rosetta stone' between modalities, and represent each new data point as its similarities to the points of the same modality in the collection. In other words, we compute a relative representation for each new data point:

Definition 2.1. Given an encoder E : X → R d and a subset of samples { x 1 , . . . , x n } denoted as anchors, we define the relative representation of x ′ as the n -dimensional vector:

for some similarity function sim, e.g. the cosine similarity. x i ∈ X are input samples, e.g. images or texts; x i ∈ R d are the embeddings of x i obtained with the encoder E , i.e. the absolute representations of x i ; while x i ∈ R n are the relative representations of x i .

We observe that when each anchor is available in two or more modalities, we can compute relative representations of samples from those modalities using the same subset of anchors. Most notably, these relative representations will all live in the same space, even when they are representing samples from different modalities. This foundational insight is illustrated in Figure 2.

Relation with Kernel methods. Definition 2.1 may not look surprising to readers familiar with the literature on kernel methods [16]. Instead of presenting x ′ as a kernel, we say it is a relative representation to stress that (1) we want to explicitly represent the coordinates in our ASIF procedure as opposed to operating in an implicit feature space and (2) we do not aim at learning regression parameters, although we note this may help with the inference speed. Instead, we rely on a simpler procedure that may be interpreted as a hard version of the Watson-Nadaraya [17, 18] regression with a distance threshold. Nevertheless, it is worth noting that while the ASIF procedure hinges on the ability to compute similarities between samples, such computation can be achieved using a kernel function, thus sidestepping the need for explicit representations generated by unimodal encoders. Although integrating kernel methods could offer benefits, as alluded to earlier, delving into these prospects is beyond the scope of this paper. Our central focus remains on illustrating how single-domain pre-trained networks can be aligned without additional training.

ASIF: relative representations inducing a meaningful common space. Consider the embedding spaces of any two good text and visual encoders, we expect captions of images that are close in the visual space to be themselves close in the language space. This fact makes a representation defined in terms of similarities against a set of ground-truth multimodal pairs almost mode-invariant, i.e. an image and its caption share almost the same representation.

That is why we can assign the best caption to a new image x ∗ just by performing nearest neighbors in this new space: we can consider the relative representation of x ∗ respect to the image collection ( x 1 , . . . , x n ) as if it was the relative representation of its ideal caption y ∗ with respect to the counterpart collection ( y 1 , . . . , y n ) , see Figure 4. The whole procedure to set up an ASIF model and use it to find the best caption for a new image follows.

ASIF recipe. Ingredients:

Two good encoders, each mapping a single data modality to a vector space. Let X and Y be the mode domains, for instance a pixel space and a text space, we need E 1 : X → R d 1 and E 2 : Y → R d 2 .

A collection of ground truth multimodal pairs: D = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } , for instance captioned images.

Procedure to find the best caption among a set of original ones ˆ Y = { ˆ y 1 , . . . , ˆ y c } for a new image x ∗ :

Compute and store the embeddings of the multimodal dataset D with the encoders E 1 , E 2 and discard D . Now in memory there should be just D E = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } ;

Compute the n -dimensional relative representation for each candidate caption ˆ y i = [ sim (ˆ y i , y 1 ) , . . . , sim (ˆ y i , y n )] , where sim is a similarity function, e.g. cosine similarity. Then for each ˆ y i set to zero all dimensions except for the highest k , and raise them to p ≥ 1 . Finally normalize and store the processed c vectors ˆ y i . Choose k and p to taste, in our experiments k = 800 and p = 8 ;

Compute the relative representation of x ∗ using the other half of the embedded multimodal dataset D E and repeat the same processing with the chosen k and p ;

We consider the relative representation of the new image x ∗ as if it was the relative representation of its ideal caption y ∗ , i.e. we define y ∗ := x ∗ . So we choose the candidate caption ˆ y i most similar to the ideal one, with i = argmax i ( sim ( y ∗ , ˆ y i )) .

To assign one of the captions to a different image x ∗∗ repeat from step 3.

Properties of ASIF models. The above construction yields several intriguing properties for free:

No training and 'data centric' . As we have seen, an ASIF model is built on top of two independently pretrained encoders and the embeddings of a multimodal dataset, and so without training or finetuning any neuron. Being deployable or updatable in seconds, an ASIF model is radically 'data centric' [12]. For example, it is trivial to adjust the model by adding or forgetting specific samples. The latter usecase is particularly important, as the right to use specific assets may change over time and removing the effect of specific samples from a trained network requires sophisticated forgetting techniques, e.g. [711]. In our procedure, the encoders should be pre-trained with established data sets that do not change over time, while removing the effect of a multimodal pair is as simple as deleting its embeddings.

Data efficiency: Being able to exploit two pretrained encoders, ASIF models require far less groundtruth multimodal pairs to become effective. As confirmed by our experiments, ASIF reaches competitive zero-shot performance on diverse classification benchmarks by using a fraction of the multimodal data of its predecessors, reaching a respectable accuracy even with thousands of pairs (we refer to Section 3 for more details). This is in line with classical work in computer vision, where prototypical networks [19] are a strong baseline in the extremely few-shot regime.

Interpretability: Sparse relative representations make the ASIF models interpretable classifiers. In fact, we can trace back every prediction to a small set of data points in the multimodal dataset -corresponding to the dimensions that are nonzero in both the image and the label relative representations- accountable for the outcome (at most k ), see Figure 6. This enables visualizations of the relevant samples contributing to a correct or incorrect prediction at no extra cost, in stark contrast with other approaches that are often costly [20] and fragile [21, 22].

Relation to k-Nearest Neighbors. Like k-NN, ASIF is a non-parametric supervised learning algorithm that requires an explicit representation of every entry in the training dataset at test time. Differently from k-NN, ASIF can perform open-ended classification, as shown e.g. in Figure 4 with two competing brand new captions. Indeed, ASIF is functionally equivalent to CLIP, and can function as a drop-in replacement in applications using CLIP-like models.

Implementation that scales. Clearly, our method pays the price of avoiding training with a larger memory footprint and increased computation at inference time, since we need to compute not only the embeddings but also the cosine similarities against the multimodal dataset. As such, our approach should not be considered a general one-stop replacement for CLIP, although in our experiments we managed to scale ASIF to 1.6M pairs while maintaining a reasonable inference speed. Our non-optimized implementation of ASIF is less than 2x slower than CLIP. On a positive note, there are two well-established techniques that could radically enhance the efficiency of ASIF and potentially enable scalability to billions of entries. The memory footprint required to store all the embeddings of the multimodal dataset can be dramatically reduced using product quantization [23], while inverse indexing [24] can be used to circumvent the need for computing the cosine similarities against the entire dataset at test time. These techniques are both implemented e.g. in the FAISS library [25]. Finally, we find that the distribution of pairs chosen during inference is rather short-tailed, presenting opportunities to massively prune the model even at deployment time , deleting from memory the data that is never used. It should be noted, however, that the assessment of the performance of large-scale optimized ASIF models is beyond the scope of this work. While it is an interesting direction for future research, the focus of our current study is on establishing the potential the ASIF method.

2.1 Design choices and implementation of ASIF models.

Curating the multimodal dataset. While neural models like CLIP or LiT are defined just by the weights of the two encoders, to univocally identify an ASIF model we also need the embeddings of the multimodal dataset. Even if two image-text ASIF models rely on the very same encoders, they comply to different visual understandings of the world if they are based on different collections of image-text pairs, therefore achieving different performances on the same tasks. Unlike conventional neural vision models, ASIF enables effective curation of the training dataset through swift iterations, given the absence of training and the smaller datasets. Furthermore, ASIF provides the means to assess the impact of each training pair, as demonstrated in Figure 6.

Salient hyperparameters. While using the raw relative representations already produces an ASIF multimodal model with non-trivial capabilities, we found that two simple treatments greatly improve performance and efficiency, and also foster interpretability.

i) Sparsification. We set to 0 all the entries of the n -dimensional relative representation except for the top k . In our experiments n and k are respectively in the order of millions and hundreds. In this way we cut off the small noisy signals from the dissimilar entries, that accumulated during comparisons would destroy the signal from the few similar entries. Furthermore we get highly interpretable representations that can be efficiently compared, since we have just k nonzero features, each one linked to a single entry in the multimodal dataset.

ii) Exponentiation. We raise all the nonzeroed similarities sim ( x ′ , x i ) to p , with p ≥ 1 . This non-

̸

linearity weighs more the contribution of the most similar entries in the relative representation. Besides the pivotal choice of the ground-truth multimodal pairs, the number of non-zero elements k and the exponent p are the salient hyperparameters to consider when deploying an ASIF model. In general, we found that picking a p = 1 may help, while choosing a k ≪ n is always crucial. For more details see Section 3.

2.2 Closely Related Works

Classics. In [26], Stanley Ulam affirms that a mathematical formalization of the word 'as'-on a par with the connectives 'and', 'or', 'implies' and 'not'-would be a key milestone to artificial intelligence. This idea of analogies being at the core of cognition is shared by Hofstadter [27], who states that a concept is a collection of analogies, in line with what the ASIF procedure prescribes.

Retrieval augmented foundation models. Recent works in NLP enhance unimodal language models with retrieval to reduce the size of the architectures and the training dataset while also making results more transparent [28, 29]. Our work is in line with this view, that the ASIF procedure extends to the multimodal case. Importantly, ASIF offers a new perspective on data centric AI [12], where data and the external memory implement the alignment between modalities. Networks with discrete Key-Value bottlenecks [30] are closely related to our approach, with the critical differences that our memory is not learned and that their decoder is trained for classification. Retrieval and memory-augmented models have also been successful in Reinforcement Learning [31], physical reasoning [32], and code generation [33]. Finally, we notice that making predictions on new samples by exploiting the similarity with a dictionary of previous data points is a common approach in computer vision [19] for few-shot learning. Our procedure is also related to compressed sensing algorithms where a signal (in our case an image) is sensed as a sparse combination of fixed atoms [34, 35] with an iterative projection procedure [36, 37] and only transmitting the coefficients to the receiver (text modality).

Learning multimodal models. Following the intuition outlined by early works on aligning text and image embeddings [39, 40], today large multimodal models are conquering the computer vision scene thanks to their wide applicability and easy transfer to new downstream tasks [1, 2, 15, 41]. We identify two key leaps respect to traditional models like ResNet [42]: (i) Free text allows to learn visual concepts beyond a finite set of predefined categories and to exploit the structure of language to compose them, as masterfully seen in Dall-E [43]. (ii) The recognition tag transitioned from being an output pulled out of the image by the neural stack (the label) to become an input that should be interpreted, and therefore processed by its own encoder (the free text caption). This corresponds to an epistemological perspective shift, as discussed by Norelli et al. [4]. Data and learning efficiency are clear challenges for large multimodal models, that often require hundreds of millions of examples. Efforts such as [2, 29] attempt to reduce this. ASIF presents a different take on this problem, showing how much can be achieved by simply remembering the training data efficiently.

3 Empirical Evidence

In the following we compare ASIF to traditional multimodal models based on contrastive training, CLIP and LiT. We then take a closer look to the classification of a single image, unpacking the relative representations and following the classification algorithm step by step. As a prelude to the above, we start by discussing the pretrained encoders and dataset forming the ASIF models we tested.

Pretrained encoders and multimodal dataset used. For our primary experiment, we utilized vision transformers as image encoders, pretrained either in a supervised manner (DEIT base, [44]) or in an unsupervised manner (DINO VITs8, [45]), on Imagenet 1k [46] and 21k [47] respectively. The embedding size was 768 for DEIT and 384 for DINO. This configuration aligns with that used in LiT [2], with the sole distinction being that, unlike LiT, we used a pre-trained, frozen text encoder. Regarding the text encoder, we employed the SentenceT transformer [48], trained on a dataset comprising more than 1 billion sentences obtained from the internet. We employed the first 1.6M entries of the Conceptual Caption dataset (CC12M, [49]) as our multimodal dataset. This dataset amasses images and filtered alt-texts collected from the internet. To optimize performance on a single Tesla T4 GPU, we limited our analysis to the initial 1.6M pairs. In our 'scaling-laws' experiments, we also utilized DEIT tiny and small vision transformers [44] and two smaller SentenceT encoders.

Zero-shot performance. We assessed the quality of our ASIF multimodal model by comparing its zero-shot classification performance against CLIP and LiT on four datasets: CIFAR100, Imagenet, Imagenetv2, and PETS [46, 50-52]; see Table 1. We crafted label prompts as in Zhai et al. [2, Table 11].

Remarkably, we achieve competitive performance with CLIP and LiT using two frozen pretrained encoders and a fraction of the image-text pairs.

The two ASIF models reported in Table 1 differ for the vision encoder, that is respectively supervisedly (DEIT) and unsupervisedly pretrained (DINO). We tuned k and p on the ImageNet validation set, in both cases we used k = 800 and p = 8 . The comparison between the two versions is not completely fair since the visual transformer architecture of DINO is smaller (e.g. the codings are 384-dimensional rather than 768) but corroborates the effectiveness of ASIF with encoders pretrained using no supervision. In the appendix we report the results of a wider collection of ASIF models based on different visual and textual backbones.

Summary: Overall, we observe that our ASIF procedure can achieve competitive zero-shot results with a fraction of the image-text pairs used in prior work (assuming access to other pre-trained, even unsupervisedly, unimodal models).

ASIF scaling laws. In Figure 5 we show the full zero-shot accuracy trend on Imagenet as the size of the multimodal dataset grows for different choices of k and p . ASIF models become effective very quickly: we reach a non-trivial 18% classification accuracy using just the first 10,000 pairs in the multimodal dataset. We recall that building an ASIF model from scratch still requires a lot of unimodal data to pretrain the encoders, but now this data may come untied and even without any label.

We tested ASIF further with smaller image and text encoders on multimodal datasets of different sizes ( 10 2 to 10 6 image-text pairs from CC12M) to provide early evidence about ASIF scaling laws. We used DEIT tiny, small, and base vision trans-

non-zeros:50,val_exp:1

0.5

non-zeros:50,val_exp:8

non-zeros:800,val_exp:1

non-zeros:800,val_exp:8

0.4

non-zeros:1600,val_exp:1

non-zeros:1600,val_exp:8

0.3

0.2

0.1

0.0

102

103

104

105

106

Sizeof themultimodal dataset

formers [44], along with two smaller SentenceT encoders. As we see in Figure 5 and in the appendix, Imagenet classification accuracy grows smoothly with the size of the multimodal dataset, and performance does not saturate earlier with smaller encoders. These results are promising but still preliminary, further experiments with larger multimodal datasets are left for future work.

Summary: Our experiments show a steady improvement in ASIF's performance as the size of the multimodal dataset increases. Performance deteriorates with smaller encoders, but even then there is no sign of saturation or plateau.

Large stone triumphal

Queryimage

0.85

The arch at Valley Forge

0.85

0.85

0.80

arch with trees on the

0.80

0.95

Most similarimages in

themultimodal dataset

other side in Dougga

*+

Neoclassical Architecture

Ruins of the Roman

0.84

Painting - A View through

0.84

0.81

0

0.78

triumphal arch at Palmyra

0.78

0.96

0

triumphal arch

Three Arches of the ...

as photographed in 2006.

photoofa

The Arch of Titus in

A watercolor sketch or

anbso

0.84

landmark.

Rome,Italy.Rome

0.84

0.89

0

0.77

illustration of the

Brandenburg gate

0.77

0

0

Corresponding

caption

A

0.84

Severus,Rome

The Arch of Septimus

0.84

0.84

0

0.76

Model of the Arch of

Constantine Probably by

0.76

0.84

0

<PERSON>

0.97

The Arch of Titus, Rome

0.97

0.88

0

0.82

The Arch of Constantine

and the <PERSON> in

0.82

0.87

0

0.74

The Arch of <PERSON>

0.74

0.83

0

Rome.

and the aqueduct

AC Milan's players

<PERSON> and warrior

A picture of one of the

0.91

celebrate on a bus after

winning the championship

0.91

0

0

0.81

on a chariot. <PERSON>

bronze statue atop ...

0.81

0

0

0.72

most famous German

landmarks:...

0.72

0

0

Arch of Constantine near

Damage to the Umayyad

0.91

Photo looking up at the

0.91

0.83

0

0.81

the Colosseum in Rome,

0.81

0.87

0

0.69

Mosque in Damascus,

0.69

0

0.87

Arch of Titus

Italy stock photography

Syria

0.88

The faded triumph of

<PERSON>'s Arch in

0.88

0.93

0

0.81

The Arch of Trajan: all

what you need to know

0.81

0.86

0

0.68

Black and white shot of

architectural columns in

0.68

0

0

Benevento

the park

0.87

The triumphal arch of

0.87

0.95

0

0.80

The Arch of Triumph or

Arch of <PERSON>,

0.80

0.95

0.66

0.65

Valley of the Kings

Temple of Hatshepsut,

0.65

0

0

Palmyra, Syria, 2005

Adjusting an ASIF model in seconds. Consider classifying EuroSAT [53] satellite images using ASIF. Initially, the zero-shot performance outperforms random chance but is not impressive ( 29 . 4% unsup. configuration, 10 classes). CLIP, while better, also falls short with a 54 . 1% performance rate.

Now, imagine we acquire 100 new image-text pairs from EuroSAT. Our goal is to develop a new multimodal model based on an updated training set, anticipating the need to process more satellite images in the future. With CLIP, this would require us to fine-tune or retrain the model from scratch using the updated dataset. In contrast, ASIF requires only to obtain and store the new pairs' embeddings. The model retains its usability for all the previous images, but it now also makes accurate predictions for satellite images, achieving an 82 . 2% ± 2 . 0 accuracy on EuroSAT. 2

Similarly, if we need to remove samples from the multimodal training dataset-either because they are faulty (as shown in Figure 6) or because we have lost the license to use them-the process is as simple as deleting the corresponding embeddings to get a new model.

Summary: ASIF enables quick model adjustments and data handling without the need for retraining, unlike traditional models such as CLIP. This demonstrates its practicality in real-world scenarios, such as the emergence of a new setting or the loss of rights for assets used during training.

Deep dive into a classification. To better understand why and how the ASIF procedure works, we are going to follow step by step the single classification depicted in Figure 6, showing the entries of the multimodal dataset used to build the relative representations. For simplicity we assume k = 23 .

We want to find the Imagenet label of the upper left image in Figure 6. The first step is to compute its relative representation with respect to the multimodal dataset, this procedure selects the 23 most similar images in our collection. The most similar are indeed triumphal archs that-as similarity decreases-turn to generic structures with archs or ancient monuments. No other image in the multimodal dataset is involved in the classification, we could replace all the entries in our collection except for these 23, and as long as no new image is more similar to the input image, every computation in the classification process would remain the same. This consistency of the ASIF procedure is in contrast to traditional approaches like CLIP and LiT, where the parameters defining the encoder computations are shaped by the whole multimodal dataset through contrastive learning.

Now, we approach the pivotal step of the ASIF procedure: jumping into the text space by treating the relative representation of the input image as if it were the relative representation of its ideal caption. The vector remains the same, but its meaning changed: now it signifies how much the ideal caption should be similar to 23 captions.

The efficacy of this jump hinges entirely on the quality of the multimodal dataset. Looking at the 23 captions in Figure 6, we are confident that the most fitting candidate caption corresponds to one of the prompts associated with the triumphal_arch label in ImageNet, such as 'a photo of a triumphal arch' . Conversely, a dataset featuring 23 non-informative captions, such as file names 'IMG_20180823.jpg' or camera settings 'D90 18.0-70.0 mm f/3.5-4.5' , would not allow the model to recognize the correct class even with the best image and text encoders. This limitation is actually a defining feature of the ASIF procedure, as the meaning attributed to the input is ultimately determined by the multimodal dataset and not by the encoders: by acting just on the image-text pairs, we have full control over the model's output.

Summary: Our simple (non-cherry picked) example showcases how the ASIF predictions can be easily attributed to specific examples in the multimodal training data by construction. This feedback can be used to explain predictions and grow high quality datasets in data centric AI, for example by inspecting which examples contribute to incorrect classifications.

4 Discussion

The effectiveness of the ASIF procedure raises questions on the role of memory and retrieval in machine learning, while at the same time opens new opportunities for products based on multimodal models, opening many avenues for future works. In the following we will discuss these aspects.

Perception and interpretation disentangled. In ASIF there is no trace of the multimodal data in the weights of the encoders, which are pretrained on different unimodal datasets. Nonetheless, the relative representations and the outcome of any classification task fundamentally depend on the multimodal dataset. This state of affairs reflects the factorization of perception and interpretation in the two stages constituting an ASIF model; the encoding and the construction of the relative representations. Such factorization is desirable because it relieves the black-box neural encoders from the responsibility of attributing meaning to their inputs, as envisaged by Eco [54, Par. 3.3.1.3]. Therefore, we can consider the neural encoders as mere sensors and focus only on the second stage to analyze, explain, and act on the interpretations of an ASIF model, as shown in Figure 6.

Learning or retrieval? As we have seen, the ASIF procedure requires no training: it does not distill the multimodal data into any learnable parameter. Rather, it prescribes a rigid memorization of the embeddings of the multimodal dataset, where each entry has its fixed-size spot, similarly to a retrieval process. On the other hand it seems impossible to not describe ASIF as a learning algorithm; for instance it satisfies the fundamental characterization given by Mitchell [55]: the more the multimodal data the more ASIF improves, as we can clearly see in Figure 5. Ultimately, an ASIF model is functionally comparable to CLIP. ASIF blurs the border between learning and retrieval by questioning the effectiveness of storing information only in the weights, and rather advocates to combine learning representations with external memories. We encourage more work on memory augmented neural networks and towards understanding the implications of memory for generalization.

Generalization to new distributions. The empirical performance of ASIF calls for a discussion on zero-shot and out-of-distribution generalization in foundation models trained with huge data sets. Clearly, the performance of ASIF will depend strongly on the multimodal data used for alignment. As an example, the good performance on Imagenet may not be particularly surprising in light of the qualitative evaluation seen in Figure 6. There, our query image might as well had been part of our multimodal data set, as the semantic gap with its neighbours appears to be small. Despite this, our choice of downstream evaluation and pre-training data set is identical compared to prior work [2]. As such, while it appears clear that ASIF should fail when the semantic gap between downstream task and 'training data' is large, it is unclear why it should be different for more standard models like CLIP [1] and LiT [2]: if a gap does exist, future work should work to address it. In the meanwhile, we recommend that future foundation models are benchmarked on significantly broader sets of downstream tasks, ideally with some analysis of the semantic distance between test and training data (if any). Alternatively, our results may be interpreted in light of the strong performance of unimodal models. There may be a limited benefit of training from scratch on less curated multimodal data sets compared to standard and well established unimodal data sets, although we posit that at even larger scales the difference may be more significant.

Limitations. The simple ASIF procedure presented here offers a strong baseline for multimodal models, but its performance still falls apart to CLIP and LiT when the mutimodal dataset is abundant

and the cost of training is not a concern. Additionally, the large dimensionality of the relative representations, even if sparse, poses challenges for directly applying ASIF to tasks like text-to-image generation. We recognize that the experiments reported in this paper do not provide a comprehensive examination of the myriad of downstream tasks multimodal models are known to adeptly handle, it is important to note that such broad coverage was explicitly out of scope for this work. Our primary objective here was to introduce and justify the ASIF procedure, illustrating its effectiveness on the representative task of zero-shot classification. In making this choice we followed [2], that used the very same datasets to showcase the benefits of a locked image encoder, that is their main claim. We anticipate and welcome a more extensive evaluation of ASIF in the context of a wider range of tasks in future research endeavors.

Conclusions. We presented a simple procedure called ASIF to assemble a fully functional multimodal model like CLIP from two unimodal pretrained encoders and a collection of image-text pairs without tuning a single neuron. While achieving competitive zero-shot classification results with its predecessors using a fraction of the data, the proposed ASIF procedure enriches multimodal models with editability-a new model based on different pairs can be deployed in seconds-and interpretable representations. The effectiveness of ASIF models also clashes with the dominant view of a learning algorithm as a way to distill data into the parameters of a model, and raises questions on the role of memory and retrieval in machine learning.

Acknowledgments

AN, MF, and FL partially worked on ASIF when they were at Amazon Web Services in Tübingen, Germany.

This paper is financially supported by the PRIN 2020 project no.2020TA3K9N (LEGO.AI), PNRR MURproject PE0000013-FAIR, and ERC Grant no.802554 (SPECGEO).

References

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18123-18133, 2022.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

Antonio Norelli, Giorgio Mariani, Luca Moschella, Andrea Santilli, Giambattista Parascandolo, Simone Melzi, and Emanuele Rodolà. Explanatory learning: Beyond empiricism in neural networks. arXiv preprint arXiv:2201.10222 , 2022.

Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation , 1(4):541-551, 1989.

Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision , pages 843-852, 2017.

Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 792-801, 2021.

Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In European Conference on Computer Vision , pages 383-398. Springer, 2020.

Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9304-9312, 2020.

Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems , 32, 2019.

Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In International Conference on Machine Learning , pages 3832-3842. PMLR, 2020.

Andrew Ng. Unbiggen ai. IEEE Spectrum , 2022. URL https://spectrum.ieee.org/ andrew-ng-data-centric-ai .

Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. 11th International Conference on Learning Representations, ICLR 2023 , 2022.

Gary Wang, Kyle Kastner, Ankur Bapna, Zhehuai Chen, Andrew Rosenberg, Bhuvana Ramabhadran, and Yu Zhang. Understanding shared speech-text representations. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5. IEEE, 2023.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, YunHsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021.

Thomas Hofmann, Bernhard Schölkopf, and Alexander J Smola. Kernel methods in machine learning. The annals of statistics , 36(3):1171-1220, 2008.

Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications , 9(1): 141-142, 1964.

Geoffrey S Watson. Smooth regression analysis. Sankhy¯ a: The Indian Journal of Statistics, Series A , pages 359-372, 1964.

Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 4077-4087, 2017.

Alessandro Achille, Aditya Golatkar, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Lqf: Linear quadratic fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15729-15739, June 2021.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning , pages 1885-1894. PMLR, 2017.

Samyadeep Basu, Phil Pope, and Soheil Feizi. Influence functions in deep learning are fragile. In International Conference on Learning Representations , 2021.

Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence , 33(1):117-128, 2010.

Sivic and Zisserman. Video google: a text retrieval approach to object matching in videos. In Proceedings Ninth IEEE International Conference on Computer Vision , pages 1470-1477 vol.2, 2003. doi: 10.1109/ICCV.2003.1238663.

Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data , 7(3):535-547, 2019.

Gian-Carlo Rota. The barrier of meaning. Letters in Mathematical Physics , 10(2):97-99, 1985.

Douglas R Hofstadter. Analogy as the core of cognition. The analogical mind: Perspectives from cognitive science , pages 499-538, 2001.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 2206-2240. PMLR, 2022.

Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv: Arxiv-2208.03299 , 2022.

Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf. Discrete key-value bottleneck. arXiv preprint arXiv:2207.11240 , 2022.

Anirudh Goyal, Abram Friesen, Andrea Banino, Theophane Weber, Nan Rosemary Ke, Adria Puigdomenech Badia, Arthur Guez, Mehdi Mirza, Peter C Humphreys, Ksenia Konyushova, et al. Retrieval-augmented reinforcement learning. In International Conference on Machine Learning , pages 7740-7765. PMLR, 2022.

Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael C Mozer, and Yoshua Bengio. Neural production systems. Advances in Neural Information Processing Systems , 34:25673-25687, 2021.

Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. Retrieval-augmented generation for code summarization via hybrid gnn. In International Conference on Learning Representations , 2021.

Jian Wang, Seokbeop Kwon, and Byonghyo Shim. Generalized orthogonal matching pursuit. IEEE Transactions on signal processing , 60(12):6202-6216, 2012.

Stéphane G Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. IEEE Transactions on signal processing , 41(12):3397-3415, 1993.

Francesco Locatello, Rajiv Khanna, Michael Tschannen, and Martin Jaggi. A unified optimization view on generalized matching pursuit and frank-wolfe. In Artificial Intelligence and Statistics , pages 860-868. PMLR, 2017.

Francesco Locatello, Anant Raj, Sai Praneeth Karimireddy, Gunnar Rätsch, Bernhard Schölkopf, Sebastian Stich, and Martin Jaggi. On matching pursuit and coordinate descent. In International Conference on Machine Learning , pages 3198-3207. PMLR, 2018.

Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64-73, 2016.

Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. Advances in neural information processing systems , 26, 2013.

Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3128-3137, 2015.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv: Arxiv-2204.06125 , 2022.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357. PMLR, 2021.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV) , 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik. Imagenet-21k pretraining for the masses. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual , 2021.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bertnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11 2019.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR , 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, ON, Canada, 2009.

B. Recht, R. Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? International Conference On Machine Learning , 2019.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition , pages 3498-3505, 2012. doi: 10.1109/CVPR.2012.6248092.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7):2217-2226, 2019.

Umberto Eco. Kant and the platypus: Essays on language and cognition . HMH, 2000.

Tom Mitchell. Machine learning , volume 1, 9. McGraw-hill New York, 1997. ISBN 0070428077.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021.

Appendix

In this appendix, we further showcase the interpretability of ASIF models when used for classification in Figure 7. Then we provide additional details for the scaling laws and EuroSAT experiments presented in the main paper, and report additional results about the impact of the size of the encoders (Table 2), and of the image training dataset. We also report further evidence that the ASIF construction is not overly sensitive to its hyperparameters. Lastly, we discuss more in detail the idea that captions of similar images are alike in Figure 10.

Trainingsamplesresponsiblefortheprediction

Competing captions

a satellite image of a suburban area,a cul de sac,oraresidential area with houses

0.3

a satelliteimageof anurban area,anindustrial area,anindustrialzone,a city,orfactories

0.25

a satellite image of a permanent crop, arable land, or an orchard

Text similarity

0.2

a satellite image of a highway,motorways,highways,ora street

Queryimage

0.15

a satelliteimageofameadow,herbaceousvegetation,grass,orfields

0.1

a satellite image of a pasture, farmland, or farms

0.05

asatelliteimageofacanal,ariver,awaterway,orastream

a satellite image of a forest,woodland, or trees

asatelliteimageofanocean,awater,asea,rareservoir

0.1

Image similarity

0.1

0.2

0.3

a satellite image of farmland,farms, or an annual crop

B

C

The ocean

Cognitive

waters aremade

of oxygen&

Equipment

hydrogen

Advisor

Photo of ocean and land from the air

Aerial view of the red sea

The ocean waters are made of oxygen & bydrogen

background stock photography

Many jelly fish swimming in the ocean and glowing colors

Trainingsamplesresponsiblefortheprediction

Competing captions

a satelliteimage ofameadow,herbaceousvegetation,rass,orfields

0.2

A

a satelliteimageof anurban area,anindustrialarea,anindustrialzone,a city,orfactories

a satellite image of farmland, farms,or an annual crop

Text similarity

0.15

a satelliteimageofapasture,farmland,orfarms

Query image

a satelliteimage ofacanal,ariver,awaterway,ora stream

0.1

asatelliteimageofahighway,motorways,highways,orastreet

0.05

DC

asatelliteimageofasuburban area,aculdesac,oraresidentialarea

a satellite mage of a permanent crop, arable land, or an orchard

asatelliteimageofaforest,woodland,ortrees

0.01

0.02

0°0

0.04

0.05

asatelliteimageofanocean,awater,asea,orareservoir

Image similarity

B

C

D

E

Cotryakwildst

wit forestrizntrysyseeroylty free

round paiingTeturedbckroudfclodsan sky

dw

The red area o this Google Maps screesho shows 3057 N.

RockwelSherdevlop

complex

A Additional details on the scaling laws experiment

Models used in the scaling laws experiments. As discussed in the main paper, we tested ASIF with smaller image and text encoders to provide early evidence about ASIF scaling laws. We used three different instances of DEIT [44] vision transformers, the tiny (5.6M parameters, 192-dimensional embeddings), small (22M, 384), and base (87M, 768), and the original VITb16 vision transformer [56] (86M, 768). The DEIT models were pre-trained on a smaller dataset, the standard Imagenet1k training set [46], while VITb16 was pretrained on Imagenet21k [47]. As text encoders, we used smaller versions of SentenceT [48], with 23M and 33M parameters (both 384dimensional embeddings), in contrast to the 110M parameters of the main model (768).

Figure 8 shows that, with smaller encoders producing smaller embedddings, we do not observe a performance saturation within 1.6M image-text couples. Further experiments with larger datasets are left for future work.

Impact of image pre-training data. In Table 2 we report the complete results of ASIF models using DEIT encoders [44]. We observe the expected positive correlation between the size of the encoders and the classification accuracy. Interestingly, ASIF with the largest instance of DEIT beats the one based on the standard VIT pretrained on Imagenet21k on three out of four of test datasets, while losing more than 10 points on CIFAR. These results may be interpreted in light of the similarity of the datasets we are using, with features useful to classify CIFAR images less overlapping with Imagenet1k features with respect to the other datasets.

ASIF Imagenetlk 0-shot accuracy with different backbones (train dataset: CC12M)

0.6

0.5

0.4

0.3

0.2

0.1

encoders:DEIT tiny (8M)

-SentenceT MiniLM-6(23M)

encoders: DEIT small (22M) - SentenceT MiniLM-12 (33M)

encoders: DEIT base (87M) - SentenceT mpnet base (110M)

103

104

10°

100

Number of anchors

Impact of the Image encoder (train dataset: CC12M)

0.6

0.5

Imagenet1k 0-shot accuracy

0.4

0.3

0.2

0.1

encoders:DEIT tiny (8M)- SentenceT mpnet base (110M)

encoders: DEIT small (22M) - SentenceT mpnet base (110M)

encoders: DEIT base (87M) - SentenceT mpnet base (110M)

103

104

10°

100

Number of anchors

Impact of the Text encoder (train dataset: CC12M)

0.6

0.5

0.4

0.3

0.2

0.1

encoders: DEIT base (87M) - SentenceT MiniLM-6 (23M)

encoders: DEIT base (87M) - SentenceT MiniLM-12 (33M)

encoders: DEIT base (87M) - SentenceT mpnet base (110M)

103

104

10°

100

Number of anchors

B Additional details on the EuroSAT experiment.

EuroSAT, a renowned benchmark for satellite image classification, serves as a testing ground for out-of-distribution generalization in zero-shot and few-shot scenarios [53]. The dataset contains 27,000 images labeled under ten categories. Our ASIF model with a DINO visual backbone (denoted as 'ASIF unsup' in table 1) achieved a zero-shot classification score of 29 . 4% . While significantly better than random chance, this modest performance is not surprising considering the scarce presence of satellite images in the CC12M dataset.

As a further experiment, we randomly selected 100 images from the EuroSAT dataset and incorporated them into our ASIF training set, raising the total to 1,500,100 image-text pairs and leaving 26,900 images for testing. We created captions for the EuroSAT images using the template ' a satellite image of [CLASS NAME]'. This way the ASIF model improves dramatically, reaching a classification accuracy of 82 . 5 ± 2 . 8% on EuroSAT (average ± standard deviation of 5 trials).

Contrarily, CLIP [1], while demonstrating better zero-shot accuracy at 54 . 1% , is trained on a private dataset comprising 400 million images. This dataset may contain a larger number of satellite images than our 1.6 million subset of CC12M. Given the substantial improvement observed when we added just 100 EuroSAT images, it's reasonable to speculate that CLIP's enhanced performance might stem from its larger database of satellite images. However, confirming this theory is impossible due to the private nature of CLIP's training set.

We can, nevertheless, examine the presence of satellite images in the CC12M dataset. Using ASIF models' unique interpretability property, we can trace the training samples behind each classification. Figure 7 displays two EuroSAT samples, one classified correctly and the other not, along with the corresponding CC12M pairs responsible for the classifications. We note that our subset of CC12M is lacking in satellite images, and the few available often have misleading captions, such as a map of a drainage network tagged as "a satellite image of a canal, a river, a waterway, or a stream" instead of an urban area.

The images shown are an adaptation of the interactive plot to analyze any ASIF image classification we provided in the code demo attached in the supplementary material.

C ASIF sensibility to its hyperparameters

Finally, we present evidence about the sensitivity of the ASIF model to the hyperparameters p and k . Specifically, we show the hyperparameter search for PETS and CIFAR100 in Figure 9. Table C with results on the parameters fine-tuned on the two datasets reveals marginal improvements over the standard choice of k=800 and p=8. This suggests that the ASIF model is relatively insensitive to the choice of these hyperparameters.

8

0.70

8

0.64

7

0.65

7

0.62

6

0.60

6

0.60

dxa

5

dxa

5

0.58

0.55

4

Val

4

0.56

0.50

3

3

0.54

0.45

2

2

0.52

0.40

1

0.50

0

250

500

750

1000

1250

1500

0

250

500

750

1000

1250

1500

Anatomy of a

Kiwi Cucumber Salad

Freshly picked

DateIdeas-Glow

spinning wheel.

flowers are poured

in the Dark Path

Saxonyfor

Fresh Mint shown in

into acopper

Materials

illustration.

a shallow bowl with

still.

Spinning Wool, Hand

salad servers.

Spinning,Spinning

Wheels, Spin

Image similarities

Image similarities

Image similarities

Image similarities

104

104

104

10

102

103

10.09

0.49

103

102

10.04

10.24

103

102

10.

10.44

103

10²

10.27

10

10

10

10²

100

100

100

0.2

0.0

Text similarities

0.2

0.4

0.6

0.8

10

0.2

0.0

0.2

Text similarities

i0.4

0.6

0.8

1.0

104

0.2

0.0

0.2

Text similarities

0.4

0.6

0.8

1.0

104

0.2

0.0

0.2

Text similarities

0.4

0.6

0.8

1.0

103

10.09

10

10.08

!0.14

103

10.10

!0.43

10²

600

0.21

102

102

102

102

10

10

101

10²

100

100

100

100

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.0

0.2

0.4

90

0.8

1.0

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Worldglobe

Blitz theComic Guy

LetterKthat look

characterlooking

looks at a big

Class Comics,

like bird logo

througha

daisy in springtime

Bristol Board,

RA

vector

telescope.Isolated

royaltyfree

Desert Island,

onwhitebackground

illustration

Modern Retro, Comic

LOREM IPSUM

stock illustration

Artist, Blitz,

Image similarities

Image similarities

Image similarities

Image similarities

10

104

10

103

103

10.21

103

0.15

10²

!0.14

j0.47

0.76

0.47

0.49

10²

102

102

10²

10

10

101

10

100

100

10°

10°

0.2

0.0

0.2

Text similarities

0.4

0.8

1.0

0.2

0.0

Text similarities

0.2

0.4

0.8

1.0

0.2

0.0

0.2

Text similarities

0.4

0.8

1.0

0.2

0.0

0.2

Text similarities

0.8

1.0

10²

103

103

10

10.11

10.12

10.12

0.17

!!0.13

0.23

10.30

10²

102

102

102

10

102

101

10²

100

100

100

100

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.0

0.2

0.4

0.6

0.8

10

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a method to obtain multimodal representations from pretrained unimodal models and a multimodal dataset, and demonstrates the effectiveness of the method on the zero-shot classification task. The method is to map candidate images and texts into the same representation space, which consists of similarities of the candidates to each image/text in an anchor multimodal dataset.\n\nSTRENGTHS:\n- The proposed method is very simple and intuitive, by representing an image or text as a vector of similarities to an anchor set of images/texts.\n- No gradient updates are required. One can directly benefit from pretrained unimodal models. No training parameters are involved.\n- The advantage of quick model adjustments via data handling without model retraining.\n- The method gives (to a certain extent) explainable multimodal representations.\n- The paper is clearly written and easy to understand, including a comprehensive discussion section.\n\nWEAKNESSES:\n- A larger multimodal dataset leads to better zero-shot classification performance, but also leads to a bigger representation vector. E.g. a 1.6M-d vector for an image or text, while manageable for retrieval tasks, might be difficult to maneuver for other downstream tasks (e.g. any generative tasks). I understand that the authors stated that effectiveness or applications to tasks other than zero-shot classification is out of the scope of this work, but I'd like to see more discussions on the limitations or impacts of the size of representation vectors.\n- The method is sensitive to the curated multimodal dataset. The authors ablated on the size of the dataset, but not the source of the dataset. E.g.what would be the effect of using a different multimodal dataset from CC? (COCO, LAION, etc?)\n\nQUESTIONS:\nAs stated in the section above.\n\nLIMITATIONS:\nI don't see potential negative societal impacts of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to align vision and language without learning a parametric model. Main idea: having a support set of image-text pairs, the structure of the visual data should match the structure of the language data. More precisely, the distances from one query image to all images in the support set (denoted as relative representations) should be similar to the distances from the associated text and all texts in the support set. For visual tasks that can be formalized as text retrieval, this method can be applied without learning. The method has relatively good performance given its simplicity.\n\nSTRENGTHS:\nStrong points:\n S1. The method is simple and can be easily applied to potentially many vision-language retrieval tasks. Being able to use any independently pre-trained vision and language model is a big advantage. Another big advantage is that it does not require any training.\n\nS2. The model can be easily adapted by simply changing the support set of image-text pairs.\n\nS3. Some interpretability is obtained by inspecting the nearest images and captions used to obtain the relative representations.\n\nWEAKNESSES:\nWeak Points:\n\nW1. As the authors notice, the model is slower at training time. Although optimizations might reduce it.\n\nW2. It is not clear if this approach can be used for other downstream tasks, like VQA, or captioning. Although this might be outside of the scope of the paper it might be worth discussing.\n\nW3. The paper will benefit from more comparisons with recent vision-language models. Relevant experiments will be on image-to-text retrieval and text-to-image retrieval on COCO and Flickr30K datasets compared to methods like BLIP [A]\n\nW4. Comparing with some baselines will be beneficial. If we have a support set of the classification task, with ground truth image-label pairs a good baseline is a simple K-nearest neighbor on the same visual representation used in ASIF. For example, using the 100 image-text pairs of EuroSAT we can create a KNN relying on the same image encoder as ASIF. How does this KNN compare with ASIF in terms of performance?\n\n[A] Li, Junnan, et al. \"Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.\" ICML, 2022.\n\nQUESTIONS:\nQ1: Can the relative representations be used for other downstream tasks, different than image classification or retrieval?\n\nLIMITATIONS:\nThe authors have discussed the limitations regarding inference speed. \n\nThere is no specific negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ASIF, transferring independently pre-trained image/text encoders to the classification task without further finetuning.  The proposed method only needs a small amount of paired image-text data as anchors, and represent new data samples using the relative representation to the anchor samples. The simple method achieves reasonable results on different image classification benchmarks.\n\nSTRENGTHS:\n1. The proposed approach is efficient since it does not large amount of image and text data pairs to train extra multimodal representations that align images and text. Also, the required image-text pairs can be only 1.6M, which is small comparing with original CLIP and other multimodal classification models. \n2. The needed image-text pairs are flexible. By adding more anchor samples for specific image classes, the proposed method could obtain better performances for such specific classes.\n\nWEAKNESSES:\n1. Even though ASIF does not need large amount of paired image-text data, it still requires pre-trained single-modal models which are already trained on large amounts of image or text data samples. From this perspective, I didn't see the necessity of such setting. \n2. The prediction results are relying on anchor samples. If the anchor samples are not representative enough, the method will not be that useful. Imagine for a target dataset, if the provided anchor samples do not come from the same dataset or do not include similar samples as the target samples, the method may fail. If the authors can provide more experimental results to justify the influence of different chosen anchor samples, it will be more promising. \n3. The paper should be re-organized and the writing should be improved a lot. For example, the \"ASIF recipe\" can be written in an algorithm format, instead of current style.\n\nQUESTIONS:\n1. In Figure 4(b), why the image representation for x* is \"absolute representations\"? This term is never mentioned in the text. \n2. In Figure 4(a), the first sample x1 goes through the image encoder and get the embedding for x1. However, for x2, why such process involves y2 (the black letter y2 under the box for x2)?\nIn general, all the figures in this work should be re-designed since current presentation is very confusing and difficult to follow. Also, the resolution of figures should be improved.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presents a novel approach to aligning text-image modalities without any training. The method is based on the assumption that images and their captions have similar relative embeddings, even when trained independently. By leveraging paired multimodal data, relative representations can be computed within each modality, and they can serve as a medium for cross-modality communication. The major strength of this approach is its novelty, as it removes the need for training to align different modalities of data.

STRENGTHS:
The idea is novel and has the potential to introduce a new paradigm in multimodal alignment research. Adapting to a new distribution of data is easy as it only requires a new set of coupled data without training. The concept of separating perception and interpretation can open up new research opportunities and can be applied to various multimodal problems.

WEAKNESSES:
- Although the approach is interesting, a more comprehensive analysis is needed to convince the readers, including a global analysis of similarities between features from text and image encoders, potentially compared with CLIP and LiT.
- Figure 2 requires additional explanation or a clearer illustration to motivate and convince the reader.
- Figure 4 could be improved for better understanding. The diagram is not intuitive at first, without fully reading and understanding the method section.

QUESTIONS:
- How can you provide more convincing evidence that the features are communicable across different modalities?
- How sensitive is the selection of text encoders?

LIMITATIONS:
The method has not been verified across a wide range of multimodal downstream tasks, but it is clearly stated in the limitation section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to leverage single-modal pre-trained text & image encoders and a relatively small image-text dataset to create a CLIP-like open-vocabulary visual recognition model without training. The authors claimed that the proposed model ASIF is more training efficient, more interpretable, and can easier handle data copyright issues than CLIP and alternatives such as LiT. More specifically, relative representations w.r.t. the available image-text pairs are computed for image and text respectively. To remove noise, only top-k values are kept. The resulting sparse embeddings can be used for image classification tasks in a way similar to CLIP. The above process requires no training, and the anchoring image-text pairs can be removed without re-training. Moreover, by inspecting the top-K anchoring image-text pairs, human can understand how ASIF makes its predictions.\n\nSTRENGTHS:\nS1. Using relative representation to create a CLIP-like open vocabulary visual recognition model is new.\n\nWEAKNESSES:\nW1. Empirical results are weak.\n\nW2. The major claims (efficient training, data copyright, interpretability) have some flaws.\n\nW3. Scalability / generalization (to more tasks) remains unclear. \n\nThe only empirical results this paper presents are image classifications in 4 benchmarks, and the accuracy are far below widely used visual recognition models. I understand given that no training is required, this can still be considered strong results. However, this also means that this method is unlikely to be widely adopted. Moreover, for ASIF with supervised visual encoder (DEIT base), the ImageNet accuracy is also considerably lower than the original DEIT results. This raises a concern that the assumption of a available supervised encoder might not be realistic. Given the 1.6M anchor data, I also suggest the authors to run a baseline of taking existing single-modal encoders (unsupervised) and finetune on these image-text pairs with contrastive loss, so that readers can better understand the trade-offs of ASIF and training.\n\nI also have concerns on the claims. The uni-modal encoder also requires training, and it seems that the text encoder used was trained on 1B internet text, which is much larger than CLIP's 400M. Moreover, if the pre-trained encoders have data copyright issues, there's no way to remove them under the current ASIF framework without re-training. For the interpretability claim, the same procedure could be done for CLIP-like models given an anchoring dataset. (Retrieve nearest neighbors using CLIP's text/image embeddings in the image-text pairs.) Therefore, the interpretability is not ASIF's exclusive advantage.\n\nI acknowledge that the scalability limitation has been properly addressed by the authors. Although some possible techniques to improve it are discussed (L154-L169), I personally don't think this can replace a well-tuned CLIP model. For many real-world applications, inference cost may be more crucial than training cost. On the other hand, the current ASIF is only tested on image classification. I would suggest the authors explore different tasks for future version. Instead of arguing the potential by pure hypothesis, it would probably be more convincing to demonstrate the potential on a wide range of tasks / modality, given that current image classification accuracy are not promising.\n\nAdditional Suggestions:\n- To make ASIF look more promising, I'd suggest the authors try to utilize relative representations to improve CLIP. In this way, users can choose between finetuning / ASIF depending on the training / inference cost trade-off while getting strong accuracy.\n- From the accuracy difference of sup vs unsup encoders, it seems the data quality of pretrained encoders also plays a major role. Studying training data quality vs anchoring data quality might also be an interesting direction.\n\n\n---------------------------------\nUpdate after rebuttal\n================\n\nI appreciate the detailed answers from the authors. Most of my questions are answered and I have a better understanding of this submission; hence the increased confidence from 4 to 5. W2 was properly addressed and the authors promised to clarify the claims in revision. Unfortunately, W1 and W3 remains concerns (no additional results are provided) that outweigh the strengths. I've increased overall rating from 3 to 4 but still leaning negative.\n\nQUESTIONS:\nMy questions and suggestions are included in the weaknesses section.\n\nLIMITATIONS:\nLimitations are properly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}