ASIF: Coupled Data Turns Unimodal Models to Multimodal without Training
Authors: Antonio Norelli
Marco Fumero
Valentino Maiorca
Luca Moschella
Emanuele Rodolà
Francesco Locatello
Venue: NeurIPS 2023 poster (2023)
Keywords: Representation learning
Multimodal models
Analogy
Sparsity
Nonparametric
Relative representations
Language
Semiotics
TL;DR: The meaning was already there: connecting text and images without training a neural network to do so.

Abstract: CLIP proved that aligning visual and language spaces is key to solving many vision tasks without explicit training, but required to train image and text encoders from scratch on a huge dataset. LiT improved this by only training the text encoder and using a pre-trained vision network. In this paper, we show that a common space can be created without any training at all, using single-domain encoders (trained with or without supervision) and a much smaller amount of image-text pairs. Furthermore, our model has unique properties. Most notably, deploying a new version with updated training samples can be done in a matter of seconds. Additionally, the representations in the common space are easily interpretable as every dimension corresponds to the similarity of the input to a unique entry in the multimodal dataset. Experiments on standard zero-shot visual benchmarks demonstrate the typical transfer ability of image-text models. Overall, our method represents a simple yet surprisingly strong baseline for foundation multi-modal models, raising important questions on their data efficiency and on the role of retrieval in machine learning.

ASIF: Coupled Data Turns Unimodal Models to Multimodal without Training
Antonio Norelli
Marco Fumero" Valentino Maiorca"
Luca Moschella"
Emanuele Rodolà"
Francesco Locatello
*Sapienza Università di Roma, dipartimento di Informatica
Institute of Science and Technology Austria (ISTA)
Abstract
CLIP [1] proved that aligning visual and language spaces is key to solving many vision tasks without explicit training, but required train image and text encoders from scratch on huge dataset. LiT [2] improved this by only training the text encoder and using a pre-trained vision network. In this paper, we show that a common space can be created without any training at all, using single-domain encoders (trained with or without supervision) and much smaller amount of image- text pairs. Furthermore, a new version with updated training samples can be done in a matter of seconds. Additionally, the representations in the common space casily interpretable as every dimension corresponds to the similarity of the input to a unique image- text pair in the multimodal dataset. Experiments benchmarks demonstrate the typical transfer ability image-text models. Overall, method represents simple yet surprisingly strong baseline for foundation multimodal models, raising important questions on their data efficiency and on the role of retrieval in machine learning.
model has unique properties. Most notably, deploying
standard zero-shot visual
Introduction
CLIP
LiT
ASIF
Trained
Large multimodal models such CLIP [1 are rapidly becoming the stan- dard for foundation models 3 in com- puter vision. This is largely due to their zero-shot and open-world capabil- ities that enable diverse suites of down- stream tasks, from classification to de- tection and visual search.
from scratch
Image
Image
Image
Frozen pretrained
Figure 1: ASIF simple recipe to align the representations of two frozen pre-tras pre-trained encoders.
Overall, Radford al. [I] demonstrated that treating image recognition as language interpretation allows generalizing to a multitude tasks without training explicitly for them. By building an interpreter, CLIP changed the way computer vision is approached [4] rather than extracting the "dog" label from an image like a CNN (). CLIP tests the hypothesis of a dog being in the image against all the other hypotheses. The success of this image-text association is testament to the power of deep learning: the CLIP model was the first of its kind, and building it required joint training of two large neural encoders vast collection of image-text pairs.
Correspondence to Antonio Norelli <norelli di uniromal.ll.mmmmmm lit> A demo sufficient to reproduce the main results in the paper within minutes, even from smartphone, can be found here: https :// thtbb..tttuttttt///g www. htmm/ooranaa/AAS
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Still, training neural networks at such scale presents several challenges beside the obvious infrastructure and training costs. Notably, it requires collecting massive training sets, mak- ing it difficult to interpret the predictions of the model in light of their training data. Ad- ditionally, the training assets are often not owned by the institution training the model []] This introduces several additional challenges, from reproducibility to the difficulty of en- suring that an asset owner can remove their data from the model [ННи Overall, these considerations make large multimodal mod- els relatively inaccessible to researchers and practitioners until checkpoints access to demo is granted. And even then, the ability tweak the models by adding removing training data or to interpret their re- sults is limited.
E1
8/
x
released or
Vi
E2
1/
Vi
M/
3
yk
1/
this paper, we present ASIF, a simple non-parametric procedure that turns pre- trained uni-modal image and text encoders into multimodal model using relatively small" collection of image-text pairs and no additional training, depicted in Figure [ The resulting model functionally equivalent CLIP, effectively producing aligned representations of images and their captions.
Figure 2:
The ASIF construction. An ASIF model defined by two unimodal pretrained encoders and collection of coupled embeddings. This is suf- ficient to compare elements from different modes through representations made of similarities with ground-truth pairs: yj more similar to x² than y/-
Intuition. The key insight is that captions of similar images are themselves similar (Fig, 3 and therefore a representation crafted using just similarities to ground-truth multimodal pairs is quasi mode-invariant (Fig.
Freshly picked poured copper still.
The ASIF procedure is not only efficient but also has several intriguing features built-in. One of the key advantages is the ability to easily edit the model- adding new image-text pairs or removing outdated ones is as simple as encoding or canceling the corresponding embeddings. Furthermore, the multimodal representations are highly interpretable, as each dimension corresponds to the similarity of the input to specific entry in the multimodal dataset.
Image similarbiis
10.
8087
Contribution. Our results are surprising and raise several questions. Despite (1) the simplicity of the upproach, (2) multimodal dataset that is up to 250 times smaller than in prior work and (3) the lack of ac- tually training the model on multimodal data, ASIF achieves zero-shot classification accuracy on downstream datasets that is comparable to CLIP 1 22 This raises important questions on the data efficiency in foundation models, making ASIF a very powerful and cheap baseline for future work, and opening new doors for data centric Al
10a
10a
-02
0.0
0.2
04
0.6
0.8
1.0
10a
10a
1010
In summary, we:
Introduce the ASIF procedure, which turns two pretrained unimodal black-box encoders into an interpretable multi- modal model without tuning a neuron.
-02 0.0
02
04
06
08
10
Figure 3: Captions of simi- lar images are themselves similar. Distribution of sim- ilarities of 100k embedded pairs in the training set ver- sus the above image and cap- tion embeddings. We high- lighted in orange the 1000 pairs with the highest image similarity.
Demonstrate the effectiveness of ASIF models on zero-shot image classification tasks, where they achieve performance in the same ballpark of CLIP with significantly fewer image-text pairs.
Discuss the unique properties of ASIF, its implications on the role of memory and retrieval in machine learning, and the new opportunities it opens.
'CLIP [1] experiments used from 400M to 15M captioned images training samples, LiT 2 from 901M to 10M. Our experiments use 1.6M.

---

Multimodal dataset
Test
samples
.11
14
A bsolute representations
Image Encoder
Relative representations
BF
Text Encoder
g+
Invester with
31
BU
M
$9
V1
91
b
used it were
We select y since (0t,3/) > Co,,
11
21
-14
84
86 74 -11 188 87 30
8+
Br
34.
By
d Figure 4: Zero shot classification with ASIF. In this example we determine the best caption for the image x" from the two candidates, y and 7/-- a. Compute and store the embeddings of the multimodal dataset and the test samples. b. Compute the relative representation of the test image and the candidate captions. c. We consider the relative representation of x² with respect to the image collection æi, .
Fn if it was the relative representation of n² the ideal caption for x with respect to the corresponding caption collection yi.....9lu Yn- d. We choose the candidate caption most similar to the ideal one.
2 Aligning Pre-Trained Models with ASIF
In the following we present how a collection of captioned pictures implicitly defines a common space for images and texts through relative representations [13] allowing to build multimodal model without training. Here we focus exclusively on vision and language as modalities due to the wider availability of relevant models and paired data. However, we anticipate that our procedure could be more generally applicable. Indeed, subsequent research by other teams has already utilized ASIF as baseline in other modalities, such as audio (14). Before delving into the specifics of our method, we will briefly review existing techniques for establishing this common space.
Contrastive training to build a common space. With multimodal models, we refer to architectures that embed inputs of diverse modes into the same space. The better multimodal model. the closer are representations of different modes of the same object. So far, this common space has been obtained as the result of contrastive training of one (2 or both the neural mode encoders I [5]. Using a collection of image-text pairs as training set, a contrastive loss promotes the closeness of image and text embeddings from the same pair, while spacing out the ones from distinct pairs. Zhai et al. 2 train just the text encoder to match the image embeddings of pretrained visual encoder. Once the encoders are trained, a multimodal model can be adapted to perform any visual classification task just by crafting a caption for each label and selecting the one with the embedding closer to the embedding the input image (zero-shot image classification).
Relative representations. Our idea to build a common latent space is to use a collection of coupled data as a "rosetta stone" between modalities, and represent each new data point as its similarities to the points of the same modality in the collection. In other words, we compute relative representation for each new data point:
Definition 2.1. Given an encoder E : X + Rd and a subset of samples (æ1...., 2n denoted as anchors, we define the relative representation of x' as the n-dimensional vector:
z' [sim (0 (1),
sim (2' xx))
for some similarity function sim, e.g. the cosine similarity. E X are input samples, e.g. images or texts; J. E d are the embeddings of x; obtained with the encoder F., i.e. the absolute representations of æz; while x; R are the relative representations of z,.
3

---

We observe that when each anchor is available in two or more modalities, we can compute relative representations of samples from those modalities using the same subset of anchors. Most notably, these relative representations will all live in the same space, even when they are representing samples from different modalities. This foundational insight is illustrated Figure 2
Relation with Kernel methods. Definition 2.1 may not look surprising to readers familiar with the literature on kernel methods (16]. Instead of presenting x' kernel, we say it is a relative representation to stress that (1) we want to explicitly represent the coordinates in our ASIF procedure as opposed to operating in an implicit feature space and (2) we do not aim at learning regression parameters, although we note this may help with the inference speed. Instead, we rely on simpler procedure that may be interpreted as a hard version of the Madsaan-aaaraya [17] [S] with a distance threshold. Nevertheless, it is worth noting that while the ASIF procedure hinges on the ability to compute similarities between samples, such computation can be achieved using kernel function, thus sidestepping the need for explicit representations generated by unimodal encoders. Although integrating kernel methods could offer benefits, as alluded earlier, delving into these prospects is beyond the scope of this paper. Our central focus remains on illustrating how single-domain pre-trained networks can be aligned without additional training.
regression
ASIF: relative representations inducing a meaningful common space. Consider the embedding spaces of any two good text and visual encoders, we expect captions of images that are close in the visual space to be themselves close in the language space. This fact makes representation defined in terms of similarities against set of ground-trutt multimodal pairs almost moodeinnvarinnt, i.e. an image and its caption share almost the same representation.
That is why in this new space: we can consider the relative representation of x² respect to the image collection (x) .T.) as if it was the relative representation of its ideal caption y* with respect to the counterpart collection (yi,..., Vn), see Figure 4 The whole procedure to set an ASIF model and use it to find the best caption for a new image follows.
assign the best caption
image x² just by performing nearest neighbors
ASIF recipe. Ingredients:
Two good encoders, each mapping a single data modality to vector space. Let X and Y be the mode domains, for instance a pixel space and a text space, we need Ej X + Rdl and E, Y + R242
A collection of ground truth multimodal pairs: D (II1,11),....... .... (Ens Wn)}, for instance captioned images.
Procedure to find the best caption among a set of original ones ý = {ūi.... .......... dct for a new image x=:
1. Compute and store the embeddings of the multimodal dataset D with the en- coders E,, and discard D. Now in memory there should be just DE [(21,v1),. V.).., (I,,, }n:};
2. Compute the n-diimnnsional relative representation for each candidate caption 9; = (i,, 3/ ... sim(0:, vw)). where sim is similarity function, e.g. cosine similarity. Then for each 0 set to zero all dimensions except for the highest k; and raise them to
p 2 1. Finally normalize and store the processed vectors y:- Choose i and p to taste, our experiments k = 800 and p = 8;
3. Compute the relative representation x² using the other half of the embedded multi- modal dataset DE and repeat the same processing with the chosen k and p;
4. We consider the relative representation of the new image x" as if it was the relative representation of its ideal caption y", i.e. we define y" x=. So we choose the candidate caption 0: most similar to the ideal one, with = argmax, (sim(y" (:)).
To assign one of the captions to a different image x=* repeat from step 3.

---

Properties of ASIF models. The above construction yields several intriguing properties for free:
No training and "data centric". As have seen, ASIF model is built top of two independently pretrained encoders and the embeddings of a multimodal dataset, and so without training or finetuning any neuron. Being deployable updatable in seconds, ASIF model is radically "data centric" [12] For example, it is trivial to adjust the model by adding or forgetting specific samples. The latter use- case particularly important, as the right to use specific assets may change over time and removing the effect of specific samples from a trained network requires sophisticated forgetting techniques, e.g. IZ- IT. In our procedure, the encoders should be pre-trained with established data sets that do not change over time, while removing the effect of a multimodal pair is as simple as deleting its embeddings.
Data efficiency: Being able to exploit two pretrained encoders, ASIF models require far less ground- truth multimodal pairs to become effective. As confirmed by our experiments, ASIF reaches competi- tive zero-shot performance diverse classification benchmarks by using a fraction of the multimodal data of its predecessors, reaching a respectable accuracy even with thousands of pairs (we refer to Section for more details). This in line with classical work in computer vision, where prototypical networks 19 are a strong baseline in the extremely few-shot regime.
Interpretability: Sparse relative representations make the ASIF models interpretable classifiers. In fact, we can trace back every prediction to a small set of data points in the multimodal dataset corresponding to the dimensions that nonzero in both the image and the label relative representations- accountable for the outcome (at most k), see Figure This enables visualizations of the relevant samples contributing with other approaches that are often costly 20 and fragile 21 22
correct incorrect prediction no extra cost, in stark contrast.
Relation to k-Nearest Neighbors. Like K-NN, ASIF is a non-parametric supervised learning algorithm that requires an explicit representation of every entry in the training dataset at test time. Differently from k-NN, ASIF can perform open-ended classification, as shown e.g. in Figure 4 with two competing brand new captions. Indeed, ASIF is functionally equivalent to CLIP, and can function as drop-in replacement in applications using CLIP-like models.
Implementation that scales. Clearly, our method pays the price of avoiding training with larger memory footprint and increased computation at inference time, since we need to compute not only the embeddings but also the cosine similarities against the multimodal dataset. As such, our approach should not be considered general one-stop replacement for CLIP, although in our experiments we managed to scale ASIF to 1.6M pairs while maintaining a reasonable inference speed. Our non-optimized implementation of ASIF is less than 2x slower than CLIP. On positive note, there are two well-established techniques that could radically enhance the efficiency of ASIF and potentially enable scalability to billions of entries. The memory footprint required to store all the embeddings of the multimodal dataset can be dramatically reduced using product quantization [23], while inverse indexing [24] can be used to cireumvent the need for computing the cosine similarities against the entire dataset at test time. These techniques are both implemented e.g. in the FAISS library (25). Finally, find that the distribution of pairs chosen during inference is rather short-tailed, presenting opportunities to massively prune the model even at deployment time, deleting from memory the data that is never used. It should be noted, however, that the assessment of the performance of large-scale optimized ASIF models is beyond the scope of this work. While it is an interesting direction for future research, the focus of our current study is on establishing the potential the ASIF method.
2.1 Design choices and implementation of ASIF models.
Curating the multimodal dataset. While neural models like CLIP or LiT are defined just by the weights of the two encoders, to univocally identify an ASIF model we also need the embeddings of the multimodal dataset. Even if two image-text ASIF models rely the very same encoders, they comply to different visual understandings of the world if they are based on different collections of image-text pairs, therefore achieving different performances on the same tasks. Unlike conventional neural vision models, ASIF enables effective curation of the training dataset through swift iterations, given the absence of training and the smaller datasets. Furthermore, ASIF provides the means to assess the impact of each training pair, as demonstrated in 6
Salient hyperparameters. While using the raw relative representations already produces an ASIF multimodal model with non-trivial capabilities, we found that two simple treatments greatly improve performance and efficiency, and also foster interpretability.
5

---

Method
Dataset size
ImageNet
CIFAR100
Pets 88.9
ImageNet v2
CLIP [1
400M (private)
68.6
68.7
CLIP M
15M (public)
31.3
LiT [2]
10M (public)
66.9
CLIP Zhai et 2 uu)
901M (private)
50.6
47.9
70.3
43.3
LiT [2]
901M (private)
70.1
70.9
88.1
61.7
ASIF (sup vis. encoder)
1.6M (public)
60.9*
50.2
81.5
52.2
ASIF (unsup vis. encoder)
1.6M (public)
53.0*
46.5
74.7
45.9
Table 1: Zero shot classification accuracy of different multimodal designs. CLIP and LiT implementations vary by dataset and the visual transformer used as image encoder. The first CLIP and LiT entries use VITb16 as ASIF, the last CLIP and LiT entries use VITb32 (larger patch size). The public dataset of CLIP is a curated subset of YFCC100m [38], while LiT and ASIF use CC12M,
"e used subset the ImageNet validation set tune two hyperparameters of ASU were then used the other data sets, mumber reported table lest set. When tuning on different dhsasess, accuractes slay consistent, see the appendis..
i) Sparsification. We set to O all the entries of the n-dimensional relative representation except for the top k. In experiments and A are respectively in the order of millions and hundreds. In this way we cut off the small noisy signals from the dissimilar entries, that accumulated during comparisons would destroy the signal from the few similar entries. Furthermore we get highly interpretable representations that can be efficiently compared, since we have just k nonzero features, each one linked single entry in the multimodal dataset.
ii) Exponentiation. We raise all the nonzeroed similarities sim(æ', (z,, 2.) to with 2 1. This non- linearity weighs more the contribution of the most similar entries in the relative representation. Besides the pivotal choice of the ground-truth multimodal pairs, the number of non-zero elements k and the exponent p are the salient hyperparameters to consider when deploying an ASIF model. In general, we found that picking a F 1 may help, while choosing a k & n is always crucial. For more details see Section 3
2.2 Closely Related Works
Classics. In [26). Stanley Ulam affirms that a mathematical formalization of the word "as"-on a par with the connectives "and", "or". "implies" and "nol" be a key milestone to artificial intelligence. This idea of analogies being at the core of cognition is shared by Hofstadter [27]. who states that a concept is a collection of analogies, in line with what the ASIF procedure prescribes.
Retrieval augmented foundation models. Recent works in NLP enhance unimodal language models with retrieval to reduce the size of the architectures and the training dataset while also making results more transparent [28, (29]. Our work is in line with this view, that the ASIF procedure extends to the multimodal case. Importantly, ASIF offers a new perspective on data centric AI (12), where data and the external memory implement the alignment between modalities. Networks with discrete Key- Velue bottlenecks [30] are closely related our approach, with the critical differences that our memory is not learned and that their decoder is trained for classification. Retrieval and memory-augment models have also been successful in Reinforcement Learning (3I, physical reasoning [32], and code generation [33]. Finally, we notice that making predictions on new samples by exploiting the similarity with dictionary of previous data points is a common approach in computer vision 19] for few-shot learning. Our procedure is also related to compressed sensing algorithms where a signal (in our case an image) is sensed as a sparse combination of fixed atoms [34 35 with an iterative projection procedure (7] and only transmitting the coefficients to the receiver (text modality).
Learning multimodal models. Following the intuition outlined by early works on aligning text and image embeddings 33 40), today large multimodal models are conquering the computer vision scene thanks to their wide applicability and easy transfer new downstream tasks [T 2 15 41).
We identify two key leaps respect to traditional models like ResNet [42]: (i) Free text allows to learn visual concepts beyond a finite set of predefined categories and exploit the structure of language to compose them, as masterfully seen in Dall-E 43 (ii) The recognition tag transitioned from being an output pulled out of the image by the neural stack (the label) become an input that should be interpreted, and therefore processed by its own encoder (the free text caption). This corresponds to an epistemological perspective shift, as discussed by Norelli al. (4. Data and learning efficiency are clear challenges for large multimodal models, that often require hundreds of millions of examples. Efforts such as 29] attempt to reduce this. ASIF presents different take on this problem, showing how much can be achieved by simply remembering the training data efficiently.

---

3 Empirical Evidence
In the following we compare ASIF to traditional multimodal models based contrastive training, CLIP and LIT. We then take a closer look to the classification of a single image, unpacking the relative representations and following the classification algorithm step by step. As a prelude the above, we start by discussing the pretrained encoders and dataset forming the ASIF models we tested.
Pretrained encoders and multimodal dataset used. For our primary experiment, we utilized vision transformers as image encoders, pretrained either in supervised manner (DEIT base, an unsupervised manner (DINO VITs8, 45 on Imagenet 1k [46] and 21k 47 respectively. The embedding size was 768 for DEIT and 384 for DINO. This configuration aligns with that used in LiT (2). with the sole distinction being that, unlike LiT, we used a pre-trained frozen text encoder. Regarding the text encoder, we employed the SentenceT transformer [48], trained on a dataset comprising more than 1 billion sentences obtained from the internet. We employed the first 1.6M entries of the Conceptual Caption dataset (CC12M, 49]) as our multimodal dataset. This dataset single
44 or in
amasses images and filtered alt-texts collected from the internet. To optimize performance Tesla T4 GPU, we limited our analysis to the initial 1.6M pairs. In our "scaling- experiments, we also utilized DEIT tiny and small vision transformers [4 and two smaller SentenceT encoders.
Zero-shot performance. We assessed the quality of our ASIF multimodal model by comparing its zero-shot classification performance against CLIP and LiT on four datasets: CIFAR100 Imagenet, Im- agenetv2, and PETS (6,55555)) see Table [
We crafted label prompts as in Zhai et al. [2] Table 11].
Remarkably, we achieve competitive performance with CLIP and LiT using two frozen pretrained encoders and a fraction of the image-text pairs.
The two ASIF models reported in Table differ for the vision encoder, that is respectively supervisedly (DEIT) and unsupervisedly pretrained (DINO). We tuned k and p on the ImageNet validation set, in both cases we used k = 800 and 8. The comparison between the two versions is not completely fair since the visual transformer architecture of DINO is smaller (e.g. the codings are 384-dimensional rather than 768) but corroborates the effectiveness of ASIF with encoders pretrained using no supervision. In the appendix we report the results of wider collection of ASIF models based on different visual and textual backbones.
Summary:
Overall, we observe that ASIF procedure can achieve competitive zero-shot results with a fraction of the image-text pairs used in prior work (assuming access to other pre-trained, even unsupervisedly, unimodal models).
ASIF scaling laws. In Figure 5 we show the full zero-shot accuracy trend on Imagenet as the size of the multimodal dataset grows for different choices of k and p. ASIF models become effective very quickly: we reach a non-trivial 18% classification accuracy using just the first 10,000 pairs in the multimodal dataset. We recall that building an ASIF model from scratch still requires lot of uni- modal data to pretrain the encoders, but now this data may come untied and even without any label.
ncmizeros: 50. vap_ 1
magaastl aecuracy
0.5
nenoeeros: 800,
nem-zeros: 30. vel_ 8
500 esp:
0.4
ncn-2eros: -600, exx:
nemonemns: 1600 vol, expe
0.3
0.1
10a
10a
30a
20'
10"
We tested ASIF further with smaller image and text encoders on multimodal datasets of different sizes (102 to 10 image-text pairs from CC12M) to provide early evidence about ASIF scaling laws. We used DEIT tiny, small, and base vision trans- formers 44 along with two smaller SentenceT encoders. As we see in Figure 5 and in the appendix, Imagenet classification accuracy grows smoothly with the size of the multimodal dataset, and per- formance does not saturate earlier with smaller encoders. These results promising but still preliminary, further experiments with larger multimodal datasets are left for future work.
Size of the mo muttimoda dataset
Figure 5: ASIF is a learning algorithm: Im- agenet accuracy improves smoothly as we in- crease the size of the multimodal dataset. Colors show the impact of k and p (non-zeros, val exp).
Summary: Our experiments show a steady improvement in ASIF's performance as the size of the multimodal dataset increases. Performance deteriorates with smaller encoders, but even then there no sign of saturation or plateau.
7

---

Larpe Mutroceed other dade in aggs
Query image
0.00
Most Nostaimitr imaiiitrrrrggoooo images The mumimmooi citavee
x
Nooc aodia Arch incl-rs by Thee Archen of thrugh
Ruine Ramen teemein anth Falmana 1.70
0.01
t'umphal
RUM
Thrms Phore andras.
5.84
0.00
C27
A Iestomsoooo dkutch Srandentinng gala
Cornaspondngg
0.94
Aech Saptrnux
Romm
Conmantin Protesblyy
0:57
The Arch Hhmm
3.57
2.00
DVD
and AIEESSRR
0.00
und copootet
AC Milart players
0.11
.PERSONN parber on courta vorrssiie
C32
Askhee lamou German
0.7
Our riie near Buly sllck pho holel
Dursge Uimaped Moopue Darmiccaa
C.00
Arth of Tile
6/0
<P0 DND Archis
2.08
2.00
0.01
The Arch Thejer all
0.00
Back and white Uc parn
The Huunshed anch
0.00
The Anch of TV anrilt Puligge Sprta 2009
0.00
0.00
5.60
Valke of Kinge
6..5
oin
Figure 6: ASIF representations are interpretable and amendable. Thorough analysis of the relative representations -the four vectors in green- produced by ASIF to assign the best caption to a given image x². Every dimension corresponds to the similarity of the input image (text) to unique image (text) entry in the multimodal dataset. We can visualize the training samples leading to correct or incorrect predictions, which is useful curate better data. For example, the second training sample is broken, we can remove it and produce an updated ASIF model in seconds.
Adjusting an ASIF model in seconds. Consider classifying EuroSAT [53] satellite images using ASIF. Initially, the zero-shot performance outperforms random chance but is not impressive (29.4% unsup. configuration, 10 classes). CLIP, while better, also falls short with a 54.1% performance rate.
Now, imagine we acquire 100 new image-text pairs from Eur Our goal is to develop new multimodal model based on an updated training set, anticipating the need to process more satellite images in the future. With CLIP, this would require us to fine-tune or retrain the model from scratch using the updated dataset. In contrast, ASIF requires only to obtain and store the new pairs' embeddings. The model retains its usability for all the previous images, but it now also makes accurate predictions for satellite images, achieving an 82.2% 2.0 accuracy on EuroSAT
Similarly, if we need to remove samples from the multimodal training dataset-either because they are faulty (as shown in Figure 6 or because we have lost the license use them-the process is as simple as deleting the corresponding embeddings to get new model.
Summary: ASIF enables quick model adjustments and data handling without the need for retraining, unlike traditional models such as CLIP. This demonstrates its practicality in real-world scenarios, such the emergence of new setting the loss of rights for assets used during training.
Deep dive into a classification. To better understand why and how the ASIF procedure works, we are going to follow step by step the single classification depicted in Figure 6 showing the entries of the multimodal dataset used to build the relative representations, For simplicity we assume k = 23.
We want to find the Imagenet label of the upper left image in Figure 6 The first step is to compute its relative representation with respect the multimodal dataset, this procedure selects the 23 most similar images in our collection. The most similar are indeed triumphal archs that-as similarity decreases turn to generic structures with archs or ancient monuments. No other image in the multimodal dataset is involved in the classification, we could replace all the entries in our collection except for these 23, and as long as no new image is more similar to the input image, every computation in the classification process would remain the same. This consistency of the ASIF procedure is in contrast to traditional approaches like CLIP and LiT, where the parameters defining the encoder computations are shaped by the whole multimodal dataset through contrastive learning.
Now, we approach the pivotal step of the ASIF procedure: jumping into the text space by treating the relative representation of the input image as if it were the relative representation of its ideal caption. The vector remains the same, but its meaning changed: now signifies how much the ideal caption should be similar to 23 captions.
Why does CLIP perform better zero-shot? Likely, its 400M pairs have more samples close to satellite images than ASIF's 1.6M samples from CC12M, evidenced by ASIF's drastic improvement with just 100 new samples.
8

---

The efficacy of this jump hinges entirely on the quality of the multimodal dataset. Looking at the 23 captions in Figure 6 we are confident that the most fitting candidate caption corresponds to one of the prompts associated with the triumphal__ arch label in ImageNet, such as "a photo of a triumphal arch". " Conversely, dataset featuring 23 non-informative captions, such as file names "IMG_20 1el_20088880 888223.jp' or camera settings "D90 18.0 18.0-70.0 mmn f/3.5-4.5", would not allow the model to recognize the correct class even with the best image and text encoders. This limitation is actually a defining feature of the ASIF procedure, as the meaning attributed to the input is ultimately determined by the multimodal dataset and not by the encoders: by acting just on the image-text pairs, we have full control over the model's output.
Summary: Our simple (non-cherry picked) example showcases how the ASIF predictions can be easily attributed to specific examples in the multimodal training data by construction. This feedback can be used to explain predictions and grow high quality datasets in data centric AI, for example by inspecting which examples contribute to incorrect classifications.
Discussion
The effectiveness of the ASIF procedure raises questions on the role of memory and retrieval in machine learning, while at the same time opens new opportunities for products based on multimodal models, opening many avenues for future works. the following we will discuss these aspects.
Perception and interpretation disentangled. In ASIF there is no trace of the multimodal data in the weights of the encoders, which are pretrained on different unimodal datasets. Nonetheless, the relative representations and the outcome of any classification task fundamentally depend on the multimodal dataset. This state of affairs reflects the factorization of perception and interpretation in the two stages constituting an ASIF model; the encoding and the construction of the relative representations. Such factorization desirable because it relieves the black- box neural encoders from the responsibility of attributing meaning to their inputs, as envisaged by Eco [54] Par. 3.3. Therefore, we can consider the neural encoders as mere sensors and focus only on the second stage to analyze, explain, and act on the interpretations of an ASIF model, as shown in Figure 6
Learning or retrieval? As we have seen, the ASIF procedure requires no training: it does not distill the multimodal data into any learnable parameter. Rather, it prescribes rigid memorization of the embeddings of the multimodal dataset, where each entry has its fixed-size spot, similarly to retrieval process. On the other hand it seems impossible to not describe ASIF as learning algorithm; for instance satisfies the fundamental characterization given by Mitchell [55]: the more the multimodal data the more ASIF improves, as we can clearly see in Figure 5 Ultimately, an ASIF model is functionally comparable to CLIP. ASIF blurs the border between learning and retrieval by questioning the effectiveness of storing information only in the weights, and rather advocates to combine learning representations with external memories. We encourage more work on memory augmented neural networks and towards understanding the implications of memory for generalization.
Generalization new distributions. The empirical performance of ASIF calls for a discussion on zero-shot and out-of-distribution generalization in foundation models trained with huge data sets. Clearly, the performance of ASIF will depend strongly on the multimodal data used for alignment. As an example, the good performance on Imagenet may not be particularly surprising in light of the qualitative evaluation seen in Figure 6 There, our query image might as well had been part of our multimodal data set, as the semantic gap with its neighbours appears to be small. Despite this, our choice of downstream evaluation and pre-training data set is identical compared to prior work (2). As such, while it appears clear that ASIF should fail when the semantic gap between downstream task and "training data" is large, it is unclear why it should be different for more standard models like CLIP ① and LiT [2] if gap does exist, future work should work to address it. In the meanwhile, we recommend that future foundation models are benchmarked on significantly broader sets of downstream tasks, ideally with some analysis of the semantic distance between test and training data (if any). Alternatively, our results may be interpreted in light of the strong performance of unimodal models. There may be limited benefit of training from scratch on less curated multimodal data sets compared to standard and well established unimodal data sets, although we posit that at even larger scales the difference may be more significant.
Limitations. The simple ASIF procedure presented here offers a strong baseline for multimodal models, but its performance still falls apart to CLIP and LiT when the mutimodal dataset is abundant

---

and the cost of training is not concern Additionally, the large dimensionality of the relative representations, even if sparse, poses challenges for directly applying ASIF to tasks like text-to-image generation. We recognize that the experiments reported in this paper do not provide comprehensive examination of the myriad of downstream tasks multimodal models are known to adeptly handle, important to note that such broad coverage was explicitly out of scope for this work. Our primary objective here was to introduce and justify the ASIF procedure, illustrating its effectiveness on the representative task of zero-shot classification, In making this choice we followed [2], that used the very same datasets to showcase the benefits of locked image encoder, that their main claim. We anticipate and welcome more extensive evaluation of ASIF in the context of a wider range of tasks in future research endeavors.
Conclusions. We presented simple procedure called ASIF to assemble a fully functional multi- modal model like CLIIP from two unimodal pretrained encoders and a collection of image-text pairs without tuning single neuron. While achieving competitive zero-shot classification results with its predecessors using a fraction of the data, the proposed ASIF procedure enriches multimodal models with editability-a new model based different pairs can be deployed in seconds-and interpretable representations. The effectiveness of ASIF models also clashes with the dominant view of learning algorithm as way to distill data into the parameters of model, and raises questions on the role of memory and retrieval in machine learning.
Acknowledgments
AN, MF, and FL partially worked on ASIF when they were at Amazon Web Services in Tübingen, Germany.
This paper is financially supported by the PRIN 2020 project no. n0.20200A3K9N (LEGO.AI), PNRR MUR project PE0000013-FAIIR, and ERC Grant no. 802554 (SPECGEO).
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748-8763. PMLR, 2021.
[2] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123-18133, 2022.
[3] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunitics and risks of foundation models. arXiv preprint arXiv:2108.07255 07.58 2021.
[4] Antonio Norelli, Giorgio Mariani, Luca Moschella, Andrea Santilli, Giambattista Parascandolo, Simone Melzi, and Emanuele Rodolà. Explanatory learning: Beyond empiricism in neural networks. arXiv preprint arXiv:2201.10222, 2022.
[5] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
[6] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843-852. 052,,552,44 2017.
[7] Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privacy forgetting deep networks. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 792-801, 2021.
10

---

[8] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrub- bing deep networks of information accessible from input-output observations. In European Conference Computer Vision, pages 383-398. Springer, 2020.
[9] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 9304-9312, 2020.
[10] Antonio Ginart, Melody Guan, Gregory Valiant, and James Zou. Making forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
[11] Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In International Conference on Machine Learning. pages 3842-284. PMLR, 2020.
[12] Andrew Ng. Unbiggen ai. IEEE Spectrum, 2022. URL https://spectrum/ ieee org/ andrew-ng datg-dta--aa centric-ail
[13] Luca Moschella, Valentino Maiorea, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. 1/th International Conference on Learning Representations, ICLR 2023, 2022.
[14] Gary Wang. Kyle Kastner, Ankur Bapna, Zhehuai Chen, Andrew Rosenberg, Bhuvana Ramab- hadran, and Yu Zhang. Understanding shared speech-text representations. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[15] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- langua-a representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021.
[16] Thomas Hofmann, Bernhard Schölkopf, and Alexander J Smola. Kernel methods in machine learning. The annals of statistics,
2008
[17] Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications, 9(1): 141-142, 1964.
[18] Geoffrey S Watson. Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series A, pages 359-372, 1964.
[19] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gareett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4077-4087, 2017.
[20] Alessandro Achille, Aditya Golatkar, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Lqf: Linear quadratic fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15729-15739, June 2021.
[21] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894, PMLR, 2017.
[22] Samyadeep Basu, Phil Pope. and Soheil Feizi. Influence functions in deep learning are fragile. In International Conference on Learning Representations, 2021.
[23] Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1): 1371:1187, 2010.
[24] Sivic and Zisserman. Video google: text retrieval approach to object matching in videos. In Proceedings Ninth IEEE International Conference on Computer Vision, pages 1470-1477 vol.2, 2003. doi: 10. 10.119//CCVV.2033133866.

---

[25] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-544 2019.
[26] Gian-Carlo Rota. The barrier of meaning. Letters in Mathematical Physics, 10(2):97-99, 1985.
[27] Douglas R Hofstadter. Analogy as the core of cognition. The analogical mind: Perspectives from cognitive science, pages 1999538. 2001.
[28]
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katic Millican, George van den Driessche, Jean- Sran-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, ICML. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2206-2240. PMLR, 2022.
[29] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yen Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning 2022.
with retrieval augmented language models. arXiv preprint arXiv:
[30] Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf. Discrete key-value bottleneck. arXiv preprint arXiv:2207.11220, 2022.
[31] Anirudh Goyal, Abram Friesen, Andrea Banino, Theophane Weber, Nan Rosemary Ke, Adria Puigdomenech Badia, Arthur Guez, Mehdi Mirza, Peter Humphreys, Ksenia Konyushova, et al. Retrievallaugmented reinforcement learning. In International Confer- ence on Machine Learning, pages 7740-7765. PMLR, 2022.
[32] Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael C Mozer, and Yoshua Bengio. Neural production systems. Advances in Neural Information Processing Systems, 34:25673-25693 2021.
[33] Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. Retrieval- augmented generation for code summarization via hybrid gnn. In International Conference on Learning Representations, 2021.
[34] Jian Wang, Seokbeop Kwon, and Byonghyo Shim. Generalized orthogonal matching pursuit. IEEE Transactions on signal processing, 20(12):6202-66 6216, 2012.
[35] Stéphane G Mallat and Zhifeng Zhang. Matching pursuits with fime-frry dictionaries. IEEE Transactions on signal processing, (1(1)):3397-3415 1993.
[36] Francesco Locatello, Rajiv Khanna, Michael Tschannen, and Martin Jaggi. A unified opti- mization view on generalized matching pursuit and foane-wolfe. In Artificial Intelligence and Statistics, pages 860-866 668. PMLR, 2017.
[37] Francesco Locatello, Anant Raj, Sai Praneeth Karimireddy, Gunnar Rätsch, Bernhard Schölkopf, Sebastian Stich, and Martin Jaggi. On matching pursuit and coordinate descent. In International Conference Machine Learning, pages 3198-3207. PMLR, 2018.
[38] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73 2016.
[39] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. Advances in neural information processing systems, 26, 2013.
[40] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128-3137, 2015.
12

---

[41] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini. and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[42] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[43] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv: 1iiii-2004.06/25. 2022.
[44] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training titt-efffciiett image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[45] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[46] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[47] Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik. Imagenet-21K k pretraining for the masses. In Joaquin Vanschoren and Sai-Kit Yeung, editors. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
[48] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties. 11 2019.
[49] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tai tail visual concepts. In CVPR, 2021.
[50] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, ON, Canada, 2009.
[51] B. Recht, R. Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? International Conference On Machine Learning. 2019.
[52] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1498-3505, 2.12. doi: 10. DB.100/CCPPR222222248888
[53] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2117-2226, 2019.
[54] Umberto Eco. Kant and the platypus: Essays on language and cognition. HMH, 2000.
[55] Tom Mitchell. Machine learning, volume 1, 9. McGraw-hill New York, 1997. ISBN 0070428077.
[56] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
13

---

Appendix
In this appendix, we further showcase the interpretability of ASIF models when used for classification in Figure / Then we provide additional details for the scaling laws and EuroSAT' experiments presented in the main paper, and report additional results about the impact of the size of the encoders (Table and of the image training dataset. We also report further evidence that the ASIF construction not overly sensitive to its hyperparameters. Lastly, we discuss more in detail the idea that captions of similar images are alike Figure I0
Training responsible the prediction
area, sac, residential area satellite image of an urban
image
industrial industrial city, factories satellite image of permanent
arable land, or orchard satellite image of highway, motorways, highways, or a street satellite image of meadow, herbaceous vegetation, grass, or fields
Tou SMARO
image of pasture, farmland,
085
waterway, stream satellite image of forest, woodland, trees
image canal,
satellite image of an ocean, a water, sea, reservair satellite image of farmiand, farms, or an annual crop
Irage aimiarty
coea
Cognitioe
Equipment Advisor
hytrager
oal leed Gearhh
Antil
Training samples responsible for the prediction
Competing captions
sateliite image of meadow. herbaceous vegetation, grass, fields
a satellite image of an urba area, an industrial area, an industrial zone, a city, or factories
satelitee image farmland, farms, annual crop
TOESmmbrry
2:15
Satelitte image pasture. farmland,
Query image
satelite image o canal, 8 sateliite image o highway, motorways, highways, or street a satelite image of suburban area, de sac, or residential area a satelite image permanent crop, arable woodland, satelitte image of ocean,
waterway, stream
D
orchard
satellite image
0:22
8.0
0.4
reservoir
Figure 7: Interpretability EuroSAT classifications through ASIE Analysis of the classification outcome of two EuroSAT query images using ASIF. The scatter plot shows the samples in the training set closer to the query image and the candidate caption of corresponding color. Image and text similarity are computed through cosine similarity in the visual space of DINO and the text space of SentenceT. The size the marks is proportional to the product of the image and text similarity. The class chosen is the one with the largest total area. Below are shown the corresponding pairs from the training dataset CC12M. We can notice the distance between the EuroSAT dataset and the 1.6M samples of CC12M used, many of the closest images not from satellite and even then may have misleading descriptions, as image A in the second example. An interactive version of this plot for any ASIF classification can be obtained using code demo attached in the supplementary material.
A Additional details on the scaling laws experiment
14

---

ASF imagenet1 0-shot accuracy with different backbones Orrinn dataset: CC)2M)
Models used in the scaling laws exper- iments. As discussed in the main pa- per, we tested ASIF with smaller image and text encoders to provide early evi- dence about ASIF scaling laws. We used three different instances of DEIT [44] vision transformers, the tiny (5.6M pa- rameters, 192-dimensional embeddings),
0.6
small (22M, 384), and base (87M, 768), and the original VITb16 vision trans- former [56] (86M, 768). The DEIT mod- els were pre-trained on a smaller dataset, the standard Imagenet k training set [46]. while VTTb16 was pretrained on Ima- genet21k [47]. As text encoders, used smaller versions of SentenceTT [48). with 23M and 33M parameters (both 384- dimensional embeddings), in contrast to the 1 110M parameters of the main model (768).
01
encidens DEI Diy (88) Sestercel MIVUM (23M)
excader: DET (22M)
Sentencal Mi 32 caa0
102
101
10'
Number anchers
Impact the image encoder (train dataset CC12MI
0.1
pagenenta
Figure g shows that, with smaller coders producing smaller embedddings, do not observe a performance satu- ration within 1.6M image-text couples. Further experiments with larger datasets are left for future work.
excoders DEIT trny (MN) Sentencel mnnnet bace (11080
exceder DET small (2M)) Sentensof mpnnet (11OM)
101
10
Number anchers
Impact of the test encoder Irain dataset CC12M)
Impact of image pre-training data. In
0.c
Table 2 we report the complete results of ASIF models using DEIT encoders [44]. We observe the expected positive corre- lation between the size of the encoders and the classification accuracy. Interest- ingly, ASIF with the largest instance of DEIT beats the one based on the stan- dard VIT pretrained on Imagenet2 on three out of four of test datasets, while losing more than 10 points on CIFAR. These results may be interpreted in light of the similarity of the datasets we are using, with features useful to classify CI- FAR images less overlapping with Ima- genet1k k features with respect to the other datasets.
excoder: DETT base (570) Sertenco! M (3M)
encoles: Dell base (8/90- Sentencel MeniM-3 (53M)
mmniet
10'
10
10"
32"
Number of anchors
Figure 8: ASIF performance does not saturate earlier with smaller encoders. Classification accuracy keeps growing without saturating but is lower for smaller mod- els. Furthermore, we observe that the quality of the vision encoder is more relevant than the quality of the text en- coder with respect to zero-shot Imagenet classification.
B Additional details on the EuroSAT experiment.
EuroSAT, a renowned benchmark for satellite image classification, serves as a testing ground for out-of-distribution generalization in zero-shot and few-shot scenarios [53]. The dataset contains 27,000 images labeled under ten categories. Our ASIF model with a DINO visual backbone (denoted as "ASIF unsup' in [) achieved a zero-shot classification score of 29.4%. While significantly better than random chance, this modest performance is not surprising considering the scarce presence of satellite images in the CC12M dataset.
As a further experiment, we randomly selected 100 images from the EuroSAT dataset and incorporated them into our ASIF training set, raising the total to 1,500, 100 image-text pairs and leaving 26,900 images for testing. We created captions for the EuroSAT images using the template "a satellite image of [CLASS NAMEJ", This way the ASIF model improves dramatically, reaching a classification accuracy of 82.5 + 2.8% on EuroSAT (average + standard deviation of 5 trials).
15

---

ASIF backbones (Params, pre-training data) DEITúny (5.6M, Imlk) STminiLó (23M, see Sec.
ImNet
CIFAR
PETS
ImNet-v2
46.5
37.3
75.6
38.3
DEITsmall (22M, Imlk) STminiL12 (33M, Sec.
59.3
46.0
80.4
50.3
DEITbase (87M, Imlk) STbase (110M, see Sec. 3
60.9
50.2
81.5
52.2
VITb16 (86M, Im21k) STbase (110M, see
55.4
63.3
71.5
45.6
Table 2: Zero shot classification accuracy of ASIF models with different backbones. We observe that the ASIF procedure remains effective even with smaller encoders pre-trained on reduced visual datasets such ImagenetIk.
Contrarily, CLIP [ while demonstrating better zero-shot accuracy at 54.1%, is trained on a private dataset comprising 400 million images. This dataset may contain a larger number of satellite images than our 1.6 million subset of CC12M. Given the substantial improvement observed when we added just 100 ET images, it's reasonable speculate that CLIP'' enhanced performance might stem from its larger database of satellite images. However, confirming this theory is impossible due to the private nature of CLIP's training set.
can, nevertheless, examine the presence of satellite images in the CC12M dataset. Using ASIF models' unique interpretability property, can trace the training samples behind each classification. Figure displays two EuroSAT samples, one classified correctly and the other not, along with the corresponding CC12M pairs responsible for the classifications. We note that subset of CC12M is lacking in satellite images, and the few available often have misleading captions, such as a map of a drainage network tagged "a satellite image of canal, river, waterway, stream" instead of an urban area.
The images shown are an adaptation of the interactive plot to analyze any ASIF image classification we provided in the code demo attached in the supplementary material.
C ASIF sensibility to its hyperparameters
Finally, we present evidence about the sensitivity of the ASIF model to the hyperparameters p and k. Specifically, we show the hyperparameter search for PETS and CIFARIO0 in Figure Table with results the parameters tnnettued the two datasets reveals marginal improvements over the standard choice of k=800 and p=8. This suggests that the ASIF model is relatively insensitive to the choice of these hyperparameters.
Tuned PETS
Parameters p,k
CIFAR
PETS
(200,8)
60.9
72.3
CIFAR ImageNetIK
(1600.6)
64.9
63
(800,8)
63.3
71.5
Table 3: Hyperparams search: tuning on each dataset per row.
0.65
0.30
0.58
34
0.56
0.45
500
750
1500
250
500
1800
1250
Non Zorcs
Rec Zores
Figure 9: Hyperparameters search Left Pets, Right CIFAR100. Highlighted in the red square the performance achieved tuning on ImagenetIK.
16

---

spiiring whred.
with well and Pess sslad servers
flwers puarol coppor
Ri tie Dark Harth Mattnnat
Camor
5p gorning NotL Rand Spriing Speniing
37
39
50
12-
66
plobe charseter Hlrough undee bluck Bush viice
surptised helgehoo losks
euy
that bird vector
Cbas Comice, Bistdd Burd
dans spr naline reya Eltaination
Isstated
Mazwin (larr, Comk Arizi dit.
Imago
Imag
Image
216
E44
3.-
Figure 10: Caption of similar images are themselves similar. For image-text pairs, we show in the first row the distribution of the image similarities against 100k images in the train set in blue (CC12M), and highlight the 1000 most similar orange. The dashed lines indicate the mean of the two distributions. In the second row, show the text similarities against the captions of the same 100k (blue) and 1000 (orange) images. captions similar images are themselves similar, expect the dashed orange line in the second to be the right of the blue dashed line, observe. The average gap between the orange and blue lines in the second row over 10,000 image-text couples from CC12M is 0.098 + 0.070.
17


Reviews:
[{"review_id": "YUyJIsblsG", "reviewer": "Reviewer_t279", "summary": "This paper proposes to leverage single-modal pre-trained text & image encoders and a relatively small image-text dataset to create a CLIP-like open-vocabulary visual recognition model without training. The authors claimed that the proposed model ASIF is more training efficient, more interpretable, and can easier handle data copyright issues than CLIP and alternatives such as LiT. More specifically, relative representations w.r.t. the available image-text pairs are computed for image and text respectively. To remove noise, only top-k values are kept. The resulting sparse embeddings can be used for image classification tasks in a way similar to CLIP. The above process requires no training, and the anchoring image-text pairs can be removed without re-training. Moreover, by inspecting the top-K anchoring image-text pairs, human can understand how ASIF makes its predictions.", "questions": "My questions and suggestions are included in the weaknesses section.", "limitations": "Limitations are properly addressed.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "S1. Using relative representation to create a CLIP-like open vocabulary visual recognition model is new.", "weaknesses": "W1. Empirical results are weak.\n\nW2. The major claims (efficient training, data copyright, interpretability) have some flaws.\n\nW3. Scalability / generalization (to more tasks) remains unclear. \n\nThe only empirical results this paper presents are image classifications in 4 benchmarks, and the accuracy are far below widely used visual recognition models. I understand given that no training is required, this can still be considered strong results. However, this also means that this method is unlikely to be widely adopted. Moreover, for ASIF with supervised visual encoder (DEIT base), the ImageNet accuracy is also considerably lower than the original DEIT results. This raises a concern that the assumption of a available supervised encoder might not be realistic. Given the 1.6M anchor data, I also suggest the authors to run a baseline of taking existing single-modal encoders (unsupervised) and finetune on these image-text pairs with contrastive loss, so that readers can better understand the trade-offs of ASIF and training.\n\nI also have concerns on the claims. The uni-modal encoder also requires training, and it seems that the text encoder used was trained on 1B internet text, which is much larger than CLIP's 400M. Moreover, if the pre-trained encoders have data copyright issues, there's no way to remove them under the current ASIF framework without re-training. For the interpretability claim, the same procedure could be done for CLIP-like models given an anchoring dataset. (Retrieve nearest neighbors using CLIP's text/image embeddings in the image-text pairs.) Therefore, the interpretability is not ASIF's exclusive advantage.\n\nI acknowledge that the scalability limitation has been properly addressed by the authors. Although some possible techniques to improve it are discussed (L154-L169), I personally don't think this can replace a well-tuned CLIP model. For many real-world applications, inference cost may be more crucial than training cost. On the other hand, the current ASIF is only tested on image classification. I would suggest the authors explore different tasks for future version. Instead of arguing the potential by pure hypothesis, it would probably be more convincing to demonstrate the potential on a wide range of tasks / modality, given that current image classification accuracy are not promising.\n\nAdditional Suggestions:\n- To make ASIF look more promising, I'd suggest the authors try to utilize relative representations to improve CLIP. In this way, users can choose between finetuning / ASIF depending on the training / inference cost trade-off while getting strong accuracy.\n- From the accuracy difference of sup vs unsup encoders, it seems the data quality of pretrained encoders also plays a major role. Studying training data quality vs anchoring data quality might also be an interesting direction.\n\n\n---------------------------------\nUpdate after rebuttal\n================\n\nI appreciate the detailed answers from the authors. Most of my questions are answered and I have a better understanding of this submission; hence the increased confidence from 4 to 5. W2 was properly addressed and the authors promised to clarify the claims in revision. Unfortunately, W1 and W3 remains concerns (no additional results are provided) that outweigh the strengths. I've increased overall rating from 3 to 4 but still leaning negative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We invite further dialogue with the reviewer in the next phase to foster an updated assessment of our research that builds upon the discourse. We also thank the reviewer for the interesting points regarding the interpretability and scalability/inference tradeoffs\n\n\"*The only empirical results this paper presents are image classifications in 4 benchmarks” and “lack of evidence” on other tasks*\n- The primary evaluation follows the protocol of LiT (table 2 of LiT paper) and employs exactly the same four classification benchmarks used in ASIF.\n- In addition, we also present performance results on EuroSAT (as indicated in the penultimate paragraph of the Empirical Evidence section and detailed in appendix B) and we discuss ASIF's performance on audio data from published follow-up work (appendix C).\n\n*\"ASIF with supervised visual encoder (DEIT base), the ImageNet accuracy is also considerably lower than the original DEIT results. This raises a concern that the assumption of a available supervised encoder might not be realistic\"*\n\nA comparison against supervised learning on ImageNet is unfair as we focus on open vocabulary classification (also LiT ImageNet accuracy would be considerably lower). Similarly, it would be unfair to report performance of DEIT on any other data set without finetuning. The trade-off between specialization and generalization in image classification is well-known and has been widely discussed in recent literature, including multimodal open-ended models [1, 2, 3].\n\n[1] Wortsman, Mitchell, et al. \"Robust fine-tuning of zero-shot models.\" CVPR 2022.\n\n[2] Wortsman, Mitchell, et al. \"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.\" ICML 2022.\n\n[3] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" ICML 2021.\n\n\"*uni-modal encoders also require training, and it seems that the text encoder used was trained on 1B internet text, which is much larger than CLIP's 400M*\"\n\nIt is essential to clarify that we are not comparing like-for-like here. It is important to distinguish free-form text, which lacks labels, from a collection of captioned images, which are comparatively more scarce and challenging to gather. What is intriguing is the modest quantity of captioned images that ASIF relies upon (1.6M) after the encoders are pre-trained from purely unpaired and potentially unlabeled data. \n\n\"*Interpretability*\"\n\nOur approach is different because the prediction for us is mechanicistically caused by the nearest neighbour, if you were to change the anchor set then the prediction would change and not just the explanation. There is a vast literature on explainability, arguing that explanations need to faithfully summarise the internal model computations, see [1]. In our case, the interpretability pertains to the assignment of captions to visual features. \n\n[1] B-cos Networks: Alignment is All We Need for Interpretability, Bohle et al., CVPR 2022\n\n“*Trade-offs between training and inference costs*”\n\nThe reviewer is correct that inference cost is usually a blocker for industrial applications of foundation models, as large-scale deployment can eclipse even the high training costs. Nevertheless, we contend that the training cost remains a relevant consideration, and the additional attributes of ASIF can potentially position it as the preferred model.\nThe idea the reviewer suggest of merging ASIF with CLIP is very interesting. While it is beyond the scope for this submission, being able to navigate the training/inference cost trade-off is very valuable depending on the application. We will be happy to add a discussion in the paper about the inference cost vs training trade-offs and how ASIF may be an extreme point with valuable in-betweens. As it is not obvious how to do this, we would leave this open to future research and acknowledge the reviewer contribution to the discussion.\n\n“*Studying training data quality vs anchoring data quality might also be an interesting direction.*”\n\nWe agree with the reviewer and have already begun investigating a closely related avenue for our future research. Specifically, we are delving into the realm of degrading the quality of encoders within an ASIF model while maintaining acceptable performance levels. One possibility is to modulate the quality of the encoders by acting on the training dataset. We thank the reviewer for this valuable suggestion.\n"}, {"review_id": "QpF2YrKlNs", "reviewer": "Reviewer_BGFe", "summary": "This paper presents a novel approach to aligning text-image modalities without any training. The method is based on the assumption that images and their captions have similar relative embeddings, even when trained independently. By leveraging paired multimodal data, relative representations can be computed within each modality, and they can serve as a medium for cross-modality communication. The major strength of this approach is its novelty, as it removes the need for training to align different modalities of data.", "questions": "- How can you provide more convincing evidence that the features are communicable across different modalities?\n- How sensitive is the selection of text encoders?\n", "limitations": "The method has not been verified across a wide range of multimodal downstream tasks, but it is clearly stated in the limitation section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The idea is novel and has the potential to introduce a new paradigm in multimodal alignment research. Adapting to a new distribution of data is easy as it only requires a new set of coupled data without training. The concept of separating perception and interpretation can open up new research opportunities and can be applied to various multimodal problems.", "weaknesses": "- Although the approach is interesting, a more comprehensive analysis is needed to convince the readers, including a global analysis of similarities between features from text and image encoders, potentially compared with CLIP and LiT.\n- Figure 2 requires additional explanation or a clearer illustration to motivate and convince the reader.\n- Figure 4 could be improved for better understanding. The diagram is not intuitive at first, without fully reading and understanding the method section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review, we provide some brief comments on the points raised.\n\n“*More comprehensive analysis of similarities between features from text and image encoders to convince the readers*”\n\nThanks for the suggestion, in the appendix we conduct an analysis of the two feature spaces that looks at the relative distances (Fig. 7 and 10), we imagine to give it more space when presenting the work, in a future poster or blog post. Maybe the reviewer is imagining a different analysis and comparison with CLIP and LiT, we would really appreciate an expansion on this point in the discussion period, to us is not obvious how to do this given e.g. the different dimensionality of the feature spaces.\n\n“*How can you provide more convincing evidence that the features are communicable across different modalities?\nHow sensitive is the selection of text encoders?*”\n\nIn this work we demonstrated this effective communication through zero-shot and few-shot visual benchmarks as well as discussing ASIF performance in the audio domain. Still, we acknowledge that further evidence would be beneficial to the adoption of the method. In this work, we focused on introducing the new ASIF methodology but we expect to have further evidence as well as a deeper analysis on the components –including the text encoder– in future work. We also point the reviewer to [1] for a further discussion of the communication allowed by relative representations.\n\n[1] Moschella, Luca, et al. \"Relative representations enable zero-shot latent space communication.\" published at ICLR 2023.\n"}, {"review_id": "zKfPpfsbyn", "reviewer": "Reviewer_UoCi", "summary": "This paper proposes ASIF, transferring independently pre-trained image/text encoders to the classification task without further finetuning.  The proposed method only needs a small amount of paired image-text data as anchors, and represent new data samples using the relative representation to the anchor samples. The simple method achieves reasonable results on different image classification benchmarks. ", "questions": "1. In Figure 4(b), why the image representation for x* is \"absolute representations\"? This term is never mentioned in the text. \n2. In Figure 4(a), the first sample x1 goes through the image encoder and get the embedding for x1. However, for x2, why such process involves y2 (the black letter y2 under the box for x2)?\nIn general, all the figures in this work should be re-designed since current presentation is very confusing and difficult to follow. Also, the resolution of figures should be improved. ", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed approach is efficient since it does not large amount of image and text data pairs to train extra multimodal representations that align images and text. Also, the required image-text pairs can be only 1.6M, which is small comparing with original CLIP and other multimodal classification models. \n2. The needed image-text pairs are flexible. By adding more anchor samples for specific image classes, the proposed method could obtain better performances for such specific classes. ", "weaknesses": "1. Even though ASIF does not need large amount of paired image-text data, it still requires pre-trained single-modal models which are already trained on large amounts of image or text data samples. From this perspective, I didn't see the necessity of such setting. \n2. The prediction results are relying on anchor samples. If the anchor samples are not representative enough, the method will not be that useful. Imagine for a target dataset, if the provided anchor samples do not come from the same dataset or do not include similar samples as the target samples, the method may fail. If the authors can provide more experimental results to justify the influence of different chosen anchor samples, it will be more promising. \n3. The paper should be re-organized and the writing should be improved a lot. For example, the \"ASIF recipe\" can be written in an algorithm format, instead of current style. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review, we hope to have addressed all points:\n\nOn the Weaknesses:\n\n1. “*ASIF still requires pre-trained single-modal models which are already trained on large amounts of image or text data samples.*”\n\nThe crucial difference is that unimodal data may come untied and even without any labels, while captioned images are scarcer and more challenging to gather. Furthermore, the ASIF setting may be desirable for two key properties: the ability to easily edit the model and its interpretability.\n\n2. “*If the anchor samples are not representative enough, the method will not be that useful.*”\n\nWe perceive this aspect as a strength rather than a flaw. ASIF deliberately separates perception from interpretation, and only the anchors are amenable for any classification outcome. This way, we can control the visual understanding of the model by curating the limited set of anchors. Effective curation is possible given the swift iterations and the interpretability of ASIF models. And we don't need to imagine; we conducted the experiment suggested by the reviewer! In the EuroSAT experiment, we discuss the impact of adding anchors closer to the distribution of interest (penultimate paragraph of section 3 and appendix B).\n\n3. “*The writing should be improved a lot*”\n\nWe acknowledge this opinion, but the positive feedback we received on the writing, and specifically on the “recipe” format, still largely outweigh the negative feedback. Nevertheless, we want to improve the readability of the figures for the camera ready (especially Figure 4).\n\n---\n\nOn the Questions:\n\n1. “*\"absolute representations\" used in Figure 4b but never mentioned in the text.*”\n\nThanks for bringing this up. We like “absolute representations” because it contrasts with “relative representations,” but indeed, it should be introduced in the manuscript. We will add it in the camera-ready version.\n\n2. “*Figure 4a not clear*”\n\nWe see that this subfigure is prone to misunderstanding. We are considering moving all the captions in line with y1 below. In general, we plan to polish all the figures in the camera-ready version, thanks for this feedback."}, {"review_id": "A0M0jqRVVz", "reviewer": "Reviewer_2Qsg", "summary": "The paper proposes a method to align vision and language without learning a parametric model. Main idea: having a support set of image-text pairs, the structure of the visual data should match the structure of the language data. More precisely, the distances from one query image to all images in the support set (denoted as relative representations) should be similar to the distances from the associated text and all texts in the support set. For visual tasks that can be formalized as text retrieval, this method can be applied without learning. The method has relatively good performance given its simplicity.", "questions": "Q1: Can the relative representations be used for other downstream tasks, different than image classification or retrieval?", "limitations": "The authors have discussed the limitations regarding inference speed. \n\nThere is no specific negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strong points:\n S1. The method is simple and can be easily applied to potentially many vision-language retrieval tasks. Being able to use any independently pre-trained vision and language model is a big advantage. Another big advantage is that it does not require any training.\n\nS2. The model can be easily adapted by simply changing the support set of image-text pairs.\n\nS3. Some interpretability is obtained by inspecting the nearest images and captions used to obtain the relative representations.", "weaknesses": "Weak Points:\n\nW1. As the authors notice, the model is slower at training time. Although optimizations might reduce it.\n\nW2. It is not clear if this approach can be used for other downstream tasks, like VQA, or captioning. Although this might be outside of the scope of the paper it might be worth discussing.\n\nW3. The paper will benefit from more comparisons with recent vision-language models. Relevant experiments will be on image-to-text retrieval and text-to-image retrieval on COCO and Flickr30K datasets compared to methods like BLIP [A]\n\nW4. Comparing with some baselines will be beneficial. If we have a support set of the classification task, with ground truth image-label pairs a good baseline is a simple K-nearest neighbor on the same visual representation used in ASIF. For example, using the 100 image-text pairs of EuroSAT we can create a KNN relying on the same image encoder as ASIF. How does this KNN compare with ASIF in terms of performance?\n\n[A] Li, Junnan, et al. \"Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.\" ICML, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review. We appreciate the feedback and would like to offer some brief responses to the points raised.\n\n“*Can this approach can be used for other downstream tasks, like VQA, or captioning?*”\n\nWe are indeed conducting preliminary experiments to use ASIF for captioning in a subsequent work. Our current approach is centered on ASIF, combined with an LLM, to generate plausible captions for new images. This process utilizes captions retrieved by ASIF representation and then iteratively refines the crafted caption. As illustrated in Figure 6 of the paper, we expect the LLM to generate expressions such as \"a photo of a triumphal arch\" based on the retrieved captions and hints at their similarities to the caption sought.\n\n“*More comparisons with BLIP and KNN on EuroSAT*”\n\n- Thanks for pointing this out. We focused on CLIP and LiT since ASIF can be seen as a third step in the research direction traced by LiT. Nevertheless, we urge a comparison with more recent models like BLIP in future works trying to improve ASIF performance using larger datasets and further optimizations. Our primary objective here was to introduce and justify the ASIF procedure, illustrating its effectiveness on the representative task of zero-shot classification. In making this choice, we followed LiT, which used the very same datasets to showcase the benefits of a locked image encoder, that is their main claim.\n- Thanks for the valuable suggestion of making the KNN experiment on EuroSAT, we plan to make the test, expanding the discussion in appendix B. We expect a close but slightly lower performance with respect to ASIF, since ASIF can count on more images useful to perform the classification from its original training set (as seen in Fig. 7 in the Appendix).\n"}, {"review_id": "LhBog0pil6", "reviewer": "Reviewer_QV7o", "summary": "This work proposes a method to obtain multimodal representations from pretrained unimodal models and a multimodal dataset, and demonstrates the effectiveness of the method on the zero-shot classification task. The method is to map candidate images and texts into the same representation space, which consists of similarities of the candidates to each image/text in an anchor multimodal dataset.", "questions": "As stated in the section above.", "limitations": "I don't see potential negative societal impacts of this work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is very simple and intuitive, by representing an image or text as a vector of similarities to an anchor set of images/texts.\n- No gradient updates are required. One can directly benefit from pretrained unimodal models. No training parameters are involved.\n- The advantage of quick model adjustments via data handling without model retraining.\n- The method gives (to a certain extent) explainable multimodal representations.\n- The paper is clearly written and easy to understand, including a comprehensive discussion section.\n\n", "weaknesses": "- A larger multimodal dataset leads to better zero-shot classification performance, but also leads to a bigger representation vector. E.g. a 1.6M-d vector for an image or text, while manageable for retrieval tasks, might be difficult to maneuver for other downstream tasks (e.g. any generative tasks). I understand that the authors stated that effectiveness or applications to tasks other than zero-shot classification is out of the scope of this work, but I'd like to see more discussions on the limitations or impacts of the size of representation vectors.\n- The method is sensitive to the curated multimodal dataset. The authors ablated on the size of the dataset, but not the source of the dataset. E.g.what would be the effect of using a different multimodal dataset from CC? (COCO, LAION, etc?)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review; we provide some brief comments on the points raised.\n\n“*I'd like to see more discussions on the limitations or impacts of the size of representation vectors.*”\n\nThanks for pointing this out. Indeed, the large dimensionality prevents a straightforward application of ASIF in tasks like text-to-image generation. Other tasks like captioning could be approached with different pipelines (as described in the answer to 2Qsg). Still, we agree with the reviewer that the higher dimensionality trait of ASIF should be cited in the discussion, and we will add it to the camera-ready version. We also envision follow-up works tackling the problem of reducing the dimensionality of the multimodal space built by ASIF.\n\n“*What would be the effect of using a different multimodal dataset from CC, like LAION or COCO?*”\n\n- For LAION, we expect a similar performance to CC given that they almost follow the same production pipeline (scraping from the internet). LAION would be the natural next step after CC given its larger size. \n- Our very first working demo was built using a subset of COCO, which indeed worked and led us to this work. The problem with COCO is its small size (328k samples) and the relatively narrow distribution of captions: while they are much more precise than CC, they also cover much fewer concepts. CC captions are sometimes imprecise but include different captioning styles and cover a wider set of words. Still, we agree that assessing the performance with the COCO dataset is valuable, and we will add COCO to the training dataset choices in the demo notebook provided in the supplementary material."}]

Author rebuttal:
We thank the reviewers for showing keen interest in our ideas, and for their thorough and quite valuable comments. This general response summarizes the main points raised and how we have addressed them. Specific answers are then provided to each reviewer in response to their remarks.

1.  **Empirical Results and Benchmarks:** The concern regarding the limited empirical results was addressed by clarifying that our evaluation followed established protocols (same as LiT) and included additional results on EuroSAT. We also expanded on ASIF's performance on audio data.

2.  **Comparison with Supervised Encoders and ImageNet Accuracy:** We explained that our focus on open vocabulary classification makes direct comparisons with models like DEIT unfair. We emphasized the specialization and generalization trade-off in image classification, supported by recent literature.

3.  **Training Data of Encoders:** We clarified the concerns about the large training set of unimodal encoders, remarking that ASIF's distinctive capability is to rely on a modest amount of captioned images, which are more scarce and challenging to collect than unpaired and unlabeled data.

4.  **Interpretability and Inference Costs:** We provided insights into our approach to interpretability and responded to concerns about the trade-offs between training and inference costs. A commitment was made to discuss these trade-offs further in the paper.

5.  **Anchor Samples and Model Control:** The importance of representative anchor samples was discussed, highlighting ASIF's deliberate separation of perception from interpretation, allowing for control and interpretability.

6.  **Writing and Figure Clarity:** Acknowledging some concerns about the writing style and figure clarity, we committed to improving these aspects in the camera-ready version of the paper.

7.  **Potential Extensions and Sensitivities:** Responses were provided on the potential use of ASIF for other downstream tasks, sensitivity to text encoders, and comparisons with other models. We highlighted ongoing experiments and plans for future work.

8.  **Representation Vectors and Dataset Choices:** We agreed to discuss the limitations or impacts of the size of representation vectors and the effects of using different multimodal datasets. We also outlined our experience with different datasets and plans to expand on this in supplementary material.

In summary, we believe that we have addressed the main concerns and provided clarification on several key aspects of our methodology. We also appreciated the valuable suggestions for future research and will consider them in ongoing and subsequent work. Thanks once again for the thoughtful engagement with our paper.


Meta-review: The paper demonstrates aligning multimodal encoders' representation spaces can be done without training using reasonable amount of image text pairs. This is an interesting problem and this paper is the first to show this. The reviewer ratings are 3 accepts and 2 borderline rejects. In the final comments, one rejecting reviewer mentioned the paper's goal is novel and accepting it wouldn't hurt, but is afraid that the authors will not improve the paper further if the paper is accepted. The other rejecting reviewer mentioned the method is interesting and promising but has reservation about the "competitive with CLIP/LiT" performance claims in the paper. The AC agrees that the addressed problem is an interesting and important one and this paper represents an important first step, thus recommends acceptance of the paper. At the same time, the authors are encouraged to address the concerns especially from the two leaning negative reviewers to improve the camera-ready publication.
Decision: Accept (poster)