ASIF: coupled data turns unimodal models to multimodal without training
Authors: Antonio Norelli
Marco Fumero
Valentino Maiorca
Luca Moschella
Emanuele Rodolà
Francesco Locatello
Venue: Submitted to ICLR 2023 (2023)
Keywords: Representation learning
Multimodal models
Analogy
Sparsity
Relative representations

Abstract: Aligning the visual and language spaces requires to train deep neural networks from scratch on giant multimodal datasets; CLIP trains both an image and a text encoder, while LiT manages to train just the latter by taking advantage of a pretrained vision network. In this paper, we show that sparse relative representations are sufficient to align text and images without training any network. Our method relies on readily available single-domain encoders (trained with or without supervision) and a modest (in comparison) number of image-text pairs. ASIF redefines what constitutes a multimodal model by explicitly disentangling memory from processing: here the model is defined by the embedded pairs of all the entries in the multimodal dataset, in addition to the parameters of the two encoders. Experiments on standard zero-shot visual benchmarks demonstrate the typical transfer ability of image-text models. Overall, our method represents a simple yet surprisingly strong baseline for foundation multi-modal models, raising important questions on their data efficiency and on the role of retrieval in machine learning.

Under review as a conference paper at ICLR 2023
ASIF: COUPLED DATA TURNS UNIMODAL MODELS TO MULTIMODAL WITHOUT TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Aligning the visual and language spaces requires to train deep neural networks from scratch on giant multimodal datasets; CLIP (Radford et al. 2021) trains both an image and text encoder, while LiT (Zhai et al. 2002) manages to train just the latter by taking advantage of pretrained vision network. In this paper, we show that sparse relative representations are sufficient to align text and images with- out training any network. Our method relies on readily available single-domain encoders (trained with or without supervision) and modest (in comparison) num- ber of image-text pairs. ASIF redefines what constitutes multimodal model by explicitly disentangling memory from processing: here the model is defined by the embedded pairs of all the entries in the multimodal dataset, in addition the parameters of the two encoders. Experiments on standard zero-shot visual bench- marks demonstrate the typical transfer ability of image-text models. Overall, our method represents a simple yet surprisingly strong baseline for foundation multi- modal models, raising important questions on their data efficiency and on the role of retrieval in machine learning.
INTRODUCTION
Large multimodal models such CLIP (Radford et al.] 202T) are rapidly becoming the standard for foundation models (Bommasani et al. [021]
Frozen pretrained model
Trained from scratch
CLIP
LIT
ASIF
in computer vision. This is largely due to their zero-shot and open-world capabilities that enable diverse suites of downstream tasks, from classi- fication to detection and visual search. Overall, Radford et al. (2021) demonstrated that scale is the key ingredient for building common latent space for images and text, and is sufficient to con- vincingly solve a multitude of tasks without train- ing explicitly for them. In fact, CLIP's zero-shot classification accuracy on Imagenet (Deng et al.) 2009) drops from 76.2 to 31.3 when using public dataset of "just" 15M pairs (a curated subsampled version of YFCC100m (Thomee et al., 2016)) opposed to the original private dataset of 400M pairs. Recently. Zhai et al. (2022) showed improvements in data efficiency by only training the text encoder, achieving Imagenet accuracy comparable CLIP with 10M samples and outperforming with larger 901M data set.
Image
Image
Image
Figure 1: ASIF simple recipe to align the representations of two frozen pre-trained mod- els exploiting the fact that relative distances are preserved across different modes: the captions of similar images are themselves similar.
Training models at such scale presents several challenges beside the obvious infrastructure and train- ing costs. Notably, it requires collecting massive training sets, making it difficult to interpret the predictions of the model in light of their training data. Additionally, the training assets are often not owned by the institution training the model (Sun et al.) (017). This introduces several additional challenges, from reproducibility to the difficulty of ensuring that an asset owner can remove their
data from the model (Golatkar et al.) 2021 2020b all, these considerations make large multi modal models relatively inaccessible to researchers and practitioners until checkpoints are released access to demo is granted.
Ginart et al., 2019: Guo et al.) 2020). Over-

---

Under review as a conference paper at ICLR 2023
rr(x))
12
rr (1))
E
11
32
rr (1)
rr rr(2")
rr(ū))
922
E2
3/
rr(O)) 91)
rr
y1
2
Figure 2: The ASIF construction. An ASIF model defined by two unimodal pretrained encoders and collection of coupled embeddings (in turquoise). This is sufficient to compare elements from different modes through their relative representations: rr(ū;) is more similar to rr(x") than IT(
In this paper, we present ASIF, a simple procedure that turns pre-trained uni-moda image and text models into multi-modal model using relatively small) maatimuliiinnn m modal data set and no additional training as shown in Figure [ The resulting model aligns latent representations of images and text, behaving if it was contrastively trained on multi-modal data like CLIP or LiT. The key intuition that captions of similar images should be themselves similar, and therefore a representation crafted using just similarities to ground-truth tnuth multimodal pairs is quasi mode-invariant.
Our results are surprising and raise several questions. Despite (I) the simplicity the approach, (2) multi-modal data set that is up to 250 times smaller than in prior work and (3) the lack of actually training the model on multi-modal data, ASIF achieves zero-shot classification accuracy on downstream data sets that is roughly in the same ballpark of CLIP (Radford et al. 2021 Zhai al.
2022). This raises important questions on the data efficiency in foundation models, making ASIF very powerful and cheap baseline for future work, and opening new doors for data centric Al (Ng 2022). In fact, ASIF comes with several interesting properties by construction. The absence of train- ing makes the model editable: adding or removing image-text pairs and deploying a new multimodal model is a matter of seconds. Moreover, the representations are highly interpretable, since every di- mension corresponds to the similarity of the input unique entry in the multimodal dataset.
In summary, we:
Introduce the ASIF procedure, which turns two pretrained unimodal black-box encoders into an interpretable multimodal model without tuning neuron and using "limited" amount of multimodal data.
Demonstrate the effectiveness of ASIF models zero-shot image classification tasks, where they achieve performance in the same ballpark of CLIP with significantly fewer image-text pairs.
Discuss key properties of ASIF, its implications on the role of memory and retrieval in machine learning, and the new opportunities it opens.
ALIGNING
MODELS VIA RELATIVE REPRESENTATIONS
In the following present how collection of captioned pictures implicitly defines common space for images and texts through relative representations, allowing to build a multimodal model without training. Before that, we briefly discuss existing techniques build this common space.
'Compared to the data sets used train statc-of-the-art multimodal models, our experiments use 1.6M captioned images.
2

---

Under review as a conference paper at ICLR 2023
Multimodal dataset
Test sample
211
Image encoder
Ey(ey)
Text encoder
So(Bifa R
rm
8c( () Bi()))
J1P"
0
.4
FT (T(x*t
UNKNOWW IDEAL CAPTION
.11
.09
-14
54
IT( = II(x*)
tounsts visit a triumphal arch
.99
.57
rr rr(
rr rr(1))
a sunryy day the old city
Ealie
49
-06 07
87
E((0))
Edv))
Eg(ga)
Figure 3: Zero shot classification. The best caption identified by picking the one with its pro- cessed relative representation fī((;)) closest the input image fr(="). In other words, we consider the relative representation of x² with respect to the image collection Ti...., as if it was the relative representation y" the ideal caption for x with respect the corresponding caption collection y Vn-
Contrastive training to build common space. With multimodal models, we refer to architec- tures that embed inputs of diverse modes into the same space. The better is a multimodal model, the closer representations of different modes of the same object. So far, this common space has been obtained as the result a contrastive training of one (Zhai et al.) 2022) or both the neural mode encoders (Radford al.) 2021 Jia et al. 2021). Using e-g. collection of image-tex, pairs as training set, a contrastive loss promotes the closeness image and text embeddings from the same pair, while spacing to match the image embeddings of a pretrained visual encoder. Once the encoders are trained, a multimodal model can be adapted to perform any visual classification task just by crafting caption for each label and selecting the one with the embedding closer to the embedding of the input image.
the ones from distinct pairs. Zhai al. (2022) train just the text encoder
Relative representations. Our idea to build a common latent space is use a collection of coupled data as a "rosetta stone" between modalities, and represent each new data point as its similarities to the points of the same modality in the collection. other words, we compute a relative representa- tion for each new data point:
Definition 2.1 Given an encoder E : X + Rd and subset of samples (x1...., In} denoted as anchors, we define the relative representation of x! as the n-dimensional vector:
rr(x') (sim( (F(x'). E(x1)), simm ((((F(()). E(xa))
for some similarity function sim, such as the cosine similarity.
Relative representations of points from different modalities live in the same latent space if they are with respect to the same multimodal collection, see Figure? 2
Relation with Kernel methods. Definition 2.1 may not look surprising to readers familiar with the literature on kernel methods (Hofmann al. 2008). Instead of presenting rT(x') as kernel, we say it is a relative representation to stress that (1) we want to explicitly represent the coordinates in our ASIF recipe as opposed to operating in an implicit feature space and (2) we do not aim at learning regression parameters, although we note this may help with the inference speed. Instead, we rely on a simpler procedure that may be interpreted as a hard version of the Watson -adsaanNadaraya (Nadaraya) 1964 Watson 1964) regression with distance threshold. Other benefits from incorporating kernel methods explicitly are left to future works, as our work focuses on demonstrating how single-domain pre-trained networks can be aligned without further training.
3

---

Under review as a conference paper at ICLR 2023
relative representations inducing meaningful common space. Consider the embedding spaces of any two good text and visual encoder we expect captions images that are close in the visual space to be themselves close in the language space. This fact makes representation defined in terms of similarities against a set of ground-truth multimodal pairs almost mode-invariant, í.e. an image and its caption share almost the same representation.
ASIF:
That is why we can assign the best caption to a new image x" just by performing nearest neighbors: can consider the relative representation of x* respect to the image collection (2 In,) as if it was the relative representation of its ideal caption y² with respect to the counterpart collection
Ly....... Vn). see Figure 3
The whole procedure to set up an ASIF model and use it to find the best caption for new image follows.
ASIF recipe. Ingredients:
Two good encoders, each mapping single data modality to vector space. Let X and Y be the mode domains, for instance a pixel space and a text space, we need pdd
Ej X + ddl and Eg
A collection of ground truth multimodal pairs: D = ((1,,81)..... (Zns vn)}, for instance captioned images.
Procedure to find the best caption among set of original ones y = (01,... de} for a new image x":
1. Compute and store the embeddings of the multimodal dataset D with the en- coders E,, Eg and discard D. Now in memory there should be just DE = {(Er(11), E2(1)) ..... Ev(v)))
2. Compute the n n-dimensional relative representation for each candidate caption (r;))) (sim(E2 (y))),.... sim( E2( E2(yn)), where sim similarity function, e.g. cosine similarity. Then for each rı(ū;) set to zero all dimensions except for the highest k. and raise them to 2 1. Finally normalize and store the processed c vectors fr((:)) Choose k and p taste, in experiments & 800 and p = 8;
3. Compute the relative representation of x" using the other half of the embedded multi- modal dataset De and repeat the same processing with the chosen l: and p:
4. We consider the relative representation of the new image x² as if it was the rel-
ative representation of its ideal caption y", i.c. we define rī(y") = ri(x". So we choose the candidate caption ÿi most similar the ideal one, with i argmax, (emmaxx(sim{(e (f(im(f((y")))... (():))).
To assign one the captions to a different image x** repeat from step 3.
Properties of ASIF models. The above construction yields several intriguing properties for free:
No training and "data centric". As we have seen, an ASIF model built on top of two indepen- dently pretrained encoders and the embeddings of a multimodal dataset, and so without training or finetuning any neuron. Being deployable or updatable in seconds, an ASIF model radically "data centric" (Ng) 2022). For example, it is trivial to adjust the model by adding or forgetting specific samples. The latter use-case particularly important, as the right to use specific assets may change over time and removing the effect of specific samples from trained network requires sophisticated forgetting techniques, e.g. Golatkar et al. 2021 2020b a); Ginart et al. 20019; Guo et al. (2020). In our procedure, the encoders can be pre- trained with established data sets that do not change over time, and removing the effect a multi-modee example is as simple as deleting its embeddings.
Data efficiency: Being able exploit two pretrained encoders, ASIF models require far less ground- truth multimodal pairs to become effective. As confirmed by our experiments, ASIF reaches com- petitive zero-shot performance diverse classification benchmarks by using fraction of the mul-
Vision and language are the only modalities we target in this paper due to wider availabilty of models and paired data, although expect the procedure he more general.
4

---

Under review as a conference paper at ICLR 2023
timodal data of its predecessors, reaching a respectable accuracy with thousands of pairs (we refer to Section 3 for more details). This is in line with classical work in computer vision, where prototypical networks (Snell et al. 2017 are strong bascline in the extremely few-shot regime.
Interpretability: Sparse relative representations make the ASIF models interpretable classifiers. In fact, we can trace back every prediction corresponding to the dimensions that are nonzero both in the image and the caption-label- responsi- ble for the outcome most k), see Figure[5 This enables visualizations of the relevant samples con- tributing to a correct or incorrect prediction at no extra cost, in stark contrast with other approaches that are often costly (Achille al. 2021) and fragile (Koh Liang 2017 Basu et al. 2021).
small set of data points in the multimodal dataset
Key limitation Implementation that scales, Clearly, our method pays the price of avoiding training with scalability at inference time. As such, our approach should not be considered general one-stop replacement for CLIP, although in our experiments we could scale ASIF to 1.6M pairs with reasonable inference speed (non-optimized ASIF nearest neighbor procedure made inference speed less than 2x slower). If someone wanted to truly deploy scalable implementation of ASIF, there are several components that facilitate scalability. Having just nonzero entries, relative rep- resentations can be implemented using sparse vectors, with cosine similarities computed quickly in batches through sparse matrix multiplication. Relative representations can be constructed and com- pared efficiently even when the multimodal dataset scales up to many millions of entries, thanks to established techniques such as product quantization (Jegou et al. 2010) and inverse indexing (Sivic & Zisserman 2003), both implemented e.g. in the FATSS library (Johnson et al. 2019). Finally, find that the distribution of pairs chosen during inference is rather short tailed, presenting opportuni- ties to prune the model even at deployment time, deleting from memory the data that is never used. Other more advanced techniques like coresets (Bachem et al. 2017) are also possible.
2.1 DESIGN CHOICES AND IMPLEMENTATION OF ASIF MODELS.
Curating the multimodal dataset. While neural models like CLIP or LiT are defined just by the weights of the two encoders, univocally identify an ASIF model we need also the embeddings of the multimodal dataset. Even if two image-text ASIF models rely on the very same encoders, they comply to different visual understandings of the world if they are based on different collections of image-text pairs, therefore achieving different performances on the same tasks.
In contrast to traditional neural vision models, the ASIF approach makes an effective curation of the training dataset possible, since it works even with very small multimodal datasets and allows to evaluate the impact of every data point, as seen in Figure(5)
Salient byperparameters. While using the raw relative representations already provide an ASIF multimodal model with non-trivial capabilities, we found that two simple treatments greatly improve performance, efficiency, and also foster interpretability.
(i) Sparsification. We set to 0 all the entries of the n-dimensional relative representation except for the top k. In our experiments and k are respectively in the order of millions and hundreds. In this way we cut off the small noisy signals from the dissimilar entries, that accumulated during comparisons would destroy the signal from the few similar entries. Furthermore we get highly interpretable representations that can be efficiently compared, since we have just & nonzero features, each one linked to single entry in the multimodal dataset.
(ii) Exponentiatton. We raise all the nonzeroed similarities (im(E(²), E(x,)) to p. with p 2 This non-linearity weighs more the contribution of the most similar entries in the relative representation.
Besides the pivotal choice of the ground-trute multimodal pairs, the number of non-zero elements and the exponent p are the salient hyperparameters to consider when deploying ASIF model. In general, we found that picking a p * 1 may help, while choosing a k & n is always crucial. For more details see Section 3
2.2 CLOSELY RELATED WORKS
Retrieval augmented foundation models. In (Rota 1985), Stanley Ulam affirms that mathe- matical formalization of the word "as" -on a par with the connectives "and", "or", "implies" and "not"" would be key milestone artificial intelligence. The idea of having analogies at the core
5

---

Under review as a conference paper at ICLR 2023
of cognition is shared by Hofstadter (2001) which states that a concept is package of analogies, coherently to what prescribes the ASIF procedure. This inductive bias can be implemented in ma- chine learning considering relational information and retrieval augmented models. Recent works in NLP enhance unimodal language models with retrieval to reduce the size of the architectures and the training dataset while also making results more transparent (Borgeaud et al.] 20222] Izacard iet al. 2022). Our work is in line with this view, that the ASIF procedure extends to the multimodal case. Importantly, ASIF offers a new perspective on data centrie AI (Ng 2022), where data and the external memory implement the alignment between modalities. Networks with discrete Key-Value bottlenecks (Träuble et al. 2022) are closely related to our approach, with the critical differences that our memory is not learned and that their decoder is trained for classification. Key- Value bottle- necks as in (Träuble et al. 2022) could be good candidates to implement a summarization of the data set that is learned and of fixed size. Finally, we notice that making predictions on new samples by exploiting the similarity with a dictionary of previous data points is a very common approach in com- puter vision (Snell al. 2017) for few-shot learning. Our procedure is also related to compressed sensing algorithms where a signal (in our case an image) is sensed as a sparse combination of fixed atoms (Wang al. 2012 Mallat & Zhang, 1993) with an iterative projection procedure (Locatello ct al.
2017 2018) and only transmitting the coefficients to the receiver (text modality).
Learning multimodal models. Following the intuition outlined by early works on aligning text and image embeddings (Frome et al.] 2013) Karpathy & Fei-Fei (015), today large multimodal models are conquering the computer vision scene thanks to their wide applicability and easy transfer to new downstream tasks (Radford et al. [021]] Zhai et al., 2022 Jia et al., 2021 Yu et al. 2022). We iden- tify two key leaps respect traditional models like ResNet (He al. 2016): (i) Free text allows to learn visual concepts beyond finite set of predefined categories and to exploit the structure of lan- guage to compose them, as masterfully seen in Dall-E (Ramesh et al. 2022). (ii) The recognition tag transitioned from being output pulled out of the image by the neural stack (the label) to become an input that should be interpreted, and therefore processed by its own encoder (the free text caption). This corresponds to an epistemological perspective shift, as discussed by NNoelli et al. 20222)
Data and learning efficiency are clear challenges for large multimodal models, that often require hundreds of millions of examples. Efforts such as (Zhai et al. 2022 Izacard et al.) (022) attempt to reduce this. ASIF presents a different take on this problem, showing how much can be achieved by simply remembering the training data efficiently.
3 EMPIRICAL EVIDENCE
In the following we compare ASIF to traditional multimodal models based on contrastive training, CLIP and LIT. We then take a closer look to the classification of a single image, unpacking the rela- tive representations and following the classification algorithm step by step. As a prelude the above, we start by discussing the pretrained encoders and dataset forming the ASIF models we tested.
Pretrained encoders and multimodal dataset used. In our experiments we used visual trans- formers as image encoders, either trained in a supervised (VITb16, Dosovitskiy et al. (22TT) or unsupervised way (DINO VITs8, Caron et al., 202I). Both transformers were pretrained on Ima- genet 21k (Rddikk et al. 202T). In the supervised case, the labels were used during training, while in the unsupervised case, the model learned to represent two views of the same image in the same way. The embedding size is 768 for VITb16 and 384 for DINO VITs8. Note that this is the same setup as in Zhai et al. (2022). The only difference with Zhai et al. (022) is that they train the text encoder and we use frozen pre-trained one. Concerning the text encoder, we used the SentenceT pretrained transformer (Reimers& & Gurevych, 2019). Its training dataset consists of more than bil- lion sentences scraped from the internet. Here there multiple learning tasks used for pretraining, including a contrastive one with sentence pairs. The embedding size of SentenceT is 768.
As multimodal dataset we used the first 1.6M entries of the Conceptual Caption dataset (CC12M, Changpinyo et al. 2021), which collects images and filtered alt-texts scraped from the internet.
Zero-shot performance. We assessed the quality of our ASIF multimodal model by comparing its zero-shot classification performance against CLIP and LIT on four datasets: CIFAR100, Imagenet, Imagenet v2, and PETS; see Table We crafted label prompts as in LIT (Zhai at. 2022 Table 11).

---

Under review as a conference paper at ICLR 2023
Method
Dataset size
ImageNet
CIFAR100
Pets 88.9
ImageNet v2
CLIP (Radford et al.
2021)
400M (private)
68.6
68.7
CLIP (Radford et al.
2021
15M (public)
31.3
LIT (Zha et al. 20222)
10M (public)
66.9
CLIP (Zhai et al.
2022
uu)
901M (private)
50.6
47.9
70.3
43.3
LIT (Zhai et al.
2022)
901M (private)
70.1
70.9
88.1
61.7
ASIF (sup vis. encoder)
1.6M (public)
55,4*
63.3
71.5
45,6
ASIF (unsup vis. encoder)
1.6M (public) 53.0"
46.5
74.7
45.9
Table 1: Zero shot classification accuracy of different multimodal designs. CLIP and LIT im- plementations vary by dataset and the visual transformer used as image encoder. The first CLIP and LIT entries use VITbl6 as ASIF, the last CLIP and LIT entries use VITb32 (larger patch size). The public dataset of CLIP is a curated subset of YFCC100m (Thomee et al. 2016), while LIT and ASIF use CC12M. *We used the ImageNet validation set tune the two hyperparameters of ASIF which were then used on the other data sets. The number reported in the table is a test set.
Remarkably, we achieve competitive performance with CLIP and LIT using two frozen pretrained encoders and fraction of the image-text pairs.
In Figure we show the full zero-shot accuracy trend on Imagenet as the size of the multimodal dataset grows for different choices of k and p. ASIF models become effective very quickly: reach a non-trivial 18% classification accuracy using just the first 10,000 pairs in the multimodal dataset. We recall that building an ASIF model from scratch still requires a lot of unimodal data to pretrain the encoders, but now this data may come untied and even without any label.
In Table we also report the zero-shot ac- curacy of two ASIF models on four standard datasets, differing just for the vision encoder, that is respectively supervisedly and unsuper- visedly pretrained (DINO). We tuned & and p on the ImageNet validation set, in both cases we used k 800 and 8.
nonizeros: 50, exp:
Imagerettoll zerp-shot accurscy
nonizeros: 800, val
nonizeros: val exp:
non-zerm:: 1600, vol exp: 1
1600,
The comparison between the two versions not completely fair since the visual trans- former architecture of DINO is smaller (c.g. the codings are 384-dimensional rather than 768) but corroborates the effectiveness of ASIF with encoders pretrained using no su- pervision.
Size the multimodal cieteeet
Figure 4: Impact of (non-zeros), p (val exp), and
dataset size on ImageNet zero-shot classification.
Summary: Overall, we observe that our ASIF procedure can achieve competitive zero-shot results with fraction of the image-text pairs used in prior work (assuming access to other pre-trained, even unsupervisedly, uni-modal models).
Deep dive into a classification. To better understand why and how the ASIF procedure works, we going follow step by step the single classification process depicted in Figure that shows the entries of the multimodal dataset used to build the relative representations. For simplicity we assume 23.
We want to find the imagenet label of the upper left image in Figure 5 The first step is to compute its relative representation with respect to the multimodal dataset, this procedure selects the 23 most similar images in our collection. The most similar are indeed triumphal archs that -as similarity decreases- turn to generic structures with archs or ancient monuments. No other image in the mul- timodal dataset is involved in the classification, we could replace all the entries in our collection except for these 23, and as long as no new image is more similar to the input image, every computa- tion in the classification process would remain the same. This consistency of the ASIF procedure is in contrast to traditional approaches like CLIP and LIT, where the parameters defining the encoder computations shaped by the whole multimodal dataset through contrastive learning.

---

Under review as a conference paper at ICLR 2023
Lagg SEDOO Woungiaa offec sall in Doogge
Query image x
Therarch Vallo Page
Moot oineter images the multinoia diitioed
heodintical Archiedtre
2.64
Paittng- tiruugh
0.80
0.81
0.2
tetaoraa anc ul Paaryre
-TA photo of
stumphal oroly
rTA photo of
Arch of
80
0.77
inooquo)
landrork
Correspondnng capdon
mn/
Sonvra Rome
0.81 ofth Taus Rome
o0r
0.89
0.52
andihe AATRSON
0.82
1024
The Arch HFEESOO
SPrMSSUR Romen Porum flome
unia shancl SPENSOK
0.8
0.22
plctum ore ofttre kande worki
And of Comtent tnly eiice sholograhyy
Anc of This
Mosicu Dirmsssuu
Bewwerto
0.59
0.03
2.51
The Kasha Trojer
your
0.81
00:00
0.89
Bluck ur whilie the
zumphy undrof Valiaal a cowing
Trturss Auch -PCiSSoe
2006
0.00
0.00
0.60
0.03
Valley oftt Kingo
Figure 5: Deep dive into classification. We show step-by-step how ASIF produces predictions for a query image in an interpretable way. By construction, ASIF allows to visualize the samples leading to correct incorrect predictions which is useful to curate better data.
Now is time to perform the core step of the ASIF procedure: jumping in the text space by con- sidering the relative representation of the input image as if it was the relative representation of its ideal caption. So find ourselves with the same vector, but now meaning how much the ideal caption should be similar to 23 captions. Here the quality of the multimodal dataset matters; looking the 23 captions we are confident that the candidate caption closer match the 23 similarities is one of the prompts associated to the t i wriiumpallaaae triumphal arch". ". Indeed, bad multimodal dataset spawning 23 non-informative captions like file- names "IMG 201 20/80823,pp 80823.jpg" or camera settings "D90 18.0-70.0 nmn f/3.5-4.5" would have com- pletely failed in recognizing the correct class, even if supported by the best image and text encoders. Such a catastrophic failure is feature of the ASIF procedure, since it follows that the sole mul- timodal dataset give us full control of the model: the two encoders may have seen anything, what matters is the meaning attributed to it by the multimodal dataset.
label in ImageNet, like "a photo of
Summary: Our simple (non-cherry picked) example showcases how the ASIF predictions can be easily attributed to specific examples in the training data by construction. This feedback can be used to explain predictions and grow high quality datasets in data centric AI, for example by inspecting which examples contribute to incorrect classifications.
4 DISCUSSION
The effectiveness of the ASIF procedure raises questions the role of memory and retrieval in machine learning, while at the same time opens new opportunities for products based on multimodal models, opening many avenues for future works. the following we will discuss these aspects.
Perception and interpretation disentangled. ASIF redefines what constitutes multimodal model by explicitly disentangling memory from encoding: here there is no trace of the multimodal data in the weights of the encoders, which are pretrained on different datasets. Nonetheless, the relative representations and the outcome of any classification task fundamentally depend on the multimodal dataset. This state of affairs reflects the factorization of perception and interpretation in the two stages constituting an ASIF model; the encoding and the construction of the relative representations. Such factorization may be desirable, since it relieves the neural encoders from the responsibility of attributing meaning to their inputs. Therefore allowing us to "keep our nose out of the black boxes" Eco 2000, Par. 3.3.1.3) and only consider the second stage to analyze, explain, and act the interpretations of an ASIF model. Similarly to when -to teach the interpretations at the basis of a discipline- modify the schoolbooks rather than directly adjusting the synapses of the learner.
Learning or retrieval? As we have seen, the ASIF procedure requires no training: it does not distill the multimodal data into any learnable parameter. Rather, it preseribes rigid memorization of the
8

---

Under review as a conference paper at ICLR 2023
multimodal dataset, where each entry has its fixed-size spot, similarly to retrieval process. On the other hand it seems impossible to not describe ASIF as a learning algorithm; for instance satisfies the fundamental characterization given by Mitchell (1997): the more the multimodal data the more ASIF improves, as we can clearly see e.g. in Figure Ultimately, ASIF is functionally comparable to CLIP. ASIF blurs the border between learning and retrieval by questioning the effectiveness of storing information only in the weights and rather advocates to combine learning representations with external memories. We encourage more work on memory augmented neural networks and towards understanding the implications of memory for generalization.
Generalization to new distributions. The empirical performance of ASIF calls for a discussion on zero-shot and out-of--isttie -iitrri--ttiittiinnn generalization in foundation models trained with huge data sets. Clearly, the performance of ASIF will depend strongly on the multi-modal data used for alignment. As an example, the good performance on Imagenet may not be particularly surprising in light of the qualitative evaluation seen Figure 5 There, our query image might as well had been part of our multi-modal data set, the semantic gap with its neighbours appears to be small. Despite this, our choice of downstream evaluation and pre-training data set is identical compared to prior work (Zhai et al. (022). As such, while it appears clear that ASIF should fail when the semantic gap between downstream task and "training data" is large, it is unclear why it should be different for more standard models like CLIP (Radford al. [021] and LiT (Zhai al., 2022): if a gap does exist, future work should work to address it. In the meanwhile, we recommend that future foundation models are benchmarked on significantly broader sets of downstream tasks, idcally with some analysis of the semantic distance between test and training data (if any). Alternatively, our results may be interpreted in light of the strong performance of uni-modal models. There may be a limited benefit of training from scratch on less curated multi multi-modal data sets compared to standard and well established uni-modal data sets, although we posit that at even larger scales the difference may be more significant.
Uncertainty. Another intriguing consequence of an explicit memory that it enables trivial uncer- tainty estimations. If none of the k nonzero dimensions of the relative representation of a new image is used to represent any label, all similarities would be null (or smaller than some threshold) and the model could be instructed to output unknown token. We posit that this property is not specific to ASIF and that it could be generically implemented by many memory augmented networks, pro- viding new interesting perspectives the problem of estimating the prediction confidence on new, perhaps distri--iitiittionnn data.
The biggest limitation of ASIF is the inference cost when scaling the largest data sets. Additionally, while the simple ASIF procedure presented here offers a strong baseline for mul- timodal models, its performance still falls apart to CLIP and LiT when the mutimodal dataset is abundant and the cost of training is not concern. At the same time we recognize that the exper- iments reported here are far from broadly covering the multitude of downstream tasks multimodal models are recognized to be useful to solve. Our scope here was to present the ASIF procedure and provide evidence of its effectiveness on the representative task of zero-shot classification that are common in the literature. In making this choice we followed (Zhai et al.) 2022), that used the very same datasets to showcase the benefits of a locked image encoder, that is their main claim.
Limitations.
Conclusions. We presented simple procedure called ASIF to assemble fully functional multi- modal model like CLIP from two unimodal pretrained encoders and a collection of image-text pairs without tuning a single neuron. While achieving competitive zero-shot classification results with its predecessors using a fraction of the data, the proposed ASIF procedure enriches multimodal models with editability -a new model based on different pairs can be deployed in seconds- and interpretable representations. The effectiveness of the ASIF procedure also clashes with the dominant view of a learning algorithm as a way to distill data into the parameters of model, and raises questions on the role of memory and retrieval in machine learning.
REPRODUCIBILITY STATEMENT
We describe in detail the ASIF procedure in the gray frame in Section All the encoders and the datasets used in our experiments are publicly available. We have reported all the hyperparameters to fully reproduce the results in Table T the selection procedure, and the trends in Figure 4

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alessandro Achille, Aditya Golatkar, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Lqf: Linear quadratic fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15729-15739, June 2021.
Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint arXiv: 1en::0103300000 2017.
Samyadeep Basu, Phil Pope, and Soheil Feizi. Influence functions in deep learning are fragile. In International Conference on Learning Representations, 2021.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2 2021.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George van den Driessche, Jean- Bann-Btttstte Lespiau, Bogdan Damoe, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick. Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Im- proving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Ste- fanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Muchine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 2206-2240. PMLR, 2022. URL https::/pprocediin .mr.i.smm......mmm. press / wwtpe///oooeeeeeeeeeeeeeeeeeeeee html
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representatiinss ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. Ed=YicbFdNTTy
foormm?
Umberto Eco. Kant and the platypus: Essays on language and cognition. HMH, 2000.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. Advances in neural infor- mation processing systems, 26, 2013.
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9304-9312, 2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In European Conference Computer Vision, pp. 383-398. Springer, 2020b.
10

---

Under review as a conference paper at ICLR 2023
Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten, Certified data removal from machine learning models. In International Conference Machine Learning, pp. 3832- 3842. PMLR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Thomas Hofmann. Bernhard Schölkopf, and Alexander J Smola. Kernel methods in machine learn- ing. The annals of statistics, 36(3):1171-1220 2008.
Douglas R Hofstadter. Analogy as the core of cognition. The analogical mind: cognitive science, pp. 499-538, 2001.
Perspectives from
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv: Arxin-2208.03299, 2022.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions pattern analysis and machine intelligence, 33(1):117-128, 2010.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. JEEE Transactions Big Data, ((3):535-54)) 2019.
Andrej Karpathy and Fei-Fei. Deep semantc- alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128-3137, 2015.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1894-1894. PMLR, 2017.
Francesco Locatello, Rajiv Khanna, Michael Tschannen, and Martin Jaggi. A unified optimization view on generalized matching pursuit and frank-wolfe. In Artificial Intelligence and Statistics, pp. 860-868. PMLR, 2017.
Francesco Locatello, Anant Raj, Sai Praneeth Karimireddy, Gunnar Rätsch, Bernhard Schölkopf, Sebastian Stich, and Martin Jaggi. On matching pursuit and coordinate descent. In International Conference on Machine Learning, pp. 3198-3277 PMLR, 2018
Stéphane Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. IEEE Transactions signal processing, 41(12):3397-3115 1993.
Tom Mitchell. Machine learning, volume 1, 9. McGraw-hill New York, 1997. ISBN 0070428077.
Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications, 9(1): 141-142, 1964.
Andrew Ng. Unbiggen ai. IEEE Spectrum, 2022. URL https://pperrun.ieet org/
Antonio Norelli, Giorgio Mariani, Luea Moschella, Andrea Santilli, Giambattista Parascandolo, Simone Melzi, and Emanuele Rodolà. Explanatory learning: Beyond empiricism in neural net- works. arXiv preprint arXiv:2201. 10222, 2022.

---

Under review as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763. PMLR, 2021.
Adilya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv: Arxiv-2204.06125, 2022.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2019. URL https:
org/ org/abs/ 1908. 10084
Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik. Imagenet-21k pretrain- ing for the masses. In Joaquin Vanschoren and Sai-Kit Yeung (eds.). Proceedings of the Neural Information Processing Systems Track on Dalasets and Benchmarks
1. NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
00eol00633b6ee -attGa3336A-bbttaccceaa roundl html
Gian-Carlo Rota. The barrier of meaning. Letters in Mathematical Physics, 10(2):97-99, 1985.
Sivic and Zisserman. Video google: text retrieval approach to object matching in videos. In Proceedings Ninth IEEE International Conference on Computer Vision, pp. 1470-1477 vol.2, 2003. doi: 10. 2..110//CCCV.2000123366666
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for feo-shot learning, In Is- abelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4077-4087 2017. URL /ttt:s://proceediigggg neurips......................... da/ F28 2oce Abst .html
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017.
Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec 100m: The new data in multimedia research. Communications of the ACM, 59(2):64-75 2016.
Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Mozer, Kenji Kawaguchi, Yoshua Ben- gio, and Bernhard Schölkopf. Discrete key-value bottleneck. arXiv preprint arXiv:2207.112200 2022.
Jian Wang, Seokbeop Kwon, and Byonghyo Shim. Generalized orthogonal matching pursuit. IEEE Transactions on signal processing, 60(12):6002-6216. 2012.
Geoffrey S Watson. Smooth regression analysis. Sankhya: The Indian Journal of Statistics, Series A, pp. 359-372, 1964.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung. Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint 2rXiv:2005.01917, 2022.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18123-18133, 2022.
12


Reviews:
[{"review_id": "q8MtdOd31t", "reviewer": "Reviewer_hNdq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes ASIF, which can retrieve relevant captions given an image **without training**, utilizing 1) two pre-trained unimodal encoders and 2) a relatively small multimodal datasets. The key intuitive is that “captions of similar images should be themselves similar”; thus the retrieval is done through the relative distances given by the two pre-trained unimodal encoders. \n\nIn experiments, the authors show decent performance on several image classification datasets, using conceptual captions as the multi-modal dataset.", "summary_of_the_review": "I very much like the intuition of ASIF. It works decently in practice and has several nice properties nicely argued by the authors.\n\nHowever, my main concern is that the method is very similar to KNN and once we consider it as some kind of KNN, the analysis and results look less “surprising”.", "strength_and_weaknesses": "Pro: It it interesting to see such an effort on tuning unimodal encoders into multimodal models using non-parametric methods. \n\n---\n\nCon: clarification needed on relationship with respect to K Nearest Neighbor.\n\nAs noted by the authors, ASIF, in essence, is “making predictions on new samples by exploiting the similarity with a dictionary of previous data points”. This makes ASIF extremely similar to KNN. The first half of ASIP, i.e., finding the k most-similar images, looks the same as KNN. What differs is how to find the best caption given the k “neighbor” image-caption pairs. I would imagine a naive approach of finding the caption from the training set that is mostly similar to k captions (using Language Model similarity) would do pretty well (similar to the majority voting in KNN). The author also provides a nice discussion in Sec 4 “learning or retrieval”. However, if we link ASIF to KNN, then it would be clear that ASIF, like KNN, counts as a non-parametric supervised learning algorithm. \n\nIn fact, KNN has been used for evaluating self-supervised visual representation models (e.g., Table 1 of Zhou et al.) and the performance is already high (78.0). But I can see that the performance is not directly comparable as Zhou et al. used ImageNet as the KNN training set while this paper uses Conceptual Captions (CC). The impressive results in the paper might look less “surprising”, if we were to believe that ImageNet and CC have a lot of overlap. Then the current results would be somewhat a domain-shifted version of the KNN experiments in Zhou et al.\n\nI would thus appreciate a discussion on:\n\n 1) what makes ASIF different from and potentially better than KNN (and similar methods);\n\n 2) experiments on a naive KNN baseline on using CC data for ImageNet. \n\nZhou et al., iBOT: Image BERT Pre-Training with Online Tokenizer\n\n---\n\nMinor questions:\n\nFor the image classification experiments, ASIF will select a caption from the 1.6M captions. How should we map the retrieved caption to a label? Is it a simple rule-based ngram match?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cB_wy99zfzL", "reviewer": "Reviewer_MKVK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a multimodal model that does not require any training on image-text pairs. The proposed non-parametric model, ASIF, leverages independent pretrained unimodal models to extract embeddings of data points in ground-truth image-text pairs. At inference, ASIF first computes the relative representation of a given input from one modality as the vector containing similarities between the given input and each data point from the dataset. It then treats this vector as the relative representation of the input's ideal correspondence from the other modality, and retrieves the candidate that has the most similar relative representation to the vector. With some sparsification treatments, ASIF achieves quite competitive zero-shot results on several image classification benchmarks, including ImageNet, ImageNet v2, CIFAR100 and Pets, using a relatively small paired dataset.", "summary_of_the_review": "The proposed method is simple yet quite effective for building a multimodal non-parametric model, but one of the key contributions, data efficiency, should be validated with unimodal models that are trained on small-scale datasets.", "strength_and_weaknesses": "Strength\n- The paper is well-written and throughly addresses related work and implications of the proposed method.\n- The proposed method is simple yet quite effective for multimodal zero-shot tasks using a relatively small paired dataset.\n\nWeaknesses\n- However, it seems that the model performance heavily depends on pretrained unimodal models, and the results reported in the paper come from the unimodal models that are trained on large-scale unimodal data: ImageNet21K (about 14M images) and 1B sentences scraped from the internet. I don't think that it is appropriate to say that ASIF is data efficient using such models. This seems to be more than the LiT training data. Do you have additional results using unimodal models that are trained on less data, such as ImageNet1K?\n- Since the model performance also heavily depends on the ground-truth image-text pairs, you need to provide quantitative results on out-of-distribution tasks. I am not sure which dataset among ImageNet (or ImageNet v2), CIFAR100 and Pets is far distant in distribution to the subset of CC12M that you use for computing relative representations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "VRTfoSULOsm", "reviewer": "Reviewer_nLjp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an interesting idea of turning two unimodal pre-trained models into a multi-modal model without training. Specifically, each sample is encoded into a relative representation with respect to paired multi-modal data in its own modality. The relative representation serves as the bridge between two modalities. ASIF achieves a non-trivial performance but still lags behind related works such as LIT and CLIP.", "summary_of_the_review": "This paper proposed an interesting idea to bridge two uni-modal pre-trained models w/o training. However, some experiments are missing as detailed in the Weakness section.", "strength_and_weaknesses": "## Strength\n1. interesting idea for turning two unimodal pre-trained model into a multi-modal model w/o training.\n1. the paper is well-written and easy to follow\n\n## Weakness\n\n1. The major assumption of the proposed method is that \"captions of images that are close in the visual space to be themselves close in the language space.\" Please quantitatively prove this claim.\n\n1. How does ASIF scale with respect to the size of multimodal dataset? In Figure 4, it looks like ASIF still has room for performance growth if more data are given. The reviewer is interested in seeing when the performance saturates. As the multimodal dataset scales up to 400M, what's the performance gap between ASIF and CLIP/LiT?\n\n1. What's the relationship between the \"quality\" of the visual encoder and ASIF's performance? For example, do we need to take a larger model (in terms of #params) or a model trained with more data for ASIF to achieve better performance? Or maybe the pre-trained visual encoder's performance on eg ImageNet better correlates with ASIF's performance.\n\n1. sensitivity of hyper-params p and k.\nIn Figure 4, it looks like the performance is quite sensitive to p and k. It would be good to show the grid search results with respect to these two parameters. Further, how sensitive are p and k to different target datasets? For example, the best (p, k) pair for ImageNet vs for Pets. \n\n1. Even though the idea is interesting, what's the future of this method? According to the performance, it doesn't seem like a good alternative to existing multi-modal models such as CLIP or LiT.\n\n1. In Table 1, it would be good to add a column of #params for each model.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject