Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes ASIF, which can retrieve relevant captions given an image **without training**, utilizing 1) two pre-trained unimodal encoders and 2) a relatively small multimodal datasets. The key intuitive is that “captions of similar images should be themselves similar”; thus the retrieval is done through the relative distances given by the two pre-trained unimodal encoders. 

In experiments, the authors show decent performance on several image classification datasets, using conceptual captions as the multi-modal dataset.
strength_and_weaknesses: Pro: It it interesting to see such an effort on tuning unimodal encoders into multimodal models using non-parametric methods. 

---

Con: clarification needed on relationship with respect to K Nearest Neighbor.

As noted by the authors, ASIF, in essence, is “making predictions on new samples by exploiting the similarity with a dictionary of previous data points”. This makes ASIF extremely similar to KNN. The first half of ASIP, i.e., finding the k most-similar images, looks the same as KNN. What differs is how to find the best caption given the k “neighbor” image-caption pairs. I would imagine a naive approach of finding the caption from the training set that is mostly similar to k captions (using Language Model similarity) would do pretty well (similar to the majority voting in KNN). The author also provides a nice discussion in Sec 4 “learning or retrieval”. However, if we link ASIF to KNN, then it would be clear that ASIF, like KNN, counts as a non-parametric supervised learning algorithm. 

In fact, KNN has been used for evaluating self-supervised visual representation models (e.g., Table 1 of Zhou et al.) and the performance is already high (78.0). But I can see that the performance is not directly comparable as Zhou et al. used ImageNet as the KNN training set while this paper uses Conceptual Captions (CC). The impressive results in the paper might look less “surprising”, if we were to believe that ImageNet and CC have a lot of overlap. Then the current results would be somewhat a domain-shifted version of the KNN experiments in Zhou et al.

I would thus appreciate a discussion on:

 1) what makes ASIF different from and potentially better than KNN (and similar methods);

 2) experiments on a naive KNN baseline on using CC data for ImageNet. 

Zhou et al., iBOT: Image BERT Pre-Training with Online Tokenizer

---

Minor questions:

For the image classification experiments, ASIF will select a caption from the 1.6M captions. How should we map the retrieved caption to a label? Is it a simple rule-based ngram match?

clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow. The results are novel.
summary_of_the_review: I very much like the intuition of ASIF. It works decently in practice and has several nice properties nicely argued by the authors.

However, my main concern is that the method is very similar to KNN and once we consider it as some kind of KNN, the analysis and results look less “surprising”.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed an interesting idea of turning two unimodal pre-trained models into a multi-modal model without training. Specifically, each sample is encoded into a relative representation with respect to paired multi-modal data in its own modality. The relative representation serves as the bridge between two modalities. ASIF achieves a non-trivial performance but still lags behind related works such as LIT and CLIP.
strength_and_weaknesses: ## Strength
1. interesting idea for turning two unimodal pre-trained model into a multi-modal model w/o training.
1. the paper is well-written and easy to follow

## Weakness

1. The major assumption of the proposed method is that "captions of images that are close in the visual space to be themselves close in the language space." Please quantitatively prove this claim.

1. How does ASIF scale with respect to the size of multimodal dataset? In Figure 4, it looks like ASIF still has room for performance growth if more data are given. The reviewer is interested in seeing when the performance saturates. As the multimodal dataset scales up to 400M, what's the performance gap between ASIF and CLIP/LiT?

1. What's the relationship between the "quality" of the visual encoder and ASIF's performance? For example, do we need to take a larger model (in terms of #params) or a model trained with more data for ASIF to achieve better performance? Or maybe the pre-trained visual encoder's performance on eg ImageNet better correlates with ASIF's performance.

1. sensitivity of hyper-params p and k.
In Figure 4, it looks like the performance is quite sensitive to p and k. It would be good to show the grid search results with respect to these two parameters. Further, how sensitive are p and k to different target datasets? For example, the best (p, k) pair for ImageNet vs for Pets. 

1. Even though the idea is interesting, what's the future of this method? According to the performance, it doesn't seem like a good alternative to existing multi-modal models such as CLIP or LiT.

1. In Table 1, it would be good to add a column of #params for each model.
clarity,_quality,_novelty_and_reproducibility: The paper is overall well-written. However, some important experiments mentioned in the Weakness section are expected to further validate the claims and support their conclusions.
summary_of_the_review: This paper proposed an interesting idea to bridge two uni-modal pre-trained models w/o training. However, some experiments are missing as detailed in the Weakness section.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a multimodal model that does not require any training on image-text pairs. The proposed non-parametric model, ASIF, leverages independent pretrained unimodal models to extract embeddings of data points in ground-truth image-text pairs. At inference, ASIF first computes the relative representation of a given input from one modality as the vector containing similarities between the given input and each data point from the dataset. It then treats this vector as the relative representation of the input's ideal correspondence from the other modality, and retrieves the candidate that has the most similar relative representation to the vector. With some sparsification treatments, ASIF achieves quite competitive zero-shot results on several image classification benchmarks, including ImageNet, ImageNet v2, CIFAR100 and Pets, using a relatively small paired dataset.
strength_and_weaknesses: Strength
- The paper is well-written and throughly addresses related work and implications of the proposed method.
- The proposed method is simple yet quite effective for multimodal zero-shot tasks using a relatively small paired dataset.

Weaknesses
- However, it seems that the model performance heavily depends on pretrained unimodal models, and the results reported in the paper come from the unimodal models that are trained on large-scale unimodal data: ImageNet21K (about 14M images) and 1B sentences scraped from the internet. I don't think that it is appropriate to say that ASIF is data efficient using such models. This seems to be more than the LiT training data. Do you have additional results using unimodal models that are trained on less data, such as ImageNet1K?
- Since the model performance also heavily depends on the ground-truth image-text pairs, you need to provide quantitative results on out-of-distribution tasks. I am not sure which dataset among ImageNet (or ImageNet v2), CIFAR100 and Pets is far distant in distribution to the subset of CC12M that you use for computing relative representations.
clarity,_quality,_novelty_and_reproducibility: The paper is easy to read, and the proposed method can be easily implemented just by following the paper's instructions. However, one of the key contributions of the prosed method, data-efficiency, is not well-supported. The method has some technical novelty.
summary_of_the_review: The proposed method is simple yet quite effective for building a multimodal non-parametric model, but one of the key contributions, data efficiency, should be validated with unimodal models that are trained on small-scale datasets.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies an approach that creates a CLIP-like model from a pre-trained image encoder and a pre-trained text encoder using a limited number of image-text pairs using a non-parametric approach. The approach concatenates both encodings and performs nearest neighbor prediction using a dissimilarity-based representation of the resulting representations.

The reviewers are lukewarm on the paper. One of the main concerns they raise is that it is unclear in what scenarios this would be a useful alternative to the current state-of-the-art. The proposed approach achieves acceptable results using 1.6M image-text pairs, but the performs is substantially worse than that of CLIP / LIT (including variants of LIT that are trained on relatively small datasets of publicly available image-text pairs). If such data is publicly available, why wouldn’t one use it?

It should also be noted that the proposed approach is a very standard non-parametric approach that does not scale particularly well, presuming that k needs to increase logarithmically with the training set size for optimal results (as the results in Figure 4 suggest).
justification_for_why_not_higher_score: It is unclear in which scenarios the proposed method is actually useful.
Decision: Reject