Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Review: This paper proposes a novel framework, OTTER, which considers the many-to-many relationship within a batch of images and text captions for data-efficient language-supervised zero-shot recognition. An improved InfoNCE is explored to consider the many-to-many relationship between unpaired images and texts. Then to further explore the relationship between images and texts, the optimal transmission method is introduced. Extensive experiments on six popular benchmark datasets demonstrate that OTTER well addresses this challenge, and it only needs fewer samples.

main_review: Pros: 
+ The motivation of this paper is very clear.
+ Overall, the paper is well written. In particular, the INTRODUCTION section has a nice flow. 

Cons: 

 +For the optimal transmission algorithm, in the METHODS part, the author should emphasize the difference from the existing algorithm.

 +More visualizations should be added to analyze the effectiveness of the proposed method (E.g Figure 2).

 +For IN10K FH@K, the proposed OTTER does not achieve satisfactory performance. The author should analyze the reason for this result.


 Minor comments: 

- In the 4th line in 4.2 ABLATION STUDIES, it should be "define" instead of " defined ".
summary_of_the_review: Due to the limited novelty of the proposed method, I need to re-evaluate my judgments after the rebuttal.

UPDATES: The authors have addressed all of my concerns, though experiments on 400M dataset are missed by uncontrollable reasons, as initial judgement, I tend to accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: It is common to train cross-modality embedding models on image-caption datasets using contrastive objectives, assuming each caption correctly describes the corresponding image, and all other captions can be used as negative (incorrect) matches. This paper clearly illustrates that this common assumption is wrong, and it hurts the performance of the learned embeddings on down-stream tasks, namely zero-shot recognition. To address this problem, the authors propose a simple and elegant approach to refine the pairwise image-caption matching, which can be seen as a smarter variation of label smoothing. This simple trick improves the performance of the well-known CLIP model consistently across datasets and backbone architectures.
main_review: Strengths:

1. The authors showcase a fundamental problem of InfoNCE by providing interesting examples in Figure 1 and Figure 3. This is a valuable insight for the community, and may encourage future work to make more realistic assumptions about image-caption datasets.
2. The proposed method is simple and elegantly formulated. It is versatile and can be applied to many vision-language tasks besides zero-shot recognition. There are hyperparameters alpha and lambda that control how much we trust the image-caption pairs, and how much we trust the similarity estimations made by the teacher model.
3. The paper is well-written and easy to follow.
4. The experiments show consistent improvements in various training and evaluation settings, on difficult and realistic classification tasks.

Weaknesses:

1. Although the improvements shown in Table 1 are consistent, they are not significant. Hence, using OTTER may not be worth the extra computational cost. 
2. The comparison to other ZSL methods in Table 2 is unfair, since the baselines are trained on a smaller dataset and with different pretrained image and text encoders.
3. Although in Figures 1 and 3, the authors provide interesting evidence for the problem they identified, it is not clear how prevalent and important this problem is, especially considering the limited impact of the proposed method on the performance. Is it possible that only a handful of examples exist like those shown in the paper? Are there ways to quantify how common this problem is?
4. Although zero-shot recognition is a relatively new problem, learning cross-modality embeddings using image-caption datasets has been studied for a long time, with applications ranging from text-to-image retrieval to visual grounding. It is not clear whether the problem of image-caption alignment noise has been identified and studied before, or the authors are the first to identify this. It is also not clear how effective this method would be for such vision-language tasks besides zero-shot recognition.
5. In Table 1, it is strange that CLIP's performance using 400M images is close to the InfoNCE baselines trained on 3M images, on the GOI dataset. Is that only due to using pretrained image and text encoders? If so, it is important to elaborate the pretraining process, and compare the resources used for pretraining to CLIP's resources. It would also be interesting to show the performance without pretrained encoders, as a lower bound.

summary_of_the_review: The authors identify a new problem and propose a simple, elegant, and versatile solution to solve it, which has a consistent, incremental impact on the important task of zero-shot recognition. The insights and conclusions made by this paper are moderately significant, but not adequately supported, and not strong enough for acceptance.

UPDATE: after reading the authors' response, I have decided to raise my rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of zero-shot image recognition through language-supervised pretraining. The proposed method trains their visual-language encoder pair with a contrastive objective (similar to CLIP) on image-text pairs. The main problem the authors saw and aim to mitigate from previous approaches like CLIP is the existence of missing-matches between a batch of image-text pairs for training. The authors argue that, in a batch, one image can be matched to multiple texts and vice versa. They propose to use Optimal Transport optimization to solve for a soft image-text matching to replace the hard single-to-single matching and serve as the pseudo ground-truth for supervision. 

Then, the pretrained models visual and language encoders are taken to several target datasets (Google Open Images, ImageNet 10K, and ImageNet 21K+1K) for zero-shot recognition (text encoders are used to generate classifiers on the fly for labels in the target datasets). The proposed method demonstrated good results compared to several previous methods and also a set of variants for ablation studies. 

main_review: Strengths:
+ The paper is very-well written and easy to follow. In particular, the approach section is clearly written and presented in a good pedagogical manner. 
+ Overall, IMO the proposed approach solves the missing match problem in language-vision contrastive pretraining in a sensible way.
+ The results seem to suggest that the proposed approach is overall a good fix to existing language-vision pretraining methods, as it generally provides a gain (in certain cases bigger than others) across different target datasets and choice of encoder architectures. 
+ The authors have made good efforts to ablate different components of their approach.   
+ The authors have promised to release their code for training and evaluation. 

Specific things I like in the paper:
+ I like Section 3.5 as it helps clearly distinguish the proposed method to related methods with term swapping.
+ I like the visualization in Figure 3, which gives me a sense of what kind of missing matchings are mined. 

Weaknesses:
- One issue I have for the approach is that, it's not clear to me why self-distillation is necessarily beneficial? Since the authors use the original encoder (or their momentum-updated counterparts) as teachers, what extra information does the distillation bring in? At a conceptual level, the only extra source of information is the introduction of text-text and visual-visual similarities in Eq. 7. But from the results (i.e., 'running SK for 0 iterations' vs the full OTTER method), it seems to suggest that's not the only reason for the improved accuracy. It would be good if the authors could have a summary on where they think all the gains come from. 
- The authors have motivated their approach by giving the example in Figure 1 to show that, there are many missed matches between images and texts. However, there lacks a quantitative study showing how prevalent this problem is. Although one can argue from the improved results, it would be more convincing to have a ballpark of how many missed matches there are in a batch. 
- The proposed method has many parameters (alpha gamma_v gamma_t, eta, lambda), there lacks a detailed sensitivity analysis for them (Table 3 only does coarse ablation that mostly turn on or off the term). 
- Page 5, "we train on the two publicly available datasets, Conceptual Captions 3M (CC) (Sharma et al., 2018), and Wikipedia-base Image-Text Dataset (WIT)", it would be helpful to train on a larger and more noisier dataset like Conceptual Captions 12M to see how the proposed method scales (my impression is that the benefit of the proposed method should be more significant due to higher noise level there). 
- In Table 1, for three image/text encoder pairs "FBNetV3-A/DeCLUTR-Sci-base", "FBNetV3-C/DeCLUTR-Sci-base" and "ResNet50/Sentence-BERT-base", OTTER only has very marginal gain over KD, especially on IN10k, any insights on why this is case?   
- In Table 2, for OTTER, why not use ResNet50 as the image encoder to enable a more fair comparison? Again, for the "FBNetV3-C/DeCLUTR-Sci-base" setting, the gain is quite marginal between OTTER and KD. 

Questions/confusions:
- Page 3, "Directly minimizing Wasserstein loss between image/text embeddings in our case will lead to collapsed representations", any insight why this might be happening? 
- Page 3, "tau is a (trainable) temperature parameter", is there evidence showing the benefit of learning this parameter, instead handpicking one using validation? Also, what value does tau converge to? 
- Table 3, it would be good to have results for setting lambda to 0 (no entropy regularization). 
- Page 6, "with a total batch size of 512 (64 per GPU) for 10 epochs", how long does it take to train for 10 epochs? 

summary_of_the_review: Overall, I like the idea proposed in the paper to use OT to mine missing matches for language-vision pretraining. The paper is clearly written and makes intuitive sense for most part -- I'm overall positive on accepting this paper but still like to hear the authors address my concerns listed above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an image-text training method similar to CLIP. However, in training, instead of using hard targets for image-text pairs, it relies on soft image-text correspondences based on pairwise similarities and optimal transport.

As image-text pairs can possibly have more than one positive match in the training batch, the authors motivate that the usage of soft targets is more appropriate because it better reflects the possible many-to-many relationship between the image and text samples.

The authors claim that such an approach leads to better data efficiency as it achieves higher performance on CC and WIT datasets, which are significantly smaller than the one used by CLIP.
main_review: ---
Strengths:

1. The work has a clear hypothesis and is well-motivated. Many-to-many relationships / soft correspondences are generally important for vision-and-language learning since there is never really one unique and perfect correspondence between an image and text. Additionally, improving data efficiency of pre-training vision-and-language models can be beneficial to the community as many recent works rely on enormous amounts of image-text data. Others would likely upon this work in a similar direction.

2. The proposed method addresses the issue of many-to-many relationships in a technically valid way, and doesn't increase the complexity of the general approach significantly

3. The authors present experiments with a good set of baselines (LS, KD, InfoNCE) as alternative methods and ablation studies on each component of their approach

4. The experiments show that the proposed approach consistently outperforms alternative approaches (baselines)


---
Weaknesses:


1. Although the proposed approach doesn't increase the complexity significantly, it does introduce a number of new hyperparameters – many of which the models seem highly sensitive to (Table 3).
Intuitively, different datasets might have more or less similarities between samples, noise level, etc. - therefore the hyperparameters that work well on CC & WIT, which are of relatively high quality, might not be very transferable to more noisy datasets (like the one used in CLIP, for example).
The ablation studies indicate that, e.g., batch size has a very significant impact on the performance, and intuitively the best choice should depend on the particular dataset since, for a given batch size, the average number of similar samples might vary greatly among different datasets.

2. The comparison with ZSL methods trained on ImageNet1K (Table 2) is misleading. These ZSL methods are trained on ImageNet1K and evaluated on ImageNet21K because of a relatively clear separation between seen and unseen (novel) classes. However, the proposed approach is trained on the CC dataset, which likely has a substantial overlap of concepts with the test set (ImageNet21K). Therefore, the comparison of the prior works and the proposed approach and claims of improving state-of-the-art are somewhat misleading because training on CC means that the model has effectively seen images + names (likely in captions) those classes that are supposed to be "unseen". Even though all works can be said to do zero-shot, the prior methods trained on ImageNet1K do zero-shot with respect to novel classes/concepts, while the proposed method & CLIP model do something more of zero-shot with respect to a dataset (but having possibly seen at least some of the concepts/classes already). Thus, making a side-by-side comparison without discussing this difference can be misleading.

3. The problem of noise in datasets is discussed in the context of more than one possible matching for a given text/image. But what about the image-text pairs not being "accurate" - e.g. text not relevant, or not descriptive enough?


---
Other:

- Question: Why do both $M^V$ and $M^T$ need to be computed (for the OTTER variant)? Shouldn't they be the same (at least for a larger number of Sinkhorn-Knopp iterations)?
summary_of_the_review: The work introduces a valid approach for considering many-to-many correspondences in image-text datasets which is an important problem for vision-and-language learning.

Although the presented results show consistent improvements over alternative approaches, the approach does introduce a few new hyperparameters. Intuitively they might need to be tuned carefully for different datasets to be able to observe any improvement over simpler approaches. The ablation studies do suggest that within a given dataset the sensitivity to some of those hyperparameters is high.

Additionally, the work makes a side-by-side comparison with ZSL prior works, which, unlike this paper, do control the separation between seen and unseen classes/concepts in their training/test data. Such comparison and claims of exceeding ZSL their performance and reaching a new state-of-the-art might lead to misleading conclusions.

**Update (after the authors' response):**

Raised the recommendation to _6: marginally above the acceptance threshold_
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an enhanced model Optimal TransporT distillation for Efficient zero-shot Recognition (OTTER) based on Contrastive Language–Image Pre-training (CLIP). The authors present the potential many-to-many visual to text relationships that CLIP cannot handle. The authors proposed to model the many-to-many relationships as optimal transfer, which portrays the probability relationship based on two intuitions: 1) all the images and texts are equally matchable and have equal matching probabilities; 2) the matching probability from image to caption should depend on their similarity estimation. The authors further use knowledge distillation to enhance the learning of similarity matrices. 
main_review: Strengths
1)	The authors provide an interesting and solid idea based on CLIP.
2)	The presentation is very good and easy to follow. The authors clearly present their ideas and describe the technical details.
3)	The ablation study and visualization analysis of the experimental results are sufficient. The visualization of OTTER’s matching illustrates its effectiveness in handling many-to-many relationships.

Weakness
While the ablation study and visualization analysis are done, the key evaluations are missing. The details can be found as follows.

The experiment results in Tables 1 & 2 cannot plausibly prove OTTER is more effective than CLIP. The larger dataset may relate to more many-to-many relationships when training the model. For example, in the visualization in Figure 2, the maximum many-to-many relationship of a sample is 3. However, the dataset with 400M may contain too many many-to-many relationships like 7 or 8 (maybe more, the data scale is about 100 times the used datasets in this paper). In this case, OTTER may be degraded due to the multiple learning targets. In contrast, focusing on a solid target and ignoring the rest of potential targets, CLIP may still infer the generalized information provided by many-to-many relationships due to the wide range of data collection. Therefore, it is necessary to compare OTTER and CLIP on the same-scaled datasets. The authors provide OTTER using hard labels (InfoNCE) as a baseline, but ZSL methods are sensitive to hyper-parameters and training sets. I doubt that InfoNCE can represent the best performance of CLIP trained on CC (3M) and WIT(5M). Regarding that CLIP does not release the 400M dataset (mentioned in this paper by the authors), the authors may not be able to train OTTER on the 400M dataset. The provided experiments are more like baselines for self-evaluation other than state-of-the-art performance. Without the experimental results using the same training settings, I do not think the authors are able to prove the superiority of OTTER over CLIP fully.

summary_of_the_review: According to my main review, I believe this paper still has room to improve
review_title: null

Meta-review: The paper addresses the interesting  many-to-many assignement problem between a set of images and a set of text. Most reviewers, (and I agree with them) think that the  idea and its application worth being published although the performance improvement
is marginal. I request the authors to update the paper based on the discussions.
Decision: Accept (Poster)