Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces an unsupervised learning technique called: "Contrastive leave-one out boost", which relies on a contrastive learning objective called "InfoLOOB". The authors compare this objective in detail with the classical InfoNCE objective, and compare the performance of their method with CLIP on a set of zero-shot transfer learning tasks (pre-training occurs on two datasets, conceptual captions and YFCC).
main_review: Positive points:
- The method is clear and the paper well written and structured.
- The proposed CLOOB approach outperforms the chosen baseline on most tasks, demonstrating a clear performance gain.
- Ablation study A1 does seem to demonstrate that the objective introduced has strong synergies with the Hopfield network approach. This justifies the usefulness of the different components.

Concerns:
- The ablation study seems to indicate that InfoLOOB does not work well using the CLIP framework. Could the authors expand on why they thing this is the case?
- While it is true that the authors design a learning algorithm that functions well against their chosen baseline, overall most of the components (the InfoLOOB, modern Hopfield networks, ...) are not intrinsically novel. This does not detract from the performance gains presented in the paper however.
summary_of_the_review: The paper is well written and interesting. The authors combine different components to design CLOOB, a learning algorithm that shows performance gains compared to the chosen baseline. Despite this, the components used in the approach are not novel, they are the result of previous work, meaning that the technical contributions mainly consist in finding ways to adequately combine Hopfield networks and the InfoLOOB objective. I would still recommend acceptance as I do not feel lack of technical novelty alone should detract from the empirical merits of this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new contrastive learning algorithm that comes with two improvements: a) replacing InfoNCE loss with InfoLOOB loss, b) replacing normal deep neural network with Modern Hopfield Network. When applied to image<->text contrastive learning, the new algorithm shows good improvements over baseline (CLIP) on a number of zero-shot image classification benchmarks.
main_review: **Strengths**
 - The paper is well written, with clear motivation, careful algorithm derivation, and detailed experiments + ablations. The comparison between InfoLOOB and InfoNCE losses (Section 2) are carefully made and technical sound (myself didn't rigorously go through the proof, though)
 - Good empirical improvements over CLIP.

**Weaknesses**
 - Contribution / novelty is small: neither InfoLOOB nor Modern Hopfield Network is new
 - In addition to image-text contrastive learning, would love to see how it applies to other contrastive learning tasks (e.g. image-image)
summary_of_the_review: While empirical results are good, the overall contribution of the paper seems fall below the bar. It simply applies a combination of two existing techniques to image-text contrastive learning, and doesn't show how it broadly applies to other contrastive learning tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: By using the InfoNCE loss for model training, CLIP has achieved great success. In this paper, the authors propose CLOOB, short for "Contrastive Leave One Out Boost", where modern Hopfield networks are used together with the InfoLOOB objective. InfoLOOB is a leave-one-out upper bound of mutual information, and modern Hopfield networks replace the original embeddings by retrieved embeddings. Results show that CLOOB outperforms CLIP at zero-shot transfer learning across multiple architectures and datasets. 
main_review: Strengths: 

1) Writing: The paper is generally carefully written, and easy to follow and understand. The reviewer enjoys reading the paper, though did not check the theorems in detail.   

2) Novelty: The proposed CLOOB method is novel. It combines the InfoLOOB objective with modern Hopfield network for the training of a CLIP-like model. This has not been investigated before. Since CLIP is a hot topic in the field, it will be interesting for the community to know how CLIP can be further enhanced. 

3) Experiments: The authors have conducted comprehensive experiments on zero-shot transfer learning, in order to demonstrate the effectiveness of CLOOB when compared with CLIP.  

Weaknesses: 

1) Method:

a) Typically, people will try to maximize an MI lower bound, and minimize an MI upper bound. In this paper, the authors try to maximize an MI upper bound. More discussion and rationales behind this design will be appreciated. 

b) Since the proposed method is general, besides CLIP, did the authors try to apply it to self-supervised visual representation learning, such as MoCo and SimCLR etc.

c) The authors use InfoLOOB as the objective. Since InfoLOOB is an upper bound, a natural question is that why not also try using CLUB (Cheng et al., ICML 2020) as the objective?

d) What exactly is a modern Hopfield network? Eqn. (9) and (10) tried to explain this, but they are not clear enough to me. Why they are better? What's the original application they are proposed for? Since this is a core part of the proposed method, a more detailed description is needed. 

2) Experiments:

a) From results in Figure 1, it is not easy to see clearly variance of InfoLOOB is reduced via modern Hopfield network. Is there any quantitative measure on the variance?

b) It would be interesting to also test ViT backbones, instead of ResNet backbones. 

c) The ablation study in Sec. 4.1 seems important, maybe the authors should consider move them back to the main text if space permits. 

d) In Table 3, what are the performance of CLIP under stronger RN-101 and RN-50x4 backbones? It will be interesting to see how the performance gap changes while making the visual backbone stronger.


Typo:

Figure 2, caption, 4-th and 5-th row, the "image embedding U_{y_i}" should be "text embedding"?
summary_of_the_review: In summary, this paper proposes a novel method for CLIP-like model training, and results demonstrate the effectiveness of the proposed method on zero-shot transfer learning. However, the reviewer also thinks that this paper can be made stronger (see details in my comments), therefore, the reviewer decides to give a Borderline Accept recommendation at this moment.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new contrastive learning method called CLOOB, which minimized the leave-one-out upper bound (InfoLOOB) on mutual information with the modern Hopfield networks. Concretely, Hopfield networks replace the original embeddings by retrieved embeddings in the InfoLOOB objective.  The retrieved embeddings are more robust as they capture the common covariance structure of all sample embeddings, leading to a stable performance of InfoLOOB. Extensive experiments on several zero-shot datasets show that the proposed CLOOB method outperforms the well-known CLIP model across all considered architectures.
main_review: Strengths:

1. This paper is well written and easy to follow. There is almost no typo in the paper.

2. The proposed method is interesting. The introduction of Hopfield networks into contrastive learning to reduce the high variance of InfoLOOB is well motivated. In particular, CLOOB is easy to be implemented in practical applications.

3. The experiments on zero-shot datasets are well extensive and truly demonstrate the advantages of the proposed CLOOB when compared with the latest method CLIP. I appreciate the effectiveness of the proposed method in real-world applications.

4. The content of the appendix is substantial. The proof seems sound and correct, after I checked it.

Weaknesses:

1. The novelty of the proposed method seems not big enough. In my opinion, the proposed method is just to add modern Hopfield networks on top of the encoder to transform the original embedding to an informative one.  The total objective of the loss function in Eq. 13 is still the common-used  InfoLOOB loss. From this perspective, it seems like that the proposed method is just a feature-engineering algorithm, whose novelty, in my opinion, is not convincing enough. However, it is still glad to see that such simple algorithm is efficient in practical applications.

2. My major concern is the originality of the theoretical results of this paper. As far as I can tell, both Theorem 1 and Theorem 2 are the results, or can be directly obtained, from previous work. For example, the InfoNCE as the lower bound of mutual information (MI) is well-known result in the work of Poole2019ICML (cf Eq. 12 in “On Variational Bounds of Mutual Information” ), the InfoLOOB as the upper bound of MI can also be directly derived from Eq. 13 in Poole2019ICML. Therefore, in my opinion, there is no need to present the whole detailed proof of these theoretical results in Appendix, as such proof technique is not originated from the authors. Rigorously speaking, Theorem 1 and Theorem 2 in the main paper need citations as they are just the theoretical results from previous works. Authors of CLOOB could not use such theoretical results as their own.

3.  In view of Weakness 2, the theoretical contribution of this paper is limited. In my opinion, authors should give more theoretical analysis of their proposed CLOOB method, such as whether the introduction of Hopfield networks can theoretically reduce the variance of the learned model, instead of presenting the well-known lower and upper bounds on MI as their main theoretical results. 
summary_of_the_review: I appreciate the effectiveness of the proposed method. However, considering the limited theoretical contribution of this work, and insufficient novelty of the proposed method, I could only give a borderline score (incline to rejection) for this paper.
review_title: null

Meta-review: Four experts reviewed the paper and provided mixed recommendations. All reviewers found the experimental results strong, but they have different views about the technical novelty. Three reviewers considered the technical novelty as a weakness of the paper, but Reviewer z4BR was less concerned about it than the other two. After AC carefully read the paper and the authors' responses, AC agreed with the reviewers that the combination of InfoLOOB and modern Hopfield networks, which were both existing works, is incremental despite the empirical results. Besides, AC agreed with Reviewer jmHN that the theoretical results are not significant enough and could be moved to Appendices. While the empirical results are strong, they could not answer how the trend would change with bigger models and bigger datasets. Hence, while the paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject