Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes the CLOOB objective for contrastive image-text joint training and performs studies comparing against CLIP objective. Results show improvement in performance on multiple downstream tasks on zero-shot performance. However, there is a concern about novelty of this work, as both modern Hopfield networks and InfoLOOB are proposed previously. In addition, zero shot performance improvements do not consistently reflect on to linear probe performance, which is concerning and hence some experimental setups need more detailed analysis.
strengths_and_weaknesses: ### Strengths

- The method CLOOB improves performance compared to reimplemented CLIP baselines on Resnet architectures.
- Authors show some interesting analysis of the benefits of both Hopfield networks and InfoLOOB

### Weaknesses

- Given both InfoLOOB and modern Hopfield networks are previous works, the proposed method is not novel and just incremental. 
- The improvements from using InfoLOOB as the contrastive objective are already discussed in previous work of Decoupled Contrastive Learning by Yeh et.al, which also shows benefits of applying InfoLOOB against SimCLR, MoCo frameworks. 
- Given the sole method that the paper compares against is CLIP, the work only compares against 8 tasks as opposed to 30+ vision tasks that CLIP evaluates on. All the datasets are open sourced and since the work compares against a reimplementation of CLIP, showing results on more variety of evaluation datasets would be helpful to understand the improvements from CLOOB on a larger variety of tasks.
- One big difference I observed is that the batch sizes used in this work are very small (512, 1024) compared to CLIP (16K, 32K). Since the benefits of CLIP loss improves with significant large batch sizes due to large number of negative samples in the contrastive setup, it is not clear to me how the CLOOB objective compares with CLIP on larger batch sizes.
- Linear probing results in appendix are not very encouraging as CLOOB has mixed results compared to CLIP in Table A5. This is also concerning as the main paper Table 2 shows comparison with CLIP(OpenAI) that shows improvements on most tasks, but with similar experimentation setup, the results are mixed.
- Overall the paper's story has lot of repetition of the similar sentences about why Hopfield networks and InfoLOOB are better. I would encourage the authors to restructure the paper and bring more analysis from the Appendix section to the main paper instead of repetitive sentences explaining the benefits. 
questions: - How does the training time gets affected with CLOOB framework compared to CLIP framework?
- Did the authors also try with other backbones like VIT? Larger models?
- For trying out larger models, datasets larger than YFCC can be used. Example open sourced datasets are https://huggingface.co/datasets/facebook/pmd. 
limitations: Yes authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper propose CLOOB, which utilizes modern Hopfield networks to enchancing the learning of covariance structure in the original multi-modal data. In addition, in order to alleviate the impact of Hopfield networks on InfoNCE, the author proposes to use the InfoLOOB objective. Experiments on several classification datasets demenstrated the effectiveness of CLOOB.
strengths_and_weaknesses: 1. The motivation is clear, i.e. enchancing the learning of covariance structure in the original multi-modal data.

2. It is proved on the several downstream classification dataset that CLOOB improves the pre-training on both CC3M and YFCC subsets.
questions: 1. How much additional computational overhead will the Hopfield Network bring?

2. Besides CLIP, can CLOOB be used in other pre-training methods like DECLIP, FILIP? 

3. CLOOB is only transferred to the downstream classification task, how is the result on the image-text retrieval task?

4. A little worried about the results of CLOOB, the results of RN50 pre-trained on CC3M did not exceed the results of the DECLIP report.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a new contrastive learning methods which leverages Modern Hopfield Networks combined with  InfoLOOB objective. The experiments show that the proposed method has promising results when compared with the baseline method on several benchmark datasets.
strengths_and_weaknesses: strength:
1. The paper is well written with comprehensive details.
2. The proposed method outperforms the baseline methods by a large margin

weakness:
1. The novelty of the paper is below the bar.  It combines the InfoLOOB and modern Hopfield networks, which are all existing works even though the application scenario is new

 
questions: what's the time complexity of the proposed model, and how does it compared with the baseline method?
limitations: see above
review_title: null

Meta-review: metareview: The submission proposes a new objective for contrastive training with images+text. The comparison is made with CLIP. The method, CLOOB, improves upon a reimplementation of CLIP. The main concern here is that InfoLOOB and Hopfield nets are known already and whether this submission is thus incremental. Reviewer 3XnZ points out that InfoLOOB as a contrastive objective was previously discussed, for example. Like the reviewer, I am concerned that the comparisons against CLIP are not quite done properly: fewer tasks (this is improved in the rebuttal) and much smaller batches. The authors *claim* that larger batches sizes would benefit CLOOB too, but it's not something really shown in the work.

I believe the rebuttal addresses many of the reviewers' concerns. I can understand the concerns regarding novelty, but I believe the authors when they say that they use Hopfield nets in a way that wasn't used before. Given the promising results, careful ablations/studies, and relatively small (5%) time penalty, I think this could be of general interest, so I vote for this to be accepted.

confidence: Less certain
Decision: Accept