Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper argues that the standard contrastive objective adopted in large-scale image-text pre-training (e.g., CLIP) may lead to inconsistency in learned representations for zero-shot image classification application. To improve the inconsistency, the authors propose to add two regularizers to standard contrastive objective to enforce both cross-modal consistency and in-modal consistency. Pre-training on CC3M, the proposed CyCLIP achieves better ZS performance, consistency scores and linear probing results on image classification than CLIP.  
strengths_and_weaknesses: 
In general, the paper is clearly written, easy to follow. The proposed CyCLIP is simple, with just two more added regularizer losses to the standard contrastive loss, yet has shown more effective than CLIP on image classification tasks over different evaluation settings. 

My main concern is about the assumptions made by these regularizer terms, and would love to discuss with the authors. In-modal consistency seem to assume the distances measure in image representation space is at the same scale as that in the text representation space. Despite the input image and text space maybe inherently different, the assumption may not be true when the text does not fully describe the image (which is the majority case in web-crawled data).. 

- image-1: dog, person, beach, ball, sky
- text-1: A person playing with a dog on a beach
- image-2: cat, person, room
- text-2: A person playing with a cat in a room

In my personal opinion, for the above examples, the distances between the texts are much closer than the distances between images.

The symmetry of distance measures for cross-modal inconsistency also may not be valid.

- image-1: dog, person, room, ball
- text-1: A person playing a ball with the dog in a room
- image-2: cat, person, room, ball
- text-2: A person playing with a cat

If both images are the person playing a ball with the [dog/cat] in a room, then I would think <Image-1, text-2> may be greater than <image-2, text-1>. As the edit steps on text-1 to make it fully describe image-2, is just to change the word dog to cat, while more edits are needed on text-2, to make it describe image-1.
questions: - Would the added regularizers affect the performance on some applications other than image classification, for example image-to-text retrieval? 
- It is also not very clear whether the current advantage of CyCLIP would hold when scaling up to larger pretraining corpus. To perform a fast validation, maybe the authors can add 1M image-text pairs from SBU dataset or even 12M data from CC12M as additional pre-training data.
limitations: Discussed in conclusion section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper provides the first analysis of inconsistency problem in contrastive learning of image-text models, and proposes new training objectives to mitigate the issue. Specifically, it demonstrates that within-modality similarities can be inconsistent with cross-modality ones, a potential area of improvement for existing models. Based on this observation, the author propose to add two regularization terms during training to alleviate the issue. Experimental results clearly show the performance gain of the method.
strengths_and_weaknesses: Overall, I found the paper interesting to read. The observation made is inspiring and proposed method does appear to be effective. There are, however, some areas that can be improved for a stronger paper (see below).

Strengths:
1. The inconsistency issue has never been identified before, to the best of my knowledge. I think this is an interesting and important issue that needs to be addressed properly.
2. The proposed methods are intuitive and easy to use. Experimental results clearly show its effectiveness.
3. The paper is well written, with plenty of inspiring analysis.

Weakness:
1. My major concern is the experimental setup. The experiment is conducted on small-scale CLIP models with small batch sizes. As a result, the performance is far away from the real CLIP or CLIP-like methods. It therefore leaves room for doubts on how the analysis and results generalize to larger scales.
2. Another point is that while inconsistency is a concerning issue, it will be nice to show some actual cases where it can hurt model performance, even on a curated synthetic setting.
questions: Some questions/suggestions:
1. In section 2.2, do you have some good intuition/understanding why the model can still obtain good performance using text classifications while the consistency is bad?
2. I wonder if it should be N^2 instead of N in the normalization terms in Eq (3) and (4)?
3. How do you select your hyper-parameters?
4. Have you considered tasks other than image classification? e.g. how does the model perform on cross-model retrieval?
limitations: This has been covered by the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper analyzes the representation space of CLIP and identifies that the image and text distances are inconsistent — the concepts closest to the image in an image-text pair can be very different from the concepts closest to the text. They then propose losses to fix this distortion, and outperform CLIP trained on a similar amount of data. 


strengths_and_weaknesses: The paper points out a very interesting phenomenon in the representation space of CLIP, and proposes a simple method to fix the representation space. The method works quite well empirically. I find this particularly useful because it is orthogonal to many other recent approaches that build on the original CLIP formulation. It should be possible to combine this with any image-text contrastive learner. Overall, the method is presented clearly and convincingly. 

There are two main weaknesses to the paper. The first is the size of the dataset. Would the advantages of this method disappear as the amount of data available increases? At 4M samples, CyCLIP is $2$% better than CLIP-4m on ImageNet1k. How would this change as the amount of data increases, let's say to 15M? Reproducing the full CLIP-400m results is obviously infeasible, and even a 15M dataset is nontrivial, so I think the scale of the evaluation in the paper is _probably_ okay. The paper makes an attempt to address this (Fig. 4). 

The more serious weakness to me is the sparsity of the experimental evaluation of zero-shot classification. Zero-shot evaluation is computationally _extremely_ cheap. The zero-shot results are only included for 3 datasets, of which 2 are CIFAR-10/100. CIFAR-10/100 are very old at this point, and are known to contain many mislabeled samples in the test set [1]. I would like to see an evaluation for some other datasets which are more challenging and cleaner. I leave the choice of dataset up to the authors — the original CLIP paper contains many other datasets that can be used for purposes of comparison. A simple choice would be image-text retrieval, for example on the Flickr30K dataset. 


[1] https://arxiv.org/abs/2103.14749
questions: Please see the weaknesses / strengths.

1) Can this method be easily combined with other xyzCLIP methods, for example, DeCLIP? 
limitations: The authors have adequately addressed the limitations of their work.
review_title: null

Meta-review: metareview: The authors identify the inconsistency problem in CLIP for the first time and proposes a simply but effective solution to the problem. The experiments are comprehensive and the results are convincing. Reviewers' questions are well addressed in the rebuttal. The reviewers all agreed on accepting this paper. After reading the paper, I also agree with the reviewers.
confidence: Certain
Decision: Accept