Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a controlled empirical study that compares the capability of representational transfer between unsupervised image-only model (mostly SimCLR) and unsupervised vision and language models (mostly CLIP). The central question that the paper tries to answer is whether unsupervised vision-and-language models that exploit “language information” are richer than image only models.  Controls in the paper are related to experimental settings such as training data, training architectures, transformations. The primary observations emphasise that vision-and-language models are richer with caveats on the type of vision and language parallel data and scale. 
strength_and_weaknesses: Strength: 
The paper contains an interesting exposition between single modality model  v/s multimodal model and the results confirm that multimodal models are indeed better at representational transfer (in general).

Weakness:
- The claim in the paper about “transfer of representations” seems general (and tad too strong) however this is only evaluated over vision heavy transfer learning benchmarks. It is not clear if the behaviour would be similar for transfer learning on vision and language related benchmarks.   
- The salient observations (relating to language - especially descriptiveness and variability) in the paper are perhaps fairly trivial considering the vast amount of similar work from the area of vision and language and semiotics [1, 2, 3, 4, 5 inter alia].
- In general, the paper lacks empirical rigour, the paper contains a list of experimental interventions, but none are clear (expanding as the next point). 

[1] Automatic description generation from images: A survey of models, datasets, and evaluation measures. Bernardi et al. 2016

[2] On the use of human reference data for evaluating automatic image descriptions. van Miltenburg. 2020

[3] Ways of seeing. Berger. 2008 

[4] Semiotics: the basics. Chandler. 2007

[5] Underspecification in Scene Description-to-Depiction Tasks. Hutchinson et al. 2022

clarity,_quality,_novelty_and_reproducibility: - Section 3.2 lacks relevant citations from language and vision research where similar observations have been repeatedly made. Flickr captions are varied (I am also unable to relate the section to Grice, 1975, perhaps it should be to [6]). 

- The experimental settings and details in the paper are insufficient - such as: 
   - Section 3.3, it is not clear how the captions are designed and further: 
   - How should one quantify consistency? 
   - How should one measure completeness - is this based on ground truth COCO objects for a particular image? 
   - Without these details, it just seems arbitrary.  
- Section 3.3 on how many captions are enough - what is “(high-quality)” captions - are the captions always factual? Nucleus sampling doesn’t always generate factual captions. Perhaps the observations due to plateauing is due to irrelevant captions? 

[6] The construction of social reality. Searl 1995

summary_of_the_review: The current draft of the paper lacks experimental rigour, for an empirical paper most details are currently hand-wavy. The transfer based claims are perhaps supported for vision only transfer learning benchmarks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper performs a thorough investigation of CLIP-like image representation learning.

First, they try to compare the CLIP-like image-language contrastive loss and SimCLR-like image-only contrastive loss in a setting where many confounders are controlled. They conclude that adding language supervision can enable the model to learn more transferable representations than their image-only self-supervision method.

Then, they investigate the effect of data properties on CLIP by training CLIP on three different datasets. They find that language supervision can hurt the model transfer performance in low-data regimes but can be helpful in large-scale settings. Also, "descriptive" captions and low variability of captions can improve the model transfer performance.

Finally, based on their findings, they propose to filter non-descriptive captions by training a classifier to detect if a caption is COCO-like and mitigate caption variability by using GPT-J and COCO captions to generate new captions for existing image-caption datasets. They demonstrate the effectiveness of their proposed approach to CLIP.
 

strength_and_weaknesses: Strengths:
1. The paper performs a relatively systematic study on how image-text contrastive loss is compared with an image-only contrastive loss and what properties of the training data can affect the performance of image-text contrastive loss. By drawing insights from their experiments, they can further demonstrate improvements.
2. When comparing image-text contrastive and image-only contrastive losses, they do a good job of identifying potential confounders and carefully control them as much as they can, making their conclusions more convincing compared with several previous papers.
3. Whether and to what extent language supervision can improve visual representation learning is an interesting and important topic. The conclusions of this paper can be helpful and inspire researchers in the future.

Weaknesses:
1. While image-text and image-only contrastive losses have some similarities, they are not mutually exclusive but can in fact be combined together [1]. Therefore, it would be better to see what are the fine-grained differences between the two paradigms and if combing them together can combine the best of them, instead of merely showing one is better than the other.
2. They find that the "descriptiveness" of caption data can affect the model performance and define the term "descriptiveness" as the extent to which they refer to what is contained in an image, which is somewhat vague and it seems that "descriptiveness" is an antonym of the commonly used term "noise". They use a retrieval system to quantify the "descriptiveness" of a caption and find that COCO>CC>YFCC, which seems like they are just measuring the quality of the datasets. Because CC and YFCC are scraped from the web and the image-caption pairs are noisy and not well-aligned, it is normal and well-known that noisy data can lead to bad performance. A more precise definition and a more suitable quantification method are required so that the difference between their conclusion and the well-known fact that "noisy data can be harmful" is more clear.
3. They choose to study the effect of caption variability on COCO, where the human-written captions are not quite diverse as noted in the paper. The Visual Genome dataset [2] can be used because it contains many more captions per image and each caption focuses on a specific part of its image, which can make their conclusions more convincing.



[1] Mu, Norman, Alexander Kirillov, David Wagner, and Saining Xie. "Slip: Self-supervision meets language-image pre-training." ECCV 2022.
[2] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A. and Bernstein, M.S., 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. It is unclear which specific BLIP checkpoints they use for measuring the descriptiveness of captions and image captioning.

The paper mainly presents some novel empirical findings and two practical ways of improving CLIP.

The code has been submitted and the hyper-parameters have been well-documented. 
summary_of_the_review: The paper focuses on a specific question (i.e. does language supervision lead to more transferable representations than using images alone?) and presents a comprehensive analysis to answer this question. While many of their conclusions are interesting and insightful, I do find some issues that can be improved as stated in the weakness part. I am willing to discuss and change my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the author investigates different popular datasets to explore the influence of the style, and scale of a dataset on vision-language representation pretraining task. The authors find that a dataset with sufficiently large and descriptive captions will be helpful in the vision-language representation learning and the representation learning can be negatively influenced by variability of captions. According to the findings, the author proposed data augmentation methods based on BLIP to boost the pretraining performance of the methods trained with different variants of datasets. 
strength_and_weaknesses: Strength:
Raise an interesting topic that how many and what kind of captions are helpful for the vision-language pretraining.

Give out some ideas of what kind of a vision-language dataset may help model to learn better representations.

Sufficient experiments on different kinds of setting and scale of image-caption training sets to show the influence of different aspects of captions on the representation learning.

Weakness:

Some phenomena can be further explained for the reader to get better understanding. For example,
Why cannot the “more descriptive” BLIP-generated captions beat the performance of COCO dataset?

The paper evaluated the performance of representation learning only with classification-based transfer performance, while some other tasks can also be used to evaluate the capability of representations, i.e. image-text matching, and image textual grounding. The experiments from multiple different kinds of tasks can better support your claim.

Some experiments can be more completed. i.e. CLIP_s 2 and 10 in the left table of Fig. 5.

The proposed data augmentation methods are based on BLIP model which is also a supervised trained with large scale datasets. I doubt if the learned knowledge will be leaked during the data augmentation process especially in the image captioning process mentioned in Section 3.3. This can be view as a kind of knowledge distillation of BLIP model which makes it less supportive for the claim.
clarity,_quality,_novelty_and_reproducibility: The problem raised in this paper, which to explore the influence of setting and scale of vision-language dataset on the representation learning, is that somewhat novel and interesting. 

The paper is clear and easy to read. 

The experiment is shown to be reproduced easily.
summary_of_the_review: In my view, this paper investigates an interesting topic and provide some preliminary rules of dataset design with some quantitative experiments for the vision-language learning.
However, some of experiments are not so convincing and some of them need to be completed to support the claim better, like more differentiation downstream tasks. 
Therefore, I currently consider it marginally below the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the transfer learning performance of vision language models (CLIP) in comparison with SimCLR. The primary motivation here is that SimCLR is essentially an image-image analogue of CLIP allowing for fair comparisons in performance while controlling for a variety of confounders. The paper claims three primary advantages of CLIP over VL. In terms of scaling, CLIP learns more transferrable representations when the data size increases. However, the required captions need to be descriptive. The authors leverage BLIP to score caption quality and show that datasets with higher caption quality lead to CLIP outperforming SimCLR as well as models trained with more low quality captions. The thirdly, the paper also provides experimental evidence that while variability in captions adversely affects VL training, this can be somewhat compensated by adding more data through text augmentations. 
strength_and_weaknesses: **Strengths**
1. The paper is well motivated. It is of great importance to study the exact effects of vision-language training versus pure image based training. The idea of using SimCLR as a controlled baseline is inspired.
2. The experiments have clear hypotheses and support the conclusions presented. Specifically, the use of multiple captions per image as text augmentations shows improvements over standard CLIP training, highlighting the importance of both scale and the variability of captions.
3. The presented approaches to improve VL training for more transferable representations are simple and intuitive.

**Weaknesses/Queries**
1. The overall transfer accuracy reported for SimCLR is much lower than that reported in Chen et al's original work. The difference primarily seems to be an effect of the training dataset chosen (Imagenet in the original v/s COCO here). Could the authors comment on this? One suggestion could be to use Imagenet and Imagenet-captions (Fang et al.) to better understand the difference.
2. Another improvement would be to study the effect of scaling up the architectures with more parameters (Resnet-2x, Resnet-4x) and perhaps with Transformer based architectures to understand if the conclusions hold across a variety of architectures/parameters.

However, I understand that these studies are time-consuming, and the paper in its current form is a valuable study in itself.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and complete with references and experimental details. In terms of quality, as stated above, the experiments are well structured, and do address several open questions (with previously contradicting results) in a clear, concise manner. The ideas of using SimCLR, and BLIP based captioning to study the various effects of text supervision are novel and provide insight into the behavior of such models. The results are reproducible with the provided code. 
summary_of_the_review: Overall, this paper is a positive contribution the community with a well structured study comparing the effect of language supervision to that of image based contrastive training. The authors control for several of the confounders that previous studies lack, and provide actionable insights that will be useful for further understanding vision-language models.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the transfer learning performance of vision language models (CLIP) in comparison with SimCLR. The primary motivation here is that SimCLR is essentially an image-image analogue of CLIP allowing for fair comparisons in performance while controlling for a variety of confounders.

The paper claims three primary advantages of CLIP over VL. In terms of scaling, CLIP learns more transferrable representations when the data size increases. However, the required captions need to be descriptive. The authors leverage BLIP to score caption quality and show that datasets with higher caption quality lead to CLIP outperforming SimCLR as well as models trained with more low quality captions. The thirdly, the paper also provides experimental evidence that while variability in captions adversely affects VL training, this can be somewhat compensated by adding more data through text augmentations.

+ This work is of good importance to study given the popularity of vision-language training currently. It is of timely manner to present the findings to ICLR audiences. 

Concerns still linger, and I double-checked with the one with "reject" rating without hearing back from him/her. Given the authors extended response to this specific reviewer, I think partially they have addressed the concerns raised. 
justification_for_why_not_higher_score: The problem domain (vision and language) is extremely difficult to ensure so-called empirical rigour, thus yielding limited insights to the general ICLR audiences. A poster would be a good position for this work. 
justification_for_why_not_lower_score: Well, if the authors could improve their presentation and ground their claims better, it could leverage this work away from the "could be rejected" category. 

If it is rejected, I wouldn't feel too wrong either.
Decision: Accept: poster